Re: KIP-769: Connect API to retrieve connector configuration definitions

[Viktor Somogyi-Vass <vi...@gmail.com>]

Hi Folks,

I too think this would be a very useful feature. Some of our management
applications would provide a wizard for creating connectors. In this
scenario the user basically would fill out a sample configuration generated
by the UI which would send it back to Connect for validation and eventually
create a new connector. The first part of this workflow can be enhanced if
we had an API that can return the configuration definition of the given
type of connector as the UI application would be able to generate a sample
for the user based on that (nicely drawn diagram:
https://imgur.com/a/7S1Xwm5).
The connector-plugins/{connectorType}/config/validate API essentially works
and returns the data that we need, however it is a HTTP PUT API that is a
bit unintuitive for a fetch-like functionality and also functionally
different as it validates the given (dummy) request. In case of sink
connectors one would need to also provide a topic name.

A suggestion for the KIP: I think it can be useful to return the config
groups and the connector class' name similarly to the validate API just in
case any frontend needs them (and also the response would be more like the
validate API but simpler).

Viktor

On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <ry...@gmail.com> wrote:

> I think it'd be worth adding a GET version, fwiw. Could be the same handler
> with just a different spelling maybe.
>
> On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Hi Chris,
> >
> > You're right, you can achieve the same functionality using the
> > existing validate endpoint.
> > In my mind it was only for validation once you have build a
> > configuration but when used with an empty configuration, it basically
> > serves the same purpose as the proposed new endpoint.
> >
> > I think it's a bit easier to use a GET endpoint but I don't think it
> > really warrants a different endpoint.
> >
> > Thanks
> >
> > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > <ch...@confluent.io.invalid> wrote:
> > >
> > > Hi Mickael,
> > >
> > > I'm wondering about the use case here. The motivation section states
> that
> > > "Connect does not provide a way to see what configurations a connector
> > > requires. Instead users have to go look at the connector documentation
> or
> > > in the worst case, look directly at the connector source code.", and
> that
> > > with this KIP, "users will be able to discover the required
> > configurations
> > > for connectors installed in a Connect cluster" and "tools will be able
> to
> > > generate wizards for configuring and starting connectors".
> > >
> > > Does the existing "PUT
> > /connector-plugins/{connector-type}/config/validate"
> > > endpoint not address these points? What will the newly-proposed
> endpoint
> > > allow users to do that they will not already be able to do with the
> > > existing endpoint?
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> mickael.maison@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I've created KIP-769 to expose connector configuration definitions in
> > > > the Connect API
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > >
> > > > Please take a look and let me know if you have any feedback.
> > > >
> > > > Thanks
> > > >
> >
>

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Randall Hauch <rh...@gmail.com>]

Thanks for the quick response, Mickael. Responses inline below.

Randall

On Fri, Dec 3, 2021 at 10:38 AM Mickael Maison <mi...@gmail.com> wrote:
>
> Thanks Randall for the feedback!
>
> 1. Done
>
> 2. Yes that makes sense. I've updated the KIP
>
> 3.a I've updated to explicitly mention the plugin can be provided
> using the fully qualified class or the alias.
> 3.b Only concrete classes will be returned. I made the mistake when I
> crafted the examples by hand. Fixed
>
> 4. If a plugin does not override config(), the response is an empty
> array. I've updated the KIP
>
> 5. I have updated the compatibility section with more details
>
> 6. Can you clarify what you mean? KIP-507 was for securing internal
> REST endpoints.

My bad. I referenced the wrong KIP. I was simply asking that this KIP
mention the security implication of adding new endpoints. IOW, if a
client can access the existing (non-internal) endpoints, they will be
able to access these new endpoints.

>
> 7. I don't think it is necessary. I think the use case Chris mentioned
> could be useful for admins. I'm interested in exposing the
> connector-level plugins and their configs. I thought it was easy to
> add an API for worker plugins while working in this area but if there
> isn't a strong consensus, I'm happy to drop it.

I see Chris has also responded, so I'll respond to that.

>
> Mickael
>
>
>
>
>
>
> On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch <rh...@gmail.com> wrote:
> >
> > Thanks for the KIP, Mickael. And thanks to everyone else for the good
> > discussion so far.
> >
> > Overall this is pretty good, but I do have a few questions/comments:
> > 1. Can we include the HTTP method/verb in each of the endpoints in the
> > "Public Interfaces" section?
> > 2. The sample output for `GET /connector-plugins?connectorsOnly=false`
> > includes a `version` field for the connector plugin, but not for the
> > transform or converter plugins. Neither the `Transformation` or
> > `Converter` interfaces have a version nor implement the `Versioned`
> > interface, so should we add the `Versioned` interface to the
> > `Transformation` and `Converter` interfaces and add a default method
> > that returns "unknown"? Yes, we could do that in a separate KIP, but I
> > think it's appropriate to do it in this KIP for a few reasons. First,
> > we're already modifying the `Converter` interface in this KIP and it
> > would be quite straightforward to also extend the `Versioned`
> > interface and add a default method; then doing that for
> > `Transformation` would also be relatively straightforward. Second,
> > this KIP provides the justification for adding a version to the
> > `Transformation` and `Converter` interfaces.
> > 3. In the "Public Interfaces" section, there are two inconsistencies
> > between the `GET /connector-plugins` and `GET
> > /connector-plugins/<plugin>/config` methods. Both will make it
> > difficult to use the second method, since the input to the second
> > method is not directly found in the response from the first method:
> >  a. The example for `GET /connector-plugins/Cast$Value/config` uses
> > short name, or what Connect often refers to as the alias, "Cast$Value"
> > for the plugin, rather than the fully-qualified name
> > "org.apache.kafka.connect.transforms.Cast$Value". The KIP should
> > specify whether it will support both the shortened name and the
> > fully-qualified name, or whether only one of these will be supported.
> > Not supporting the fully-qualified name will make it impossible to use
> > the response from the first method as an input to the second.
> >  a. The example for the list method returns the `Transformation` base
> > and abstract classes (e.g.,
> > "org.apache.kafka.connect.transforms.Cast") rather than the concrete
> > classes (e.g., "org.apache.kafka.connect.transforms.Cast$Value" and
> > "org.apache.kafka.connect.transforms.Cast$Key"), while the `GET
> > /connector-plugins/<plugin>/config` will require the concrete class.
> > 4. The "Public Interfaces" section's description of the `GET
> > /connector-plugins/<plugin>/config` method should probably specify the
> > expected behavior for `Converter` implementations that don't override
> > the new `config()` method.
> > 5. The "Compatibility, Deprecation, and Migration Plan" section should
> > describe whether older `Converter` implementations (compiled with
> > older versions of the Connect API without Converter extending
> > Configurable) will be able to be installed into newer Connect
> > runtimes, and what the behavior will be for the `GET
> > /connector-plugins/<plugin>/config` method. (Although my suggestion
> > for #5 will partially address the latter, we still need to talk about
> > it in terms of compatibility.)
> > 6. Can we add a small section addressing the security implications of
> > these new APIs? KIP-507 [1] applied to APIs that existed at the time,
> > and this KIP should at least mention how it aligns with that
> > mechanism.
> > 7. Do we really think adding `GET /worker-plugins` is necessary? While
> > I understand the reasoning, I'm less convinced that it's worth adding
> > to the public API. First, an administrator still has to modify the
> > worker config for all workers to take advantage of that. Second,
> > adding this method increases the surface area and therefore risk of
> > additional attack vectors, despite not really being able to use the
> > resulting information in any other part of the API.
> >
> > Best regards,
> >
> > Randall
> >
> > [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
> >
> >
> > On Wed, Dec 1, 2021 at 7:52 AM Mickael Maison <mi...@gmail.com> wrote:
> > >
> > > Hi Chris,
> > >
> > > Thanks again for the feedback! I've updated the KIP based on our last
> > > discussions. I've decided to include the new endpoint for worker
> > > plugins.
> > >
> > > 1. Yes I agree, it's best to gate the new behavior.
> > > 2. Yes, it was a remnant from the original proposal. I've now removed
> > > the location field.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
> > > <ch...@confluent.io.invalid> wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > I think that's a great idea! I especially like how we can establish the
> > > > expectation that any plugin type that appears in the response from the GET
> > > > /connector-plugins endpoint will have a corresponding GET
> > > > /connector-plugins/<type>/config endpoint, but (if we decide to add them in
> > > > the future), worker plugins won't be expected to expose this kind of
> > > > information and the different root path helps give a decent hint about this.
> > > >
> > > > I also like the choice to return an empty ConfigDef from Converter::config
> > > > instead of null.
> > > >
> > > > Two things come to mind:
> > > >
> > > > 1. We may want to gate this behind a URL query parameter (maybe something
> > > > like "connectorsOnly") that defaults to the old behavior in order to avoid
> > > > breaking existing tools such as programmatic UIs that use the endpoint
> > > > today to discover the connectors that can be created by the user. We can
> > > > even plan to change the default for that parameter to the newly-proposed
> > > > behavior in the next major release, which should give people enough time to
> > > > either adapt to the expanded response format or add the query parameter to
> > > > their tooling.
> > > >
> > > > 2. The existing GET /connector-plugins endpoint doesn't contain information
> > > > on the location of the plugin on the worker's file system. Do you think we
> > > > should still include this info in the new response format? Correct me if
> > > > I'm wrong but it seems it may have been proposed originally to help prevent
> > > > already-addressed bugs in Connect classloading from striking; all else
> > > > equal, I'd personally err on the side of leaving this info out or at least
> > > > reducing permitted values for it to just "classpath" or "plugin path" in
> > > > order to avoid leaking worker file paths into the REST API, which might
> > > > bother super security-conscious users.
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <mi...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Chris,
> > > > >
> > > > > Yes to keep compatibility we want a default implementation for
> > > > > Converter.configs(), I've updated the KIP.
> > > > >
> > > > > Regarding worker plugins, the use case you described seems valuable.
> > > > > I'd prefer not mixing worker and connector plugins on the same
> > > > > endpoint but I agree using /plugins and /worker-plugins could be
> > > > > confusing.
> > > > >
> > > > > One alternative is to expose all connector-level plugins via the
> > > > > existing /connector-plugins endpoint. In that case, we'd need to keep
> > > > > the current JSON schema and not group plugins by type. As the current
> > > > > schema already has a type field for each entry, we'll still be able to
> > > > > tell them apart. Then we can have /worker-plugins and a relatively
> > > > > clean API. What do you think?
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > > > > <ch...@confluent.io.invalid> wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > I think one potential use case for exposing worker-level plugins is that
> > > > > it
> > > > > > may make it easier to determine whether a worker is set up correctly (the
> > > > > > current alternative requires looking through log files and can be a
> > > > > little
> > > > > > tedious), and might even make it possible to automatically identify
> > > > > > discrepancies within a cluster by diffing the contents of that endpoint
> > > > > > across each worker. But I don't think this has to be addressed by the
> > > > > > current KIP; the only thing that bothers me a little is that "plugins" is
> > > > > > generic and it may confuse people down the road if we add an endpoint for
> > > > > > worker-level plugins ("why is one just called 'plugins' and the other one
> > > > > > is 'worker-plugins'?"). Probably not worth blocking on, though.
> > > > > >
> > > > > > Agreed that the suggestion for improved validation should be made on the
> > > > > > KIP-802 thread.
> > > > > >
> > > > > > I also noticed that the newly-proposed config method for the Converter
> > > > > > interface doesn't have a default implementation, making it
> > > > > > backwards-incompatible. Should we add a default implementation that
> > > > > returns
> > > > > > either null or an empty ConfigDef?
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <mickael.maison@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Chris,
> > > > > > >
> > > > > > > 1. If we want to expose worker plugins, I think we should do it via a
> > > > > > > separate endpoint. But to be honest, I'm not even sure I see strong
> > > > > > > use cases for exposing them as they are either enabled or not and
> > > > > > > can't be changed at runtime. So I'd prefer to stick to "connector
> > > > > > > level" plugins in this KIP. Let me now if you have use cases, I'm open
> > > > > > > to reconsider this choice.
> > > > > > > I'll add that in the rejected alternatives section for now
> > > > > > >
> > > > > > > 2. I remembered seeing issues in the past with multiple plugin.path
> > > > > > > entries but I tried today and I was able to mix and match plugins from
> > > > > > > different paths. So my bad for getting confused.
> > > > > > > Then I agree, it makes more sense to group them by plugin type.
> > > > > > >
> > > > > > > 3. Yes this should be covered in KIP-802:
> > > > > > >
> > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > > > > >
> > > > > > > 4. No particular reason. We can support both formats like today. I've
> > > > > > > updated the KIP
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mickael
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > >
> > > > > > > > Hi Mickael,
> > > > > > > >
> > > > > > > > I think the increase in scope here is great and the added value
> > > > > certainly
> > > > > > > > justifies the proposed changes. I have some thoughts but overall I
> > > > > like
> > > > > > > the
> > > > > > > > direction this is going in now.
> > > > > > > >
> > > > > > > > 1. The new /plugins endpoint is described as containing "all plugins
> > > > > that
> > > > > > > > are Connectors, Transformations, Converters, HeaderConverters and
> > > > > > > > Predicates". So essentially, it looks like we want to expose all
> > > > > plugins
> > > > > > > > that are configured on a per-connector basis, but exclude plugins
> > > > > that
> > > > > > > are
> > > > > > > > configured on a per-worker basis (such as config providers and REST
> > > > > > > > extensions). Do you think it may be valuable to expose information on
> > > > > > > > worker-level plugins as well?
> > > > > > > >
> > > > > > > > 2. The description for the new /plugins endpoint also states that
> > > > > > > "Plugins
> > > > > > > > will be grouped by plugin.path. This will make it clear to users
> > > > > what's
> > > > > > > > available to use as it's not possible to use a Connector from one
> > > > > path
> > > > > > > with
> > > > > > > > Transformations from another.". Is this true? I thought that
> > > > > Connect's
> > > > > > > > classloading made it possible to package
> > > > > > > > converters/transformations/predicates completely independently from
> > > > > each
> > > > > > > > other, and to reference them from also-independently-packaged
> > > > > connectors.
> > > > > > > > If it turns out that this is the case, could we consider
> > > > > restructuring
> > > > > > > the
> > > > > > > > response to be grouped by plugin type instead of by classloader?
> > > > > There's
> > > > > > > > also the ungrouped format proposed in KIP-494 (
> > > > > > > >
> > > > > > >
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > > > > )
> > > > > > > > which we might consider as well.
> > > > > > > >
> > > > > > > > 3. I think this can be left for a follow-up KIP if necessary, but I'm
> > > > > > > > curious about your thoughts on adding new validate methods to all
> > > > > > > > connector-level plugins that can be used similarly to how the
> > > > > existing
> > > > > > > > Connector::validate method (
> > > > > > > >
> > > > > > >
> > > > > https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > > > > )
> > > > > > > > is used. This would allow for plugins to perform validation that's
> > > > > more
> > > > > > > > sophisticated than what the ConfigDef is capable of, such as
> > > > > validating
> > > > > > > > combinations of properties like a hostname and credentials for
> > > > > reaching
> > > > > > > it.
> > > > > > > > I know that at least Confluent's Avro, protobuf, and JSON schema
> > > > > > > converters
> > > > > > > > would benefit from this kind of feature. It's a little tangential to
> > > > > this
> > > > > > > > KIP (which at the moment is about discovering plugins and their
> > > > > > > > configuration surfaces, as opposed to validating them), but I
> > > > > figured I'd
> > > > > > > > ask since we're going to be expanding the Converter interface and it
> > > > > may
> > > > > > > be
> > > > > > > > useful to tackle this while we're in the neighborhood.
> > > > > > > >
> > > > > > > > 4. The description for the new /plugins/<type>/<name>/configdef
> > > > > endpoint
> > > > > > > > states that "Name must be the fully qualified class name of the
> > > > > plugin".
> > > > > > > > Any reason not to also support aliases (e.g.,
> > > > > "FileStreamSinkConnector"
> > > > > > > or
> > > > > > > > "FileStreamSink" instead of
> > > > > > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <
> > > > > > > mickael.maison@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks all for the feedback!
> > > > > > > > >
> > > > > > > > > Chris,
> > > > > > > > > I agree that fixing the current endpoint helps a lot. Thanks for
> > > > > > > > > raising these JIRAs and submitting a PR!
> > > > > > > > > However thinking about the issue further, I decided to expand the
> > > > > > > > > scope of the KIP to cover all user-visible plugins.
> > > > > > > > > In practice, users want to know about all available plugins not
> > > > > only
> > > > > > > > > connectors. This includes transformations, converters,
> > > > > > > > > header_converters and predicates. As we also want to retrieve
> > > > > > > > > configdef for these too, I think it makes sense to introduce a new
> > > > > > > > > endpoint to do so. Alongside we obviously need a new endpoint for
> > > > > > > > > listing all plugins.
> > > > > > > > >
> > > > > > > > > Gunnar,
> > > > > > > > > I took a look at exposing valid values via the API. I think the
> > > > > issue
> > > > > > > > > is that Validators don't expose a way to retrieve valid values.
> > > > > > > > > Changing validators will have an impact on all components so I'd
> > > > > > > > > prefer to address this requirement in a separate KIP. I agree this
> > > > > > > > > would be an interesting improvement and I'd happy to write a KIP
> > > > > for
> > > > > > > > > it too.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I have updated the KIP accordingly. Let me know if you have further
> > > > > > > > > feedback.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Mickael
> > > > > > > > >
> > > > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > > > > > > > > <gu...@googlemail.com.invalid> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I'm +1 for adding a GET endpoint for obtaining config
> > > > > definitions. It
> > > > > > > > > > always felt odd to me that one has to issue a PUT for that
> > > > > purpose.
> > > > > > > If
> > > > > > > > > > nothing else, it'd be better in terms of discoverability of the
> > > > > KC
> > > > > > > REST
> > > > > > > > > API.
> > > > > > > > > >
> > > > > > > > > > One additional feature request I'd have is to expose the valid
> > > > > enum
> > > > > > > > > > constants for enum-typed options. That'll help to display the
> > > > > values
> > > > > > > in a
> > > > > > > > > > drop-down or via radio buttons in a UI, give us tab completion in
> > > > > > > kcctl,
> > > > > > > > > > etc.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > >
> > > > > > > > > > --Gunnar
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > > > > > > > > > <ch...@confluent.io.invalid>:
> > > > > > > > > >
> > > > > > > > > > > Hi Viktor,
> > > > > > > > > > >
> > > > > > > > > > > It sounds like there are three major points here in favor of a
> > > > > new
> > > > > > > GET
> > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > >
> > > > > > > > > > > 1. You cannot issue a blank ("dummy") request for sink
> > > > > connectors
> > > > > > > > > because a
> > > > > > > > > > > topic list/topic regex has to be supplied (otherwise the PUT
> > > > > > > endpoint
> > > > > > > > > > > returns a 500 response)
> > > > > > > > > > > 2. A dummy request still triggers custom validations by the
> > > > > > > connector,
> > > > > > > > > > > which may be best to avoid if we know for sure that the config
> > > > > > > isn't
> > > > > > > > > worth
> > > > > > > > > > > validating yet
> > > > > > > > > > > 3. It's more ergonomic and intuitive to be able to issue a GET
> > > > > > > request
> > > > > > > > > > > without having to give a dummy connector config
> > > > > > > > > > >
> > > > > > > > > > > With regards to 1, this is actually a bug in Connect (
> > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix
> > > > > > > already
> > > > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > > > https://github.com/apache/kafka/pull/11369). I think it'd be
> > > > > > > better to
> > > > > > > > > > > focus on fixing this bug in general instead of implementing a
> > > > > new
> > > > > > > REST
> > > > > > > > > > > endpoint in order to allow people to work around it.
> > > > > > > > > > >
> > > > > > > > > > > With regards to 2, this is technically possible but I'm unsure
> > > > > > > it'd be
> > > > > > > > > too
> > > > > > > > > > > common out in the wild given that most validations that could
> > > > > be
> > > > > > > > > expensive
> > > > > > > > > > > would involve things like connecting to a database, checking
> > > > > if a
> > > > > > > cloud
> > > > > > > > > > > storage bucket exists, etc., none of which are possible without
> > > > > > > some
> > > > > > > > > > > configuration properties from the user (db hostname, bucket
> > > > > name,
> > > > > > > > > etc.).
> > > > > > > > > > >
> > > > > > > > > > > With regards to 3, I do agree that it'd be easier for people
> > > > > > > designing
> > > > > > > > > UIs
> > > > > > > > > > > to have a GET API to work against. I'm just not sure it's
> > > > > worth the
> > > > > > > > > > > additional implementation, testing, and maintenance burden. If
> > > > > it
> > > > > > > were
> > > > > > > > > > > possible to issue a PUT request without unexpected 500s for
> > > > > invalid
> > > > > > > > > > > configs, would that suffice? AFAICT it'd basically be as
> > > > > simple as
> > > > > > > > > issuing
> > > > > > > > > > > a PUT request with a dummy body consisting of nothing except
> > > > > the
> > > > > > > > > connector
> > > > > > > > > > > class (which at this point we might even make unnecessary and
> > > > > just
> > > > > > > > > > > automatically replace with the connector class from the URL)
> > > > > and
> > > > > > > then
> > > > > > > > > > > filtering the response to just grab the "definition" field of
> > > > > each
> > > > > > > > > element
> > > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > Chris
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Folks,
> > > > > > > > > > > >
> > > > > > > > > > > > I too think this would be a very useful feature. Some of our
> > > > > > > > > management
> > > > > > > > > > > > applications would provide a wizard for creating connectors.
> > > > > In
> > > > > > > this
> > > > > > > > > > > > scenario the user basically would fill out a sample
> > > > > configuration
> > > > > > > > > > > generated
> > > > > > > > > > > > by the UI which would send it back to Connect for validation
> > > > > and
> > > > > > > > > > > eventually
> > > > > > > > > > > > create a new connector. The first part of this workflow can
> > > > > be
> > > > > > > > > enhanced
> > > > > > > > > > > if
> > > > > > > > > > > > we had an API that can return the configuration definition
> > > > > of the
> > > > > > > > > given
> > > > > > > > > > > > type of connector as the UI application would be able to
> > > > > > > generate a
> > > > > > > > > > > sample
> > > > > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > > The connector-plugins/{connectorType}/config/validate API
> > > > > > > essentially
> > > > > > > > > > > works
> > > > > > > > > > > > and returns the data that we need, however it is a HTTP PUT
> > > > > API
> > > > > > > that
> > > > > > > > > is a
> > > > > > > > > > > > bit unintuitive for a fetch-like functionality and also
> > > > > > > functionally
> > > > > > > > > > > > different as it validates the given (dummy) request. In case
> > > > > of
> > > > > > > sink
> > > > > > > > > > > > connectors one would need to also provide a topic name.
> > > > > > > > > > > >
> > > > > > > > > > > > A suggestion for the KIP: I think it can be useful to return
> > > > > the
> > > > > > > > > config
> > > > > > > > > > > > groups and the connector class' name similarly to the
> > > > > validate
> > > > > > > API
> > > > > > > > > just
> > > > > > > > > > > in
> > > > > > > > > > > > case any frontend needs them (and also the response would be
> > > > > more
> > > > > > > > > like
> > > > > > > > > > > the
> > > > > > > > > > > > validate API but simpler).
> > > > > > > > > > > >
> > > > > > > > > > > > Viktor
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > > > ryannedolan@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I think it'd be worth adding a GET version, fwiw. Could be
> > > > > the
> > > > > > > same
> > > > > > > > > > > > handler
> > > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > You're right, you can achieve the same functionality
> > > > > using
> > > > > > > the
> > > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > > In my mind it was only for validation once you have
> > > > > build a
> > > > > > > > > > > > > > configuration but when used with an empty configuration,
> > > > > it
> > > > > > > > > basically
> > > > > > > > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think it's a bit easier to use a GET endpoint but I
> > > > > don't
> > > > > > > > > think it
> > > > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm wondering about the use case here. The motivation
> > > > > > > section
> > > > > > > > > > > states
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > "Connect does not provide a way to see what
> > > > > configurations
> > > > > > > a
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > > requires. Instead users have to go look at the
> > > > > connector
> > > > > > > > > > > > documentation
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > in the worst case, look directly at the connector
> > > > > source
> > > > > > > > > code.",
> > > > > > > > > > > and
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > with this KIP, "users will be able to discover the
> > > > > required
> > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > for connectors installed in a Connect cluster" and
> > > > > "tools
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > able
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > generate wizards for configuring and starting
> > > > > connectors".
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > > endpoint not address these points? What will the
> > > > > > > newly-proposed
> > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > allow users to do that they will not already be able
> > > > > to do
> > > > > > > > > with the
> > > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I've created KIP-769 to expose connector
> > > > > configuration
> > > > > > > > > > > definitions
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Please take a look and let me know if you have any
> > > > > > > feedback.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Mickael Maison <mi...@gmail.com>]

Hi,

I have updated the KIP accordingly.

Thanks,
Mickael

On Fri, Jan 28, 2022 at 7:01 PM Chris Egerton
<ch...@confluent.io.invalid> wrote:
>
> I think a simple approach here is fine. +1 for removing versioning for
> other plugins.
>
> On Fri, Jan 28, 2022 at 11:17 AM Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Thanks Chris and Tom for the feedback.
> >
> > I'm having a hard time making a decision on this point!
> >
> > Re-reading your previous comments, I agree that we're lacking a
> > contract on the version field, and we can see value (supporting
> > multiple versions for example) if this was improved.
> > But I'm starting to wonder if this should be handled in a different KIP.
> >
> > In this KIP, we can keep the existing Versioned interface for
> > Connectors and either not display a version for other plugins or
> > hard-code it to something like "N/A". Later if we think we're missing
> > the version for other plugins we can decide how to handle it. While it
> > would have been nice to harmonize all plugins, I don't think it's a
> > requirement or blocker for the rest of the value of this KIP.
> >
> > Unless you disagree and feel like we need versions for all plugins,
> > I'll update the KIP accordingly.
> >
> > Thanks,
> > Mickael
> >
> >
> >
> >
> >
> > On Thu, Jan 27, 2022 at 9:00 PM Tom Bentley <tb...@redhat.com> wrote:
> > >
> > > To be honest I'm very sceptical that you can evolve (without breaking
> > > compatibility) from a situation of having a large number of plugins which
> > > have already implemented version() to return whatever they please, to a
> > > situation where you can rely on it as the basis for some feature to
> > support
> > > multiple versions of plugins. It would be much safer to introduce a new
> > > versioning concept, perhaps using Jar manifests and
> > > "Specification-Version", than to try to build on Versioned.
> > >
> > > So I would prefer Javadoc that clearly said that this version number was
> > a
> > > means for the plugin author to identify the version of their plugin in
> > use
> > > by users, for the purpose of bug reporting. That at least makes is clear
> > > that Connect doesn't do anything with it except pass it through to the
> > API.
> > > It also makes clear that using "undefined" is a perfectly fine
> > > implementation for PossiblyVersioned to have.
> > >
> > > Kind regards,
> > >
> > > Tom
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, 27 Jan 2022 at 18:37, Chris Egerton <chrise@confluent.io.invalid
> > >
> > > wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > +1 for the warning at startup, it'd be great if we could get everyone
> > to
> > > > start versioning all of their components. Although given the current
> > state
> > > > of WARN-level logging in Connect (see
> > > > https://issues.apache.org/jira/browse/KAFKA-7509) there might be too
> > much
> > > > noise for most people to notice, unfortunately. Still, better than
> > nothing!
> > > >
> > > > I'm not sure it'd be wise to move quite so quickly with deprecating and
> > > > then removing the PossiblyVersioned interface. In an ideal world
> > everyone
> > > > would update their converters, SMTs, and predicates as soon as they
> > saw the
> > > > warning, but there could be some plugins out there that aren't
> > maintained
> > > > anymore and would become incompatible with a Connect runtime that
> > requires
> > > > them to provide versions. I can imagine there might be some fairly
> > trivial
> > > > but valuable SMTs or converters that were written years ago and haven't
> > > > been touched in some time but are still useful to Connect users today.
> > > >
> > > > I think it'd be fine to log a scary-looking warning message stating
> > that
> > > > the plugin may be incompatible with future versions of Connect, but
> > perhaps
> > > > we can leave the decision about how and when we actually create that
> > > > incompatibility to a future KIP. If we take the example use case of
> > loading
> > > > multiple versions of the same plugin on the same worker, for example,
> > that
> > > > feature would obviously require plugins to provide versions, but we
> > could
> > > > potentially add some failsafe logic in order to still support
> > versionless
> > > > plugins with the understanding that it would not be possible to
> > colocate
> > > > multiple versions of them on the same worker.
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Thu, Jan 27, 2022 at 7:43 AM Mickael Maison <
> > mickael.maison@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Chris,
> > > > >
> > > > > Thanks for taking another look!
> > > > >
> > > > > That's a good point which I should have mentioned in the KIP. I think
> > > > > we can have a PossiblyVersioned interface that extends Versioned and
> > > > > has the default implementation. I'd like to also print a warning at
> > > > > startup if a plugin does not override version(). Thanks for the
> > > > > suggestion!
> > > > >
> > > > > I'm even considering whether PossiblyVersioned could be deprecated
> > > > > immediately in order to be removed in the next major release. If so,
> > > > > the warning message I mentioned would state that plugins not
> > > > > overriding version() would stop working in the next major release.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > >
> > > >
> >

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Chris Egerton <ch...@confluent.io.INVALID>]

I think a simple approach here is fine. +1 for removing versioning for
other plugins.

On Fri, Jan 28, 2022 at 11:17 AM Mickael Maison <mi...@gmail.com>
wrote:

> Thanks Chris and Tom for the feedback.
>
> I'm having a hard time making a decision on this point!
>
> Re-reading your previous comments, I agree that we're lacking a
> contract on the version field, and we can see value (supporting
> multiple versions for example) if this was improved.
> But I'm starting to wonder if this should be handled in a different KIP.
>
> In this KIP, we can keep the existing Versioned interface for
> Connectors and either not display a version for other plugins or
> hard-code it to something like "N/A". Later if we think we're missing
> the version for other plugins we can decide how to handle it. While it
> would have been nice to harmonize all plugins, I don't think it's a
> requirement or blocker for the rest of the value of this KIP.
>
> Unless you disagree and feel like we need versions for all plugins,
> I'll update the KIP accordingly.
>
> Thanks,
> Mickael
>
>
>
>
>
> On Thu, Jan 27, 2022 at 9:00 PM Tom Bentley <tb...@redhat.com> wrote:
> >
> > To be honest I'm very sceptical that you can evolve (without breaking
> > compatibility) from a situation of having a large number of plugins which
> > have already implemented version() to return whatever they please, to a
> > situation where you can rely on it as the basis for some feature to
> support
> > multiple versions of plugins. It would be much safer to introduce a new
> > versioning concept, perhaps using Jar manifests and
> > "Specification-Version", than to try to build on Versioned.
> >
> > So I would prefer Javadoc that clearly said that this version number was
> a
> > means for the plugin author to identify the version of their plugin in
> use
> > by users, for the purpose of bug reporting. That at least makes is clear
> > that Connect doesn't do anything with it except pass it through to the
> API.
> > It also makes clear that using "undefined" is a perfectly fine
> > implementation for PossiblyVersioned to have.
> >
> > Kind regards,
> >
> > Tom
> >
> >
> >
> >
> >
> >
> > On Thu, 27 Jan 2022 at 18:37, Chris Egerton <chrise@confluent.io.invalid
> >
> > wrote:
> >
> > > Hi Mickael,
> > >
> > > +1 for the warning at startup, it'd be great if we could get everyone
> to
> > > start versioning all of their components. Although given the current
> state
> > > of WARN-level logging in Connect (see
> > > https://issues.apache.org/jira/browse/KAFKA-7509) there might be too
> much
> > > noise for most people to notice, unfortunately. Still, better than
> nothing!
> > >
> > > I'm not sure it'd be wise to move quite so quickly with deprecating and
> > > then removing the PossiblyVersioned interface. In an ideal world
> everyone
> > > would update their converters, SMTs, and predicates as soon as they
> saw the
> > > warning, but there could be some plugins out there that aren't
> maintained
> > > anymore and would become incompatible with a Connect runtime that
> requires
> > > them to provide versions. I can imagine there might be some fairly
> trivial
> > > but valuable SMTs or converters that were written years ago and haven't
> > > been touched in some time but are still useful to Connect users today.
> > >
> > > I think it'd be fine to log a scary-looking warning message stating
> that
> > > the plugin may be incompatible with future versions of Connect, but
> perhaps
> > > we can leave the decision about how and when we actually create that
> > > incompatibility to a future KIP. If we take the example use case of
> loading
> > > multiple versions of the same plugin on the same worker, for example,
> that
> > > feature would obviously require plugins to provide versions, but we
> could
> > > potentially add some failsafe logic in order to still support
> versionless
> > > plugins with the understanding that it would not be possible to
> colocate
> > > multiple versions of them on the same worker.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Thu, Jan 27, 2022 at 7:43 AM Mickael Maison <
> mickael.maison@gmail.com>
> > > wrote:
> > >
> > > > Hi Chris,
> > > >
> > > > Thanks for taking another look!
> > > >
> > > > That's a good point which I should have mentioned in the KIP. I think
> > > > we can have a PossiblyVersioned interface that extends Versioned and
> > > > has the default implementation. I'd like to also print a warning at
> > > > startup if a plugin does not override version(). Thanks for the
> > > > suggestion!
> > > >
> > > > I'm even considering whether PossiblyVersioned could be deprecated
> > > > immediately in order to be removed in the next major release. If so,
> > > > the warning message I mentioned would state that plugins not
> > > > overriding version() would stop working in the next major release.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > >
> > >
>

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Mickael Maison <mi...@gmail.com>]

Thanks Chris and Tom for the feedback.

I'm having a hard time making a decision on this point!

Re-reading your previous comments, I agree that we're lacking a
contract on the version field, and we can see value (supporting
multiple versions for example) if this was improved.
But I'm starting to wonder if this should be handled in a different KIP.

In this KIP, we can keep the existing Versioned interface for
Connectors and either not display a version for other plugins or
hard-code it to something like "N/A". Later if we think we're missing
the version for other plugins we can decide how to handle it. While it
would have been nice to harmonize all plugins, I don't think it's a
requirement or blocker for the rest of the value of this KIP.

Unless you disagree and feel like we need versions for all plugins,
I'll update the KIP accordingly.

Thanks,
Mickael





On Thu, Jan 27, 2022 at 9:00 PM Tom Bentley <tb...@redhat.com> wrote:
>
> To be honest I'm very sceptical that you can evolve (without breaking
> compatibility) from a situation of having a large number of plugins which
> have already implemented version() to return whatever they please, to a
> situation where you can rely on it as the basis for some feature to support
> multiple versions of plugins. It would be much safer to introduce a new
> versioning concept, perhaps using Jar manifests and
> "Specification-Version", than to try to build on Versioned.
>
> So I would prefer Javadoc that clearly said that this version number was a
> means for the plugin author to identify the version of their plugin in use
> by users, for the purpose of bug reporting. That at least makes is clear
> that Connect doesn't do anything with it except pass it through to the API.
> It also makes clear that using "undefined" is a perfectly fine
> implementation for PossiblyVersioned to have.
>
> Kind regards,
>
> Tom
>
>
>
>
>
>
> On Thu, 27 Jan 2022 at 18:37, Chris Egerton <ch...@confluent.io.invalid>
> wrote:
>
> > Hi Mickael,
> >
> > +1 for the warning at startup, it'd be great if we could get everyone to
> > start versioning all of their components. Although given the current state
> > of WARN-level logging in Connect (see
> > https://issues.apache.org/jira/browse/KAFKA-7509) there might be too much
> > noise for most people to notice, unfortunately. Still, better than nothing!
> >
> > I'm not sure it'd be wise to move quite so quickly with deprecating and
> > then removing the PossiblyVersioned interface. In an ideal world everyone
> > would update their converters, SMTs, and predicates as soon as they saw the
> > warning, but there could be some plugins out there that aren't maintained
> > anymore and would become incompatible with a Connect runtime that requires
> > them to provide versions. I can imagine there might be some fairly trivial
> > but valuable SMTs or converters that were written years ago and haven't
> > been touched in some time but are still useful to Connect users today.
> >
> > I think it'd be fine to log a scary-looking warning message stating that
> > the plugin may be incompatible with future versions of Connect, but perhaps
> > we can leave the decision about how and when we actually create that
> > incompatibility to a future KIP. If we take the example use case of loading
> > multiple versions of the same plugin on the same worker, for example, that
> > feature would obviously require plugins to provide versions, but we could
> > potentially add some failsafe logic in order to still support versionless
> > plugins with the understanding that it would not be possible to colocate
> > multiple versions of them on the same worker.
> >
> > Cheers,
> >
> > Chris
> >
> > On Thu, Jan 27, 2022 at 7:43 AM Mickael Maison <mi...@gmail.com>
> > wrote:
> >
> > > Hi Chris,
> > >
> > > Thanks for taking another look!
> > >
> > > That's a good point which I should have mentioned in the KIP. I think
> > > we can have a PossiblyVersioned interface that extends Versioned and
> > > has the default implementation. I'd like to also print a warning at
> > > startup if a plugin does not override version(). Thanks for the
> > > suggestion!
> > >
> > > I'm even considering whether PossiblyVersioned could be deprecated
> > > immediately in order to be removed in the next major release. If so,
> > > the warning message I mentioned would state that plugins not
> > > overriding version() would stop working in the next major release.
> > >
> > > Thanks,
> > > Mickael
> > >
> >
> >

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Tom Bentley <tb...@redhat.com>]

To be honest I'm very sceptical that you can evolve (without breaking
compatibility) from a situation of having a large number of plugins which
have already implemented version() to return whatever they please, to a
situation where you can rely on it as the basis for some feature to support
multiple versions of plugins. It would be much safer to introduce a new
versioning concept, perhaps using Jar manifests and
"Specification-Version", than to try to build on Versioned.

So I would prefer Javadoc that clearly said that this version number was a
means for the plugin author to identify the version of their plugin in use
by users, for the purpose of bug reporting. That at least makes is clear
that Connect doesn't do anything with it except pass it through to the API.
It also makes clear that using "undefined" is a perfectly fine
implementation for PossiblyVersioned to have.

Kind regards,

Tom






On Thu, 27 Jan 2022 at 18:37, Chris Egerton <ch...@confluent.io.invalid>
wrote:

> Hi Mickael,
>
> +1 for the warning at startup, it'd be great if we could get everyone to
> start versioning all of their components. Although given the current state
> of WARN-level logging in Connect (see
> https://issues.apache.org/jira/browse/KAFKA-7509) there might be too much
> noise for most people to notice, unfortunately. Still, better than nothing!
>
> I'm not sure it'd be wise to move quite so quickly with deprecating and
> then removing the PossiblyVersioned interface. In an ideal world everyone
> would update their converters, SMTs, and predicates as soon as they saw the
> warning, but there could be some plugins out there that aren't maintained
> anymore and would become incompatible with a Connect runtime that requires
> them to provide versions. I can imagine there might be some fairly trivial
> but valuable SMTs or converters that were written years ago and haven't
> been touched in some time but are still useful to Connect users today.
>
> I think it'd be fine to log a scary-looking warning message stating that
> the plugin may be incompatible with future versions of Connect, but perhaps
> we can leave the decision about how and when we actually create that
> incompatibility to a future KIP. If we take the example use case of loading
> multiple versions of the same plugin on the same worker, for example, that
> feature would obviously require plugins to provide versions, but we could
> potentially add some failsafe logic in order to still support versionless
> plugins with the understanding that it would not be possible to colocate
> multiple versions of them on the same worker.
>
> Cheers,
>
> Chris
>
> On Thu, Jan 27, 2022 at 7:43 AM Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Hi Chris,
> >
> > Thanks for taking another look!
> >
> > That's a good point which I should have mentioned in the KIP. I think
> > we can have a PossiblyVersioned interface that extends Versioned and
> > has the default implementation. I'd like to also print a warning at
> > startup if a plugin does not override version(). Thanks for the
> > suggestion!
> >
> > I'm even considering whether PossiblyVersioned could be deprecated
> > immediately in order to be removed in the next major release. If so,
> > the warning message I mentioned would state that plugins not
> > overriding version() would stop working in the next major release.
> >
> > Thanks,
> > Mickael
> >
>
>

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Chris Egerton <ch...@confluent.io.INVALID>]

Hi Mickael,

+1 for the warning at startup, it'd be great if we could get everyone to
start versioning all of their components. Although given the current state
of WARN-level logging in Connect (see
https://issues.apache.org/jira/browse/KAFKA-7509) there might be too much
noise for most people to notice, unfortunately. Still, better than nothing!

I'm not sure it'd be wise to move quite so quickly with deprecating and
then removing the PossiblyVersioned interface. In an ideal world everyone
would update their converters, SMTs, and predicates as soon as they saw the
warning, but there could be some plugins out there that aren't maintained
anymore and would become incompatible with a Connect runtime that requires
them to provide versions. I can imagine there might be some fairly trivial
but valuable SMTs or converters that were written years ago and haven't
been touched in some time but are still useful to Connect users today.

I think it'd be fine to log a scary-looking warning message stating that
the plugin may be incompatible with future versions of Connect, but perhaps
we can leave the decision about how and when we actually create that
incompatibility to a future KIP. If we take the example use case of loading
multiple versions of the same plugin on the same worker, for example, that
feature would obviously require plugins to provide versions, but we could
potentially add some failsafe logic in order to still support versionless
plugins with the understanding that it would not be possible to colocate
multiple versions of them on the same worker.

Cheers,

Chris

On Thu, Jan 27, 2022 at 7:43 AM Mickael Maison <mi...@gmail.com>
wrote:

> Hi Chris,
>
> Thanks for taking another look!
>
> That's a good point which I should have mentioned in the KIP. I think
> we can have a PossiblyVersioned interface that extends Versioned and
> has the default implementation. I'd like to also print a warning at
> startup if a plugin does not override version(). Thanks for the
> suggestion!
>
> I'm even considering whether PossiblyVersioned could be deprecated
> immediately in order to be removed in the next major release. If so,
> the warning message I mentioned would state that plugins not
> overriding version() would stop working in the next major release.
>
> Thanks,
> Mickael
>
> On Mon, Jan 24, 2022 at 5:41 PM Mickael Maison <mi...@gmail.com>
> wrote:
> >
> > Hi Tom,
> >
> > Thanks for the feedback.
> >
> > You're right we don't have any contracts around the version field.
> > Like Chris I've only used the current field to find the exact source
> > code of connectors. Even if it's a limited use, I think that's enough
> > to justify adding it to all plugins and making them consistent. Work
> > for improving the contract can be done in a separate KIP.
> >
> > Thanks,
> > Mickael
> >
> > On Wed, Jan 19, 2022 at 7:55 PM Chris Egerton
> > <ch...@confluent.io.invalid> wrote:
> > >
> > > Hi all,
> > >
> > > With regards to Tom's question about possible uses of the version
> string,
> > > one of the most useful applications I've found for it so far has been
> > > debugging connector issues, especially based on captured logs. Using
> the
> > > version string to identify a git tag containing the exact source code
> for a
> > > connector, and then using that as a reference when poring over log
> files
> > > that usually contain class names, method names, and/or line numbers,
> is a
> > > great way to construct a timeline of events and understand how things
> may
> > > have gone wrong. I don't think we can require this as part of the
> contract
> > > for the version method, but at least identifying this use case in the
> docs
> > > for it could be useful to help connector/SMT/converter/etc. developers
> > > understand why they should care about what this method returns and how
> it
> > > could be beneficial to themselves and their users down the road.
> > >
> > > One other potential use case that comes up is that, if/when Connect
> > > supports loading multiple versions of the same plugin type on a single
> > > worker, the version string can be used to identify plugins that share
> the
> > > same name and type but which should be exposed individually to users.
> If we
> > > want to leave room for this possibility, perhaps part of the contract
> for
> > > the method can be that "any two releases of the same plugin but with
> > > different behavior should return different values from their version
> > > method".
> > >
> > > One final note--I've just realized that, with the introduction of a
> default
> > > implementation for the Versioned interface, we've made
> Connector::version
> > > and ConnectRestExtension::version optional to implement, whereas
> before it
> > > was required that instances of these plugins explicitly implement that
> > > method. Should we allow this? An alternative could be to add a new
> > > "PossiblyVersioned" interface (name obviously subject to change) that
> > > extends the current Versioned interface and contains a default
> > > implementation that returns the "undefined" string. The to-be-updated
> > > plugin interfaces could extend from this new "PossiblyVersioned"
> interface,
> > > and the plugin interfaces that extend from the current "Versioned"
> > > interface could be left unchanged.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Wed, Jan 19, 2022 at 10:04 AM Tom Bentley <tb...@redhat.com>
> wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > Thanks for the KIP. Overall I think this looks good.
> > > >
> > > > I have a small concern about the proliferation of Versioned.
> Currently this
> > > > interface has a really weak contract. There's no requirement for
> versions
> > > > to be comparable, or documentation about when versions should be
> changed by
> > > > authors of plugins which extend this interface. For example, if I
> release
> > > > version 1.2.3 of my SMT should the version be 1.2.3 or something
> else? Is
> > > > compatibility implied for different versions of my jar file which
> return
> > > > the same result from version()? Because of this it's not really
> clear what
> > > > the consumer of the version string is supposed to do with it. So I
> think if
> > > > we're going to be using Versioned more extensively it would be good
> to make
> > > > the expectations clearer, for both authors and people wanting to
> reason
> > > > about versions.
> > > >
> > > > Many thanks,
> > > >
> > > > Tom
> > > >
> > > > On Mon, 6 Dec 2021 at 09:56, Mickael Maison <
> mickael.maison@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Arjun,
> > > > >
> > > > > Thank you for taking a look at this KIP.
> > > > >
> > > > > 1.a Yes this is exactly what this KIP is proposing.
> > > > >
> > > > > 1.b I prefer returning all plugins together instead of having
> different
> > > > > paths.
> > > > >
> > > > > 2. While we seem to agree it would be nice to expose worker
> plugins, I
> > > > > don't think we've reached a consensus on the right way to do it.
> For
> > > > > that reason, I'm dropping this feature from this KIP. We can
> address
> > > > > this issue in another KIP. I have updated the KIP.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Mon, Dec 6, 2021 at 10:15 AM Mickael Maison <
> mickael.maison@gmail.com
> > > > >
> > > > > wrote:
> > > > > >
> > > > > > Thanks Randall, updated.
> > > > > >
> > > > > > On Sun, Dec 5, 2021 at 6:05 AM Arjun Satish <
> arjun.satish@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > hey folks,
> > > > > > >
> > > > > > > Thanks a lot for the KIP and the discussions. Here are a
> couple of
> > > > > thoughts
> > > > > > > (apologies if I missed some point in this thread).
> > > > > > >
> > > > > > > 1. it seems like we are changing the behavior of
> /connector-plugins
> > > > by
> > > > > now
> > > > > > > returning the list of transforms, converters as well? This
> would
> > > > break
> > > > > > > backward compatibility, and clients would need to track what
> version
> > > > of
> > > > > > > Connect workers they are querying to keep their experience
> > > > consistent.
> > > > > I
> > > > > > > wonder if we can keep the old behavior, with one of the
> following
> > > > > methods:
> > > > > > >
> > > > > > > 1a. We add an "expand" multi-query to the /connector-plugin
> endpoint
> > > > > like
> > > > > > > how it was done in KIP-465 [1]. This would preserve backward
> > > > > compatibility,
> > > > > > > as well as allow you to get consolidated output. Acceptable
> values
> > > > for
> > > > > the
> > > > > > > new param could be: "all" (to show all plugins), or
> > > > > > > "connectors|transforms|predicates" that show a subset of the
> plugin
> > > > > types.
> > > > > > >
> > > > > > > 1b. Query params might get crummy to extend. Alternatively, we
> could
> > > > > have
> > > > > > > an additional fine-grained paths for each plugin:
> > > > > > >
> > > > > > > # to list all the source and sink plugins
> > > > > > > /connector-plugins/connectors
> > > > > > >
> > > > > > > # to list all the SMT plugins
> > > > > > > /connector-plugins/transformations
> > > > > > >
> > > > > > > # to list all the predicate plugins
> > > > > > > /connector-plugins/predicates
> > > > > > >
> > > > > > > # to list all the converter plugins
> > > > > > > /connector-plugins/converters
> > > > > > >
> > > > > > > # to list all the header converter plugins
> > > > > > > /connector-plugins/header-converters
> > > > > > >
> > > > > > > This allows us to cleanly add new types of plugins down the
> road, and
> > > > > > > naturally extend to displaying attributes of these objects.
> Configs,
> > > > > for
> > > > > > > example, can be accessed with
> > > > > > >
> > > > > > >
> > > > >
> > > >
> http://localhost:8083/connector-plugins/transformations/Cast$Value/config
> > > > > > >
> > > > > > > A migration path (for both the suggestions above) would be to
> add a
> > > > > > > deprecation notice on the current behavior of
> /connector-plugins
> > > > > (where,
> > > > > > > after this PR is merged, both /connector-plugins and
> > > > > > > /connector-plugins/connectors return the same output). And
> starting
> > > > > from
> > > > > > > the next major version release, /connector-plugins redirects to
> > > > > > > /connector-plugins/connectors (making it super easy for
> clients to
> > > > > catch
> > > > > > > up).
> > > > > > >
> > > > > > > 2. Regarding /worker-plugins, I agree that it is useful to
> have the
> > > > > REST
> > > > > > > API inform the state of the REST extensions, and Config
> Providers.
> > > > > However,
> > > > > > > users of the Connect clusters will rarely need to know how a
> config
> > > > > > > provider or REST extension is configured to construct data
> pipelines.
> > > > > It's
> > > > > > > mostly an admin task, AFAICT. Also, we have also seen these
> problems
> > > > in
> > > > > > > Connect clusters where one host is somewhat differently set up
> than
> > > > > other
> > > > > > > (contents of plugin path, classpath, local worker configuration
> > > > files,
> > > > > jvm
> > > > > > > versions, GC settings) that surface as issues in Connector
> behavior.
> > > > > > > Overall, I'm wondering if we should just tackle this problem
> in a
> > > > > separate
> > > > > > > KIP, that evaluates all of these cases. All of this might
> nicely fall
> > > > > into
> > > > > > > under the /admin endpoint and allow us to expose detailed info
> on the
> > > > > state
> > > > > > > of the worker (maybe a /admin/worker/ that returns all of this
> info).
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-465%3A+Add+Consolidated+Connector+Endpoint+to+Connect+REST+API
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Dec 4, 2021 at 10:29 AM Randall Hauch <
> rhauch@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Thanks, Mickael. Can we call out these as literals? For
> example:
> > > > > > > >
> > > > > > > > > The response structure of the objects in the array remain
> > > > > unchanged, but
> > > > > > > > this proposal does add "converter", "header_converter" and
> > > > > "transformation"
> > > > > > > > to the allowed values for the "type" field, in addition to
> the
> > > > > existing
> > > > > > > > "sink" and "source" values.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > > Randall
> > > > > > > >
> > > > > > > > On Sat, Dec 4, 2021 at 4:58 AM Mickael Maison <
> > > > > mickael.maison@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Randall,
> > > > > > > > >
> > > > > > > > > Sure. The types are sink, source, converter,
> header_converter,
> > > > > > > > > transformation and predicate. I've updated the KIP.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Mickael
> > > > > > > > >
> > > > > > > > > On Fri, Dec 3, 2021 at 10:14 PM Randall Hauch <
> rhauch@gmail.com>
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > One more thing, inline below:
> > > > > > > > > >
> > > > > > > > > > Randall
> > > > > > > > > >
> > > > > > > > > > On Fri, Dec 3, 2021 at 10:38 AM Mickael Maison <
> > > > > > > > mickael.maison@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Thanks Randall for the feedback!
> > > > > > > > > > >
> > > > > > > > > > > 1. Done
> > > > > > > > > > >
> > > > > > > > > > > 2. Yes that makes sense. I've updated the KIP
> > > > > > > > > > >
> > > > > > > > > > > 3.a I've updated to explicitly mention the plugin can
> be
> > > > > provided
> > > > > > > > > > > using the fully qualified class or the alias.
> > > > > > > > > > > 3.b Only concrete classes will be returned. I made the
> > > > mistake
> > > > > when I
> > > > > > > > > > > crafted the examples by hand. Fixed
> > > > > > > > > >
> > > > > > > > > > Can we also enumerate all of the values that can be
> included in
> > > > > the
> > > > > > > > > > "type" field in the response from the "GET
> > > > > > > > > > /connector-plugins?connectorsOnly=false"?
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 4. If a plugin does not override config(), the
> response is an
> > > > > empty
> > > > > > > > > > > array. I've updated the KIP
> > > > > > > > > > >
> > > > > > > > > > > 5. I have updated the compatibility section with more
> details
> > > > > > > > > > >
> > > > > > > > > > > 6. Can you clarify what you mean? KIP-507 was for
> securing
> > > > > internal
> > > > > > > > > > > REST endpoints.
> > > > > > > > > > >
> > > > > > > > > > > 7. I don't think it is necessary. I think the use case
> Chris
> > > > > > > > mentioned
> > > > > > > > > > > could be useful for admins. I'm interested in exposing
> the
> > > > > > > > > > > connector-level plugins and their configs. I thought
> it was
> > > > > easy to
> > > > > > > > > > > add an API for worker plugins while working in this
> area but
> > > > > if there
> > > > > > > > > > > isn't a strong consensus, I'm happy to drop it.
> > > > > > > > > > >
> > > > > > > > > > > Mickael
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch <
> > > > rhauch@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the KIP, Mickael. And thanks to everyone
> else
> > > > for
> > > > > the
> > > > > > > > good
> > > > > > > > > > > > discussion so far.
> > > > > > > > > > > >
> > > > > > > > > > > > Overall this is pretty good, but I do have a few
> > > > > > > > questions/comments:
> > > > > > > > > > > > 1. Can we include the HTTP method/verb in each of the
> > > > > endpoints in
> > > > > > > > the
> > > > > > > > > > > > "Public Interfaces" section?
> > > > > > > > > > > > 2. The sample output for `GET
> > > > > > > > /connector-plugins?connectorsOnly=false`
> > > > > > > > > > > > includes a `version` field for the connector plugin,
> but
> > > > not
> > > > > for
> > > > > > > > the
> > > > > > > > > > > > transform or converter plugins. Neither the
> > > > `Transformation`
> > > > > or
> > > > > > > > > > > > `Converter` interfaces have a version nor implement
> the
> > > > > `Versioned`
> > > > > > > > > > > > interface, so should we add the `Versioned`
> interface to
> > > > the
> > > > > > > > > > > > `Transformation` and `Converter` interfaces and add a
> > > > default
> > > > > > > > method
> > > > > > > > > > > > that returns "unknown"? Yes, we could do that in a
> separate
> > > > > KIP,
> > > > > > > > but I
> > > > > > > > > > > > think it's appropriate to do it in this KIP for a few
> > > > > reasons.
> > > > > > > > First,
> > > > > > > > > > > > we're already modifying the `Converter` interface in
> this
> > > > > KIP and
> > > > > > > > it
> > > > > > > > > > > > would be quite straightforward to also extend the
> > > > `Versioned`
> > > > > > > > > > > > interface and add a default method; then doing that
> for
> > > > > > > > > > > > `Transformation` would also be relatively
> straightforward.
> > > > > Second,
> > > > > > > > > > > > this KIP provides the justification for adding a
> version to
> > > > > the
> > > > > > > > > > > > `Transformation` and `Converter` interfaces.
> > > > > > > > > > > > 3. In the "Public Interfaces" section, there are two
> > > > > > > > inconsistencies
> > > > > > > > > > > > between the `GET /connector-plugins` and `GET
> > > > > > > > > > > > /connector-plugins/<plugin>/config` methods. Both
> will make
> > > > > it
> > > > > > > > > > > > difficult to use the second method, since the input
> to the
> > > > > second
> > > > > > > > > > > > method is not directly found in the response from
> the first
> > > > > method:
> > > > > > > > > > > >  a. The example for `GET
> > > > > /connector-plugins/Cast$Value/config` uses
> > > > > > > > > > > > short name, or what Connect often refers to as the
> alias,
> > > > > > > > "Cast$Value"
> > > > > > > > > > > > for the plugin, rather than the fully-qualified name
> > > > > > > > > > > > "org.apache.kafka.connect.transforms.Cast$Value".
> The KIP
> > > > > should
> > > > > > > > > > > > specify whether it will support both the shortened
> name and
> > > > > the
> > > > > > > > > > > > fully-qualified name, or whether only one of these
> will be
> > > > > > > > supported.
> > > > > > > > > > > > Not supporting the fully-qualified name will make it
> > > > > impossible to
> > > > > > > > use
> > > > > > > > > > > > the response from the first method as an input to the
> > > > second.
> > > > > > > > > > > >  a. The example for the list method returns the
> > > > > `Transformation`
> > > > > > > > base
> > > > > > > > > > > > and abstract classes (e.g.,
> > > > > > > > > > > > "org.apache.kafka.connect.transforms.Cast") rather
> than the
> > > > > > > > concrete
> > > > > > > > > > > > classes (e.g.,
> > > > > "org.apache.kafka.connect.transforms.Cast$Value" and
> > > > > > > > > > > > "org.apache.kafka.connect.transforms.Cast$Key"),
> while the
> > > > > `GET
> > > > > > > > > > > > /connector-plugins/<plugin>/config` will require the
> > > > concrete
> > > > > > > > class.
> > > > > > > > > > > > 4. The "Public Interfaces" section's description of
> the
> > > > `GET
> > > > > > > > > > > > /connector-plugins/<plugin>/config` method should
> probably
> > > > > specify
> > > > > > > > the
> > > > > > > > > > > > expected behavior for `Converter` implementations
> that
> > > > don't
> > > > > > > > override
> > > > > > > > > > > > the new `config()` method.
> > > > > > > > > > > > 5. The "Compatibility, Deprecation, and Migration
> Plan"
> > > > > section
> > > > > > > > should
> > > > > > > > > > > > describe whether older `Converter` implementations
> > > > (compiled
> > > > > with
> > > > > > > > > > > > older versions of the Connect API without Converter
> > > > extending
> > > > > > > > > > > > Configurable) will be able to be installed into newer
> > > > Connect
> > > > > > > > > > > > runtimes, and what the behavior will be for the `GET
> > > > > > > > > > > > /connector-plugins/<plugin>/config` method.
> (Although my
> > > > > suggestion
> > > > > > > > > > > > for #5 will partially address the latter, we still
> need to
> > > > > talk
> > > > > > > > about
> > > > > > > > > > > > it in terms of compatibility.)
> > > > > > > > > > > > 6. Can we add a small section addressing the security
> > > > > implications
> > > > > > > > of
> > > > > > > > > > > > these new APIs? KIP-507 [1] applied to APIs that
> existed at
> > > > > the
> > > > > > > > time,
> > > > > > > > > > > > and this KIP should at least mention how it aligns
> with
> > > > that
> > > > > > > > > > > > mechanism.
> > > > > > > > > > > > 7. Do we really think adding `GET /worker-plugins` is
> > > > > necessary?
> > > > > > > > While
> > > > > > > > > > > > I understand the reasoning, I'm less convinced that
> it's
> > > > > worth
> > > > > > > > adding
> > > > > > > > > > > > to the public API. First, an administrator still has
> to
> > > > > modify the
> > > > > > > > > > > > worker config for all workers to take advantage of
> that.
> > > > > Second,
> > > > > > > > > > > > adding this method increases the surface area and
> therefore
> > > > > risk of
> > > > > > > > > > > > additional attack vectors, despite not really being
> able to
> > > > > use the
> > > > > > > > > > > > resulting information in any other part of the API.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > >
> > > > > > > > > > > > Randall
> > > > > > > > > > > >
> > > > > > > > > > > > [1]
> > > > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Dec 1, 2021 at 7:52 AM Mickael Maison <
> > > > > > > > mickael.maison@gmail.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks again for the feedback! I've updated the
> KIP based
> > > > > on our
> > > > > > > > last
> > > > > > > > > > > > > discussions. I've decided to include the new
> endpoint for
> > > > > worker
> > > > > > > > > > > > > plugins.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. Yes I agree, it's best to gate the new behavior.
> > > > > > > > > > > > > 2. Yes, it was a remnant from the original
> proposal. I've
> > > > > now
> > > > > > > > removed
> > > > > > > > > > > > > the location field.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Mickael
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
> > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think that's a great idea! I especially like
> how we
> > > > can
> > > > > > > > establish the
> > > > > > > > > > > > > > expectation that any plugin type that appears in
> the
> > > > > response
> > > > > > > > from the GET
> > > > > > > > > > > > > > /connector-plugins endpoint will have a
> corresponding
> > > > GET
> > > > > > > > > > > > > > /connector-plugins/<type>/config endpoint, but
> (if we
> > > > > decide
> > > > > > > > to add them in
> > > > > > > > > > > > > > the future), worker plugins won't be expected to
> expose
> > > > > this
> > > > > > > > kind of
> > > > > > > > > > > > > > information and the different root path helps
> give a
> > > > > decent
> > > > > > > > hint about this.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I also like the choice to return an empty
> ConfigDef
> > > > from
> > > > > > > > Converter::config
> > > > > > > > > > > > > > instead of null.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Two things come to mind:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. We may want to gate this behind a URL query
> > > > parameter
> > > > > > > > (maybe something
> > > > > > > > > > > > > > like "connectorsOnly") that defaults to the old
> > > > behavior
> > > > > in
> > > > > > > > order to avoid
> > > > > > > > > > > > > > breaking existing tools such as programmatic UIs
> that
> > > > > use the
> > > > > > > > endpoint
> > > > > > > > > > > > > > today to discover the connectors that can be
> created by
> > > > > the
> > > > > > > > user. We can
> > > > > > > > > > > > > > even plan to change the default for that
> parameter to
> > > > the
> > > > > > > > newly-proposed
> > > > > > > > > > > > > > behavior in the next major release, which should
> give
> > > > > people
> > > > > > > > enough time to
> > > > > > > > > > > > > > either adapt to the expanded response format or
> add the
> > > > > query
> > > > > > > > parameter to
> > > > > > > > > > > > > > their tooling.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. The existing GET /connector-plugins endpoint
> doesn't
> > > > > > > > contain information
> > > > > > > > > > > > > > on the location of the plugin on the worker's
> file
> > > > > system. Do
> > > > > > > > you think we
> > > > > > > > > > > > > > should still include this info in the new
> response
> > > > > format?
> > > > > > > > Correct me if
> > > > > > > > > > > > > > I'm wrong but it seems it may have been proposed
> > > > > originally to
> > > > > > > > help prevent
> > > > > > > > > > > > > > already-addressed bugs in Connect classloading
> from
> > > > > striking;
> > > > > > > > all else
> > > > > > > > > > > > > > equal, I'd personally err on the side of leaving
> this
> > > > > info out
> > > > > > > > or at least
> > > > > > > > > > > > > > reducing permitted values for it to just
> "classpath" or
> > > > > > > > "plugin path" in
> > > > > > > > > > > > > > order to avoid leaking worker file paths into
> the REST
> > > > > API,
> > > > > > > > which might
> > > > > > > > > > > > > > bother super security-conscious users.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <
> > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Yes to keep compatibility we want a default
> > > > > implementation
> > > > > > > > for
> > > > > > > > > > > > > > > Converter.configs(), I've updated the KIP.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regarding worker plugins, the use case you
> described
> > > > > seems
> > > > > > > > valuable.
> > > > > > > > > > > > > > > I'd prefer not mixing worker and connector
> plugins on
> > > > > the
> > > > > > > > same
> > > > > > > > > > > > > > > endpoint but I agree using /plugins and
> > > > /worker-plugins
> > > > > > > > could be
> > > > > > > > > > > > > > > confusing.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > One alternative is to expose all
> connector-level
> > > > > plugins via
> > > > > > > > the
> > > > > > > > > > > > > > > existing /connector-plugins endpoint. In that
> case,
> > > > > we'd
> > > > > > > > need to keep
> > > > > > > > > > > > > > > the current JSON schema and not group plugins
> by
> > > > type.
> > > > > As
> > > > > > > > the current
> > > > > > > > > > > > > > > schema already has a type field for each
> entry, we'll
> > > > > still
> > > > > > > > be able to
> > > > > > > > > > > > > > > tell them apart. Then we can have
> /worker-plugins
> > > > and a
> > > > > > > > relatively
> > > > > > > > > > > > > > > clean API. What do you think?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Mickael
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > > > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think one potential use case for exposing
> > > > > worker-level
> > > > > > > > plugins is that
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > may make it easier to determine whether a
> worker is
> > > > > set up
> > > > > > > > correctly (the
> > > > > > > > > > > > > > > > current alternative requires looking through
> log
> > > > > files and
> > > > > > > > can be a
> > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > tedious), and might even make it possible to
> > > > > automatically
> > > > > > > > identify
> > > > > > > > > > > > > > > > discrepancies within a cluster by diffing the
> > > > > contents of
> > > > > > > > that endpoint
> > > > > > > > > > > > > > > > across each worker. But I don't think this
> has to
> > > > be
> > > > > > > > addressed by the
> > > > > > > > > > > > > > > > current KIP; the only thing that bothers me a
> > > > little
> > > > > is
> > > > > > > > that "plugins" is
> > > > > > > > > > > > > > > > generic and it may confuse people down the
> road if
> > > > > we add
> > > > > > > > an endpoint for
> > > > > > > > > > > > > > > > worker-level plugins ("why is one just called
> > > > > 'plugins'
> > > > > > > > and the other one
> > > > > > > > > > > > > > > > is 'worker-plugins'?"). Probably not worth
> blocking
> > > > > on,
> > > > > > > > though.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Agreed that the suggestion for improved
> validation
> > > > > should
> > > > > > > > be made on the
> > > > > > > > > > > > > > > > KIP-802 thread.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I also noticed that the newly-proposed config
> > > > method
> > > > > for
> > > > > > > > the Converter
> > > > > > > > > > > > > > > > interface doesn't have a default
> implementation,
> > > > > making it
> > > > > > > > > > > > > > > > backwards-incompatible. Should we add a
> default
> > > > > > > > implementation that
> > > > > > > > > > > > > > > returns
> > > > > > > > > > > > > > > > either null or an empty ConfigDef?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Nov 26, 2021 at 8:35 AM Mickael
> Maison <
> > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. If we want to expose worker plugins, I
> think
> > > > we
> > > > > > > > should do it via a
> > > > > > > > > > > > > > > > > separate endpoint. But to be honest, I'm
> not even
> > > > > sure I
> > > > > > > > see strong
> > > > > > > > > > > > > > > > > use cases for exposing them as they are
> either
> > > > > enabled
> > > > > > > > or not and
> > > > > > > > > > > > > > > > > can't be changed at runtime. So I'd prefer
> to
> > > > > stick to
> > > > > > > > "connector
> > > > > > > > > > > > > > > > > level" plugins in this KIP. Let me now if
> you
> > > > have
> > > > > use
> > > > > > > > cases, I'm open
> > > > > > > > > > > > > > > > > to reconsider this choice.
> > > > > > > > > > > > > > > > > I'll add that in the rejected alternatives
> > > > section
> > > > > for
> > > > > > > > now
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2. I remembered seeing issues in the past
> with
> > > > > multiple
> > > > > > > > plugin.path
> > > > > > > > > > > > > > > > > entries but I tried today and I was able
> to mix
> > > > and
> > > > > > > > match plugins from
> > > > > > > > > > > > > > > > > different paths. So my bad for getting
> confused.
> > > > > > > > > > > > > > > > > Then I agree, it makes more sense to group
> them
> > > > by
> > > > > > > > plugin type.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 3. Yes this should be covered in KIP-802:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 4. No particular reason. We can support
> both
> > > > > formats
> > > > > > > > like today. I've
> > > > > > > > > > > > > > > > > updated the KIP
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Mickael
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Nov 23, 2021 at 6:40 PM Chris
> Egerton
> > > > > > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think the increase in scope here is
> great and
> > > > > the
> > > > > > > > added value
> > > > > > > > > > > > > > > certainly
> > > > > > > > > > > > > > > > > > justifies the proposed changes. I have
> some
> > > > > thoughts
> > > > > > > > but overall I
> > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > direction this is going in now.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. The new /plugins endpoint is
> described as
> > > > > > > > containing "all plugins
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > are Connectors, Transformations,
> Converters,
> > > > > > > > HeaderConverters and
> > > > > > > > > > > > > > > > > > Predicates". So essentially, it looks
> like we
> > > > > want to
> > > > > > > > expose all
> > > > > > > > > > > > > > > plugins
> > > > > > > > > > > > > > > > > > that are configured on a per-connector
> basis,
> > > > but
> > > > > > > > exclude plugins
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > configured on a per-worker basis (such as
> > > > config
> > > > > > > > providers and REST
> > > > > > > > > > > > > > > > > > extensions). Do you think it may be
> valuable to
> > > > > expose
> > > > > > > > information on
> > > > > > > > > > > > > > > > > > worker-level plugins as well?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2. The description for the new /plugins
> > > > endpoint
> > > > > also
> > > > > > > > states that
> > > > > > > > > > > > > > > > > "Plugins
> > > > > > > > > > > > > > > > > > will be grouped by plugin.path. This
> will make
> > > > it
> > > > > > > > clear to users
> > > > > > > > > > > > > > > what's
> > > > > > > > > > > > > > > > > > available to use as it's not possible to
> use a
> > > > > > > > Connector from one
> > > > > > > > > > > > > > > path
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > Transformations from another.". Is this
> true? I
> > > > > > > > thought that
> > > > > > > > > > > > > > > Connect's
> > > > > > > > > > > > > > > > > > classloading made it possible to package
> > > > > > > > > > > > > > > > > > converters/transformations/predicates
> > > > completely
> > > > > > > > independently from
> > > > > > > > > > > > > > > each
> > > > > > > > > > > > > > > > > > other, and to reference them from
> > > > > > > > also-independently-packaged
> > > > > > > > > > > > > > > connectors.
> > > > > > > > > > > > > > > > > > If it turns out that this is the case,
> could we
> > > > > > > > consider
> > > > > > > > > > > > > > > restructuring
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > response to be grouped by plugin type
> instead
> > > > of
> > > > > by
> > > > > > > > classloader?
> > > > > > > > > > > > > > > There's
> > > > > > > > > > > > > > > > > > also the ungrouped format proposed in
> KIP-494 (
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > which we might consider as well.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 3. I think this can be left for a
> follow-up KIP
> > > > > if
> > > > > > > > necessary, but I'm
> > > > > > > > > > > > > > > > > > curious about your thoughts on adding new
> > > > > validate
> > > > > > > > methods to all
> > > > > > > > > > > > > > > > > > connector-level plugins that can be used
> > > > > similarly to
> > > > > > > > how the
> > > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > > Connector::validate method (
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > >
> > > > >
> > > >
> https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > > is used. This would allow for plugins to
> > > > perform
> > > > > > > > validation that's
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > sophisticated than what the ConfigDef is
> > > > capable
> > > > > of,
> > > > > > > > such as
> > > > > > > > > > > > > > > validating
> > > > > > > > > > > > > > > > > > combinations of properties like a
> hostname and
> > > > > > > > credentials for
> > > > > > > > > > > > > > > reaching
> > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > I know that at least Confluent's Avro,
> > > > protobuf,
> > > > > and
> > > > > > > > JSON schema
> > > > > > > > > > > > > > > > > converters
> > > > > > > > > > > > > > > > > > would benefit from this kind of feature.
> It's a
> > > > > little
> > > > > > > > tangential to
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > KIP (which at the moment is about
> discovering
> > > > > plugins
> > > > > > > > and their
> > > > > > > > > > > > > > > > > > configuration surfaces, as opposed to
> > > > validating
> > > > > > > > them), but I
> > > > > > > > > > > > > > > figured I'd
> > > > > > > > > > > > > > > > > > ask since we're going to be expanding the
> > > > > Converter
> > > > > > > > interface and it
> > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > useful to tackle this while we're in the
> > > > > neighborhood.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 4. The description for the new
> > > > > > > > /plugins/<type>/<name>/configdef
> > > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > > states that "Name must be the fully
> qualified
> > > > > class
> > > > > > > > name of the
> > > > > > > > > > > > > > > plugin".
> > > > > > > > > > > > > > > > > > Any reason not to also support aliases
> (e.g.,
> > > > > > > > > > > > > > > "FileStreamSinkConnector"
> > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > "FileStreamSink" instead of
> > > > > > > > > > > > > > > > > >
> > > > > > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael
> > > > Maison <
> > > > > > > > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks all for the feedback!
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Chris,
> > > > > > > > > > > > > > > > > > > I agree that fixing the current
> endpoint
> > > > helps
> > > > > a
> > > > > > > > lot. Thanks for
> > > > > > > > > > > > > > > > > > > raising these JIRAs and submitting a
> PR!
> > > > > > > > > > > > > > > > > > > However thinking about the issue
> further, I
> > > > > decided
> > > > > > > > to expand the
> > > > > > > > > > > > > > > > > > > scope of the KIP to cover all
> user-visible
> > > > > plugins.
> > > > > > > > > > > > > > > > > > > In practice, users want to know about
> all
> > > > > available
> > > > > > > > plugins not
> > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > > connectors. This includes
> transformations,
> > > > > > > > converters,
> > > > > > > > > > > > > > > > > > > header_converters and predicates. As
> we also
> > > > > want to
> > > > > > > > retrieve
> > > > > > > > > > > > > > > > > > > configdef for these too, I think it
> makes
> > > > > sense to
> > > > > > > > introduce a new
> > > > > > > > > > > > > > > > > > > endpoint to do so. Alongside we
> obviously
> > > > need
> > > > > a new
> > > > > > > > endpoint for
> > > > > > > > > > > > > > > > > > > listing all plugins.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Gunnar,
> > > > > > > > > > > > > > > > > > > I took a look at exposing valid values
> via
> > > > the
> > > > > API.
> > > > > > > > I think the
> > > > > > > > > > > > > > > issue
> > > > > > > > > > > > > > > > > > > is that Validators don't expose a way
> to
> > > > > retrieve
> > > > > > > > valid values.
> > > > > > > > > > > > > > > > > > > Changing validators will have an
> impact on
> > > > all
> > > > > > > > components so I'd
> > > > > > > > > > > > > > > > > > > prefer to address this requirement in a
> > > > > separate
> > > > > > > > KIP. I agree this
> > > > > > > > > > > > > > > > > > > would be an interesting improvement
> and I'd
> > > > > happy to
> > > > > > > > write a KIP
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > it too.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I have updated the KIP accordingly.
> Let me
> > > > > know if
> > > > > > > > you have further
> > > > > > > > > > > > > > > > > > > feedback.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > Mickael
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar
> > > > Morling
> > > > > > > > > > > > > > > > > > > <gunnar.morling@googlemail.com.invalid
> >
> > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I'm +1 for adding a GET endpoint for
> > > > > obtaining
> > > > > > > > config
> > > > > > > > > > > > > > > definitions. It
> > > > > > > > > > > > > > > > > > > > always felt odd to me that one has
> to issue
> > > > > a PUT
> > > > > > > > for that
> > > > > > > > > > > > > > > purpose.
> > > > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > > > > nothing else, it'd be better in
> terms of
> > > > > > > > discoverability of the
> > > > > > > > > > > > > > > KC
> > > > > > > > > > > > > > > > > REST
> > > > > > > > > > > > > > > > > > > API.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > One additional feature request I'd
> have is
> > > > to
> > > > > > > > expose the valid
> > > > > > > > > > > > > > > enum
> > > > > > > > > > > > > > > > > > > > constants for enum-typed options.
> That'll
> > > > > help to
> > > > > > > > display the
> > > > > > > > > > > > > > > values
> > > > > > > > > > > > > > > > > in a
> > > > > > > > > > > > > > > > > > > > drop-down or via radio buttons in a
> UI,
> > > > give
> > > > > us
> > > > > > > > tab completion in
> > > > > > > > > > > > > > > > > kcctl,
> > > > > > > > > > > > > > > > > > > > etc.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > --Gunnar
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr
> schrieb
> > > > > Chris
> > > > > > > > Egerton
> > > > > > > > > > > > > > > > > > > > <ch...@confluent.io.invalid>:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Viktor,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > It sounds like there are three
> major
> > > > > points here
> > > > > > > > in favor of a
> > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > GET
> > > > > > > > > > > > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. You cannot issue a blank
> ("dummy")
> > > > > request
> > > > > > > > for sink
> > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > because a
> > > > > > > > > > > > > > > > > > > > > topic list/topic regex has to be
> supplied
> > > > > > > > (otherwise the PUT
> > > > > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > > > > > returns a 500 response)
> > > > > > > > > > > > > > > > > > > > > 2. A dummy request still triggers
> custom
> > > > > > > > validations by the
> > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > > which may be best to avoid if we
> know for
> > > > > sure
> > > > > > > > that the config
> > > > > > > > > > > > > > > > > isn't
> > > > > > > > > > > > > > > > > > > worth
> > > > > > > > > > > > > > > > > > > > > validating yet
> > > > > > > > > > > > > > > > > > > > > 3. It's more ergonomic and
> intuitive to
> > > > be
> > > > > able
> > > > > > > > to issue a GET
> > > > > > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > > > > > without having to give a dummy
> connector
> > > > > config
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > With regards to 1, this is
> actually a bug
> > > > > in
> > > > > > > > Connect (
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a
> fix
> > > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > > > > implemented and awaiting committer
> > > > review (
> > > > > > > > > > > > > > > > > > > > >
> > > > https://github.com/apache/kafka/pull/11369).
> > > > > I
> > > > > > > > think it'd be
> > > > > > > > > > > > > > > > > better to
> > > > > > > > > > > > > > > > > > > > > focus on fixing this bug in general
> > > > > instead of
> > > > > > > > implementing a
> > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > REST
> > > > > > > > > > > > > > > > > > > > > endpoint in order to allow people
> to work
> > > > > around
> > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > With regards to 2, this is
> technically
> > > > > possible
> > > > > > > > but I'm unsure
> > > > > > > > > > > > > > > > > it'd be
> > > > > > > > > > > > > > > > > > > too
> > > > > > > > > > > > > > > > > > > > > common out in the wild given that
> most
> > > > > > > > validations that could
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > expensive
> > > > > > > > > > > > > > > > > > > > > would involve things like
> connecting to a
> > > > > > > > database, checking
> > > > > > > > > > > > > > > if a
> > > > > > > > > > > > > > > > > cloud
> > > > > > > > > > > > > > > > > > > > > storage bucket exists, etc., none
> of
> > > > which
> > > > > are
> > > > > > > > possible without
> > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > configuration properties from the
> user
> > > > (db
> > > > > > > > hostname, bucket
> > > > > > > > > > > > > > > name,
> > > > > > > > > > > > > > > > > > > etc.).
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > With regards to 3, I do agree that
> it'd
> > > > be
> > > > > > > > easier for people
> > > > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > > > UIs
> > > > > > > > > > > > > > > > > > > > > to have a GET API to work against.
> I'm
> > > > > just not
> > > > > > > > sure it's
> > > > > > > > > > > > > > > worth the
> > > > > > > > > > > > > > > > > > > > > additional implementation,
> testing, and
> > > > > > > > maintenance burden. If
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > > > > possible to issue a PUT request
> without
> > > > > > > > unexpected 500s for
> > > > > > > > > > > > > > > invalid
> > > > > > > > > > > > > > > > > > > > > configs, would that suffice?
> AFAICT it'd
> > > > > > > > basically be as
> > > > > > > > > > > > > > > simple as
> > > > > > > > > > > > > > > > > > > issuing
> > > > > > > > > > > > > > > > > > > > > a PUT request with a dummy body
> > > > consisting
> > > > > of
> > > > > > > > nothing except
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > class (which at this point we
> might even
> > > > > make
> > > > > > > > unnecessary and
> > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > > > > automatically replace with the
> connector
> > > > > class
> > > > > > > > from the URL)
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > then
> > > > > > > > > > > > > > > > > > > > > filtering the response to just
> grab the
> > > > > > > > "definition" field of
> > > > > > > > > > > > > > > each
> > > > > > > > > > > > > > > > > > > element
> > > > > > > > > > > > > > > > > > > > > in the "configs" array in the
> response.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM
> Viktor
> > > > > > > > Somogyi-Vass <
> > > > > > > > > > > > > > > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Folks,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I too think this would be a very
> useful
> > > > > > > > feature. Some of our
> > > > > > > > > > > > > > > > > > > management
> > > > > > > > > > > > > > > > > > > > > > applications would provide a
> wizard for
> > > > > > > > creating connectors.
> > > > > > > > > > > > > > > In
> > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > scenario the user basically
> would fill
> > > > > out a
> > > > > > > > sample
> > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > generated
> > > > > > > > > > > > > > > > > > > > > > by the UI which would send it
> back to
> > > > > Connect
> > > > > > > > for validation
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > eventually
> > > > > > > > > > > > > > > > > > > > > > create a new connector. The
> first part
> > > > > of this
> > > > > > > > workflow can
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > enhanced
> > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > we had an API that can return the
> > > > > > > > configuration definition
> > > > > > > > > > > > > > > of the
> > > > > > > > > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > > > > > > type of connector as the UI
> application
> > > > > would
> > > > > > > > be able to
> > > > > > > > > > > > > > > > > generate a
> > > > > > > > > > > > > > > > > > > > > sample
> > > > > > > > > > > > > > > > > > > > > > for the user based on that
> (nicely
> > > > drawn
> > > > > > > > diagram:
> > > > > > > > > > > > > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > connector-plugins/{connectorType}/config/validate API
> > > > > > > > > > > > > > > > > essentially
> > > > > > > > > > > > > > > > > > > > > works
> > > > > > > > > > > > > > > > > > > > > > and returns the data that we
> need,
> > > > > however it
> > > > > > > > is a HTTP PUT
> > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > > > > > bit unintuitive for a fetch-like
> > > > > functionality
> > > > > > > > and also
> > > > > > > > > > > > > > > > > functionally
> > > > > > > > > > > > > > > > > > > > > > different as it validates the
> given
> > > > > (dummy)
> > > > > > > > request. In case
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > sink
> > > > > > > > > > > > > > > > > > > > > > connectors one would need to also
> > > > > provide a
> > > > > > > > topic name.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > A suggestion for the KIP: I
> think it
> > > > can
> > > > > be
> > > > > > > > useful to return
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > > > groups and the connector class'
> name
> > > > > similarly
> > > > > > > > to the
> > > > > > > > > > > > > > > validate
> > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > case any frontend needs them
> (and also
> > > > > the
> > > > > > > > response would be
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > validate API but simpler).
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Viktor
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM
> Ryanne
> > > > > Dolan <
> > > > > > > > > > > > > > > > > ryannedolan@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think it'd be worth adding a
> GET
> > > > > version,
> > > > > > > > fwiw. Could be
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > > > > > handler
> > > > > > > > > > > > > > > > > > > > > > > with just a different spelling
> maybe.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM
> Mickael
> > > > > Maison
> > > > > > > > <
> > > > > > > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > You're right, you can
> achieve the
> > > > > same
> > > > > > > > functionality
> > > > > > > > > > > > > > > using
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > > > > > > > > > > > > In my mind it was only for
> > > > > validation once
> > > > > > > > you have
> > > > > > > > > > > > > > > build a
> > > > > > > > > > > > > > > > > > > > > > > > configuration but when used
> with an
> > > > > empty
> > > > > > > > configuration,
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > basically
> > > > > > > > > > > > > > > > > > > > > > > > serves the same purpose as
> the
> > > > > proposed
> > > > > > > > new endpoint.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I think it's a bit easier to
> use a
> > > > > GET
> > > > > > > > endpoint but I
> > > > > > > > > > > > > > > don't
> > > > > > > > > > > > > > > > > > > think it
> > > > > > > > > > > > > > > > > > > > > > > > really warrants a different
> > > > endpoint.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56
> PM
> > > > Chris
> > > > > > > > Egerton
> > > > > > > > > > > > > > > > > > > > > > > > <chrise@confluent.io.invalid
> >
> > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I'm wondering about the
> use case
> > > > > here.
> > > > > > > > The motivation
> > > > > > > > > > > > > > > > > section
> > > > > > > > > > > > > > > > > > > > > states
> > > > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > > "Connect does not provide
> a way
> > > > to
> > > > > see
> > > > > > > > what
> > > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > requires. Instead users
> have to
> > > > go
> > > > > look
> > > > > > > > at the
> > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > documentation
> > > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > > in the worst case, look
> directly
> > > > > at the
> > > > > > > > connector
> > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > code.",
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > > with this KIP, "users will
> be
> > > > able
> > > > > to
> > > > > > > > discover the
> > > > > > > > > > > > > > > required
> > > > > > > > > > > > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > > > > > > > > > > > for connectors installed
> in a
> > > > > Connect
> > > > > > > > cluster" and
> > > > > > > > > > > > > > > "tools
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > generate wizards for
> configuring
> > > > > and
> > > > > > > > starting
> > > > > > > > > > > > > > > connectors".
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > > > > > > > > > > > > endpoint not address these
> > > > points?
> > > > > What
> > > > > > > > will the
> > > > > > > > > > > > > > > > > newly-proposed
> > > > > > > > > > > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > > > > > > > > > allow users to do that
> they will
> > > > > not
> > > > > > > > already be able
> > > > > > > > > > > > > > > to do
> > > > > > > > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at
> 9:20 AM
> > > > > Mickael
> > > > > > > > Maison <
> > > > > > > > > > > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I've created KIP-769 to
> expose
> > > > > > > > connector
> > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > definitions
> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Please take a look and
> let me
> > > > > know if
> > > > > > > > you have any
> > > > > > > > > > > > > > > > > feedback.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > >
> > > > >
> > > > >
> > > >
>

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Mickael Maison <mi...@gmail.com>]

Hi Chris,

Thanks for taking another look!

That's a good point which I should have mentioned in the KIP. I think
we can have a PossiblyVersioned interface that extends Versioned and
has the default implementation. I'd like to also print a warning at
startup if a plugin does not override version(). Thanks for the
suggestion!

I'm even considering whether PossiblyVersioned could be deprecated
immediately in order to be removed in the next major release. If so,
the warning message I mentioned would state that plugins not
overriding version() would stop working in the next major release.

Thanks,
Mickael

On Mon, Jan 24, 2022 at 5:41 PM Mickael Maison <mi...@gmail.com> wrote:
>
> Hi Tom,
>
> Thanks for the feedback.
>
> You're right we don't have any contracts around the version field.
> Like Chris I've only used the current field to find the exact source
> code of connectors. Even if it's a limited use, I think that's enough
> to justify adding it to all plugins and making them consistent. Work
> for improving the contract can be done in a separate KIP.
>
> Thanks,
> Mickael
>
> On Wed, Jan 19, 2022 at 7:55 PM Chris Egerton
> <ch...@confluent.io.invalid> wrote:
> >
> > Hi all,
> >
> > With regards to Tom's question about possible uses of the version string,
> > one of the most useful applications I've found for it so far has been
> > debugging connector issues, especially based on captured logs. Using the
> > version string to identify a git tag containing the exact source code for a
> > connector, and then using that as a reference when poring over log files
> > that usually contain class names, method names, and/or line numbers, is a
> > great way to construct a timeline of events and understand how things may
> > have gone wrong. I don't think we can require this as part of the contract
> > for the version method, but at least identifying this use case in the docs
> > for it could be useful to help connector/SMT/converter/etc. developers
> > understand why they should care about what this method returns and how it
> > could be beneficial to themselves and their users down the road.
> >
> > One other potential use case that comes up is that, if/when Connect
> > supports loading multiple versions of the same plugin type on a single
> > worker, the version string can be used to identify plugins that share the
> > same name and type but which should be exposed individually to users. If we
> > want to leave room for this possibility, perhaps part of the contract for
> > the method can be that "any two releases of the same plugin but with
> > different behavior should return different values from their version
> > method".
> >
> > One final note--I've just realized that, with the introduction of a default
> > implementation for the Versioned interface, we've made Connector::version
> > and ConnectRestExtension::version optional to implement, whereas before it
> > was required that instances of these plugins explicitly implement that
> > method. Should we allow this? An alternative could be to add a new
> > "PossiblyVersioned" interface (name obviously subject to change) that
> > extends the current Versioned interface and contains a default
> > implementation that returns the "undefined" string. The to-be-updated
> > plugin interfaces could extend from this new "PossiblyVersioned" interface,
> > and the plugin interfaces that extend from the current "Versioned"
> > interface could be left unchanged.
> >
> > Cheers,
> >
> > Chris
> >
> > On Wed, Jan 19, 2022 at 10:04 AM Tom Bentley <tb...@redhat.com> wrote:
> >
> > > Hi Mickael,
> > >
> > > Thanks for the KIP. Overall I think this looks good.
> > >
> > > I have a small concern about the proliferation of Versioned. Currently this
> > > interface has a really weak contract. There's no requirement for versions
> > > to be comparable, or documentation about when versions should be changed by
> > > authors of plugins which extend this interface. For example, if I release
> > > version 1.2.3 of my SMT should the version be 1.2.3 or something else? Is
> > > compatibility implied for different versions of my jar file which return
> > > the same result from version()? Because of this it's not really clear what
> > > the consumer of the version string is supposed to do with it. So I think if
> > > we're going to be using Versioned more extensively it would be good to make
> > > the expectations clearer, for both authors and people wanting to reason
> > > about versions.
> > >
> > > Many thanks,
> > >
> > > Tom
> > >
> > > On Mon, 6 Dec 2021 at 09:56, Mickael Maison <mi...@gmail.com>
> > > wrote:
> > >
> > > > Hi Arjun,
> > > >
> > > > Thank you for taking a look at this KIP.
> > > >
> > > > 1.a Yes this is exactly what this KIP is proposing.
> > > >
> > > > 1.b I prefer returning all plugins together instead of having different
> > > > paths.
> > > >
> > > > 2. While we seem to agree it would be nice to expose worker plugins, I
> > > > don't think we've reached a consensus on the right way to do it. For
> > > > that reason, I'm dropping this feature from this KIP. We can address
> > > > this issue in another KIP. I have updated the KIP.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Mon, Dec 6, 2021 at 10:15 AM Mickael Maison <mickael.maison@gmail.com
> > > >
> > > > wrote:
> > > > >
> > > > > Thanks Randall, updated.
> > > > >
> > > > > On Sun, Dec 5, 2021 at 6:05 AM Arjun Satish <ar...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > hey folks,
> > > > > >
> > > > > > Thanks a lot for the KIP and the discussions. Here are a couple of
> > > > thoughts
> > > > > > (apologies if I missed some point in this thread).
> > > > > >
> > > > > > 1. it seems like we are changing the behavior of /connector-plugins
> > > by
> > > > now
> > > > > > returning the list of transforms, converters as well? This would
> > > break
> > > > > > backward compatibility, and clients would need to track what version
> > > of
> > > > > > Connect workers they are querying to keep their experience
> > > consistent.
> > > > I
> > > > > > wonder if we can keep the old behavior, with one of the following
> > > > methods:
> > > > > >
> > > > > > 1a. We add an "expand" multi-query to the /connector-plugin endpoint
> > > > like
> > > > > > how it was done in KIP-465 [1]. This would preserve backward
> > > > compatibility,
> > > > > > as well as allow you to get consolidated output. Acceptable values
> > > for
> > > > the
> > > > > > new param could be: "all" (to show all plugins), or
> > > > > > "connectors|transforms|predicates" that show a subset of the plugin
> > > > types.
> > > > > >
> > > > > > 1b. Query params might get crummy to extend. Alternatively, we could
> > > > have
> > > > > > an additional fine-grained paths for each plugin:
> > > > > >
> > > > > > # to list all the source and sink plugins
> > > > > > /connector-plugins/connectors
> > > > > >
> > > > > > # to list all the SMT plugins
> > > > > > /connector-plugins/transformations
> > > > > >
> > > > > > # to list all the predicate plugins
> > > > > > /connector-plugins/predicates
> > > > > >
> > > > > > # to list all the converter plugins
> > > > > > /connector-plugins/converters
> > > > > >
> > > > > > # to list all the header converter plugins
> > > > > > /connector-plugins/header-converters
> > > > > >
> > > > > > This allows us to cleanly add new types of plugins down the road, and
> > > > > > naturally extend to displaying attributes of these objects. Configs,
> > > > for
> > > > > > example, can be accessed with
> > > > > >
> > > > > >
> > > >
> > > http://localhost:8083/connector-plugins/transformations/Cast$Value/config
> > > > > >
> > > > > > A migration path (for both the suggestions above) would be to add a
> > > > > > deprecation notice on the current behavior of /connector-plugins
> > > > (where,
> > > > > > after this PR is merged, both /connector-plugins and
> > > > > > /connector-plugins/connectors return the same output). And starting
> > > > from
> > > > > > the next major version release, /connector-plugins redirects to
> > > > > > /connector-plugins/connectors (making it super easy for clients to
> > > > catch
> > > > > > up).
> > > > > >
> > > > > > 2. Regarding /worker-plugins, I agree that it is useful to have the
> > > > REST
> > > > > > API inform the state of the REST extensions, and Config Providers.
> > > > However,
> > > > > > users of the Connect clusters will rarely need to know how a config
> > > > > > provider or REST extension is configured to construct data pipelines.
> > > > It's
> > > > > > mostly an admin task, AFAICT. Also, we have also seen these problems
> > > in
> > > > > > Connect clusters where one host is somewhat differently set up than
> > > > other
> > > > > > (contents of plugin path, classpath, local worker configuration
> > > files,
> > > > jvm
> > > > > > versions, GC settings) that surface as issues in Connector behavior.
> > > > > > Overall, I'm wondering if we should just tackle this problem in a
> > > > separate
> > > > > > KIP, that evaluates all of these cases. All of this might nicely fall
> > > > into
> > > > > > under the /admin endpoint and allow us to expose detailed info on the
> > > > state
> > > > > > of the worker (maybe a /admin/worker/ that returns all of this info).
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > [1]
> > > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-465%3A+Add+Consolidated+Connector+Endpoint+to+Connect+REST+API
> > > > > >
> > > > > >
> > > > > > On Sat, Dec 4, 2021 at 10:29 AM Randall Hauch <rh...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Thanks, Mickael. Can we call out these as literals? For example:
> > > > > > >
> > > > > > > > The response structure of the objects in the array remain
> > > > unchanged, but
> > > > > > > this proposal does add "converter", "header_converter" and
> > > > "transformation"
> > > > > > > to the allowed values for the "type" field, in addition to the
> > > > existing
> > > > > > > "sink" and "source" values.
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > > Randall
> > > > > > >
> > > > > > > On Sat, Dec 4, 2021 at 4:58 AM Mickael Maison <
> > > > mickael.maison@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Randall,
> > > > > > > >
> > > > > > > > Sure. The types are sink, source, converter, header_converter,
> > > > > > > > transformation and predicate. I've updated the KIP.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mickael
> > > > > > > >
> > > > > > > > On Fri, Dec 3, 2021 at 10:14 PM Randall Hauch <rh...@gmail.com>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > One more thing, inline below:
> > > > > > > > >
> > > > > > > > > Randall
> > > > > > > > >
> > > > > > > > > On Fri, Dec 3, 2021 at 10:38 AM Mickael Maison <
> > > > > > > mickael.maison@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Thanks Randall for the feedback!
> > > > > > > > > >
> > > > > > > > > > 1. Done
> > > > > > > > > >
> > > > > > > > > > 2. Yes that makes sense. I've updated the KIP
> > > > > > > > > >
> > > > > > > > > > 3.a I've updated to explicitly mention the plugin can be
> > > > provided
> > > > > > > > > > using the fully qualified class or the alias.
> > > > > > > > > > 3.b Only concrete classes will be returned. I made the
> > > mistake
> > > > when I
> > > > > > > > > > crafted the examples by hand. Fixed
> > > > > > > > >
> > > > > > > > > Can we also enumerate all of the values that can be included in
> > > > the
> > > > > > > > > "type" field in the response from the "GET
> > > > > > > > > /connector-plugins?connectorsOnly=false"?
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 4. If a plugin does not override config(), the response is an
> > > > empty
> > > > > > > > > > array. I've updated the KIP
> > > > > > > > > >
> > > > > > > > > > 5. I have updated the compatibility section with more details
> > > > > > > > > >
> > > > > > > > > > 6. Can you clarify what you mean? KIP-507 was for securing
> > > > internal
> > > > > > > > > > REST endpoints.
> > > > > > > > > >
> > > > > > > > > > 7. I don't think it is necessary. I think the use case Chris
> > > > > > > mentioned
> > > > > > > > > > could be useful for admins. I'm interested in exposing the
> > > > > > > > > > connector-level plugins and their configs. I thought it was
> > > > easy to
> > > > > > > > > > add an API for worker plugins while working in this area but
> > > > if there
> > > > > > > > > > isn't a strong consensus, I'm happy to drop it.
> > > > > > > > > >
> > > > > > > > > > Mickael
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch <
> > > rhauch@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the KIP, Mickael. And thanks to everyone else
> > > for
> > > > the
> > > > > > > good
> > > > > > > > > > > discussion so far.
> > > > > > > > > > >
> > > > > > > > > > > Overall this is pretty good, but I do have a few
> > > > > > > questions/comments:
> > > > > > > > > > > 1. Can we include the HTTP method/verb in each of the
> > > > endpoints in
> > > > > > > the
> > > > > > > > > > > "Public Interfaces" section?
> > > > > > > > > > > 2. The sample output for `GET
> > > > > > > /connector-plugins?connectorsOnly=false`
> > > > > > > > > > > includes a `version` field for the connector plugin, but
> > > not
> > > > for
> > > > > > > the
> > > > > > > > > > > transform or converter plugins. Neither the
> > > `Transformation`
> > > > or
> > > > > > > > > > > `Converter` interfaces have a version nor implement the
> > > > `Versioned`
> > > > > > > > > > > interface, so should we add the `Versioned` interface to
> > > the
> > > > > > > > > > > `Transformation` and `Converter` interfaces and add a
> > > default
> > > > > > > method
> > > > > > > > > > > that returns "unknown"? Yes, we could do that in a separate
> > > > KIP,
> > > > > > > but I
> > > > > > > > > > > think it's appropriate to do it in this KIP for a few
> > > > reasons.
> > > > > > > First,
> > > > > > > > > > > we're already modifying the `Converter` interface in this
> > > > KIP and
> > > > > > > it
> > > > > > > > > > > would be quite straightforward to also extend the
> > > `Versioned`
> > > > > > > > > > > interface and add a default method; then doing that for
> > > > > > > > > > > `Transformation` would also be relatively straightforward.
> > > > Second,
> > > > > > > > > > > this KIP provides the justification for adding a version to
> > > > the
> > > > > > > > > > > `Transformation` and `Converter` interfaces.
> > > > > > > > > > > 3. In the "Public Interfaces" section, there are two
> > > > > > > inconsistencies
> > > > > > > > > > > between the `GET /connector-plugins` and `GET
> > > > > > > > > > > /connector-plugins/<plugin>/config` methods. Both will make
> > > > it
> > > > > > > > > > > difficult to use the second method, since the input to the
> > > > second
> > > > > > > > > > > method is not directly found in the response from the first
> > > > method:
> > > > > > > > > > >  a. The example for `GET
> > > > /connector-plugins/Cast$Value/config` uses
> > > > > > > > > > > short name, or what Connect often refers to as the alias,
> > > > > > > "Cast$Value"
> > > > > > > > > > > for the plugin, rather than the fully-qualified name
> > > > > > > > > > > "org.apache.kafka.connect.transforms.Cast$Value". The KIP
> > > > should
> > > > > > > > > > > specify whether it will support both the shortened name and
> > > > the
> > > > > > > > > > > fully-qualified name, or whether only one of these will be
> > > > > > > supported.
> > > > > > > > > > > Not supporting the fully-qualified name will make it
> > > > impossible to
> > > > > > > use
> > > > > > > > > > > the response from the first method as an input to the
> > > second.
> > > > > > > > > > >  a. The example for the list method returns the
> > > > `Transformation`
> > > > > > > base
> > > > > > > > > > > and abstract classes (e.g.,
> > > > > > > > > > > "org.apache.kafka.connect.transforms.Cast") rather than the
> > > > > > > concrete
> > > > > > > > > > > classes (e.g.,
> > > > "org.apache.kafka.connect.transforms.Cast$Value" and
> > > > > > > > > > > "org.apache.kafka.connect.transforms.Cast$Key"), while the
> > > > `GET
> > > > > > > > > > > /connector-plugins/<plugin>/config` will require the
> > > concrete
> > > > > > > class.
> > > > > > > > > > > 4. The "Public Interfaces" section's description of the
> > > `GET
> > > > > > > > > > > /connector-plugins/<plugin>/config` method should probably
> > > > specify
> > > > > > > the
> > > > > > > > > > > expected behavior for `Converter` implementations that
> > > don't
> > > > > > > override
> > > > > > > > > > > the new `config()` method.
> > > > > > > > > > > 5. The "Compatibility, Deprecation, and Migration Plan"
> > > > section
> > > > > > > should
> > > > > > > > > > > describe whether older `Converter` implementations
> > > (compiled
> > > > with
> > > > > > > > > > > older versions of the Connect API without Converter
> > > extending
> > > > > > > > > > > Configurable) will be able to be installed into newer
> > > Connect
> > > > > > > > > > > runtimes, and what the behavior will be for the `GET
> > > > > > > > > > > /connector-plugins/<plugin>/config` method. (Although my
> > > > suggestion
> > > > > > > > > > > for #5 will partially address the latter, we still need to
> > > > talk
> > > > > > > about
> > > > > > > > > > > it in terms of compatibility.)
> > > > > > > > > > > 6. Can we add a small section addressing the security
> > > > implications
> > > > > > > of
> > > > > > > > > > > these new APIs? KIP-507 [1] applied to APIs that existed at
> > > > the
> > > > > > > time,
> > > > > > > > > > > and this KIP should at least mention how it aligns with
> > > that
> > > > > > > > > > > mechanism.
> > > > > > > > > > > 7. Do we really think adding `GET /worker-plugins` is
> > > > necessary?
> > > > > > > While
> > > > > > > > > > > I understand the reasoning, I'm less convinced that it's
> > > > worth
> > > > > > > adding
> > > > > > > > > > > to the public API. First, an administrator still has to
> > > > modify the
> > > > > > > > > > > worker config for all workers to take advantage of that.
> > > > Second,
> > > > > > > > > > > adding this method increases the surface area and therefore
> > > > risk of
> > > > > > > > > > > additional attack vectors, despite not really being able to
> > > > use the
> > > > > > > > > > > resulting information in any other part of the API.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > >
> > > > > > > > > > > Randall
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Dec 1, 2021 at 7:52 AM Mickael Maison <
> > > > > > > mickael.maison@gmail.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks again for the feedback! I've updated the KIP based
> > > > on our
> > > > > > > last
> > > > > > > > > > > > discussions. I've decided to include the new endpoint for
> > > > worker
> > > > > > > > > > > > plugins.
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Yes I agree, it's best to gate the new behavior.
> > > > > > > > > > > > 2. Yes, it was a remnant from the original proposal. I've
> > > > now
> > > > > > > removed
> > > > > > > > > > > > the location field.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Mickael
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
> > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think that's a great idea! I especially like how we
> > > can
> > > > > > > establish the
> > > > > > > > > > > > > expectation that any plugin type that appears in the
> > > > response
> > > > > > > from the GET
> > > > > > > > > > > > > /connector-plugins endpoint will have a corresponding
> > > GET
> > > > > > > > > > > > > /connector-plugins/<type>/config endpoint, but (if we
> > > > decide
> > > > > > > to add them in
> > > > > > > > > > > > > the future), worker plugins won't be expected to expose
> > > > this
> > > > > > > kind of
> > > > > > > > > > > > > information and the different root path helps give a
> > > > decent
> > > > > > > hint about this.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I also like the choice to return an empty ConfigDef
> > > from
> > > > > > > Converter::config
> > > > > > > > > > > > > instead of null.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Two things come to mind:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. We may want to gate this behind a URL query
> > > parameter
> > > > > > > (maybe something
> > > > > > > > > > > > > like "connectorsOnly") that defaults to the old
> > > behavior
> > > > in
> > > > > > > order to avoid
> > > > > > > > > > > > > breaking existing tools such as programmatic UIs that
> > > > use the
> > > > > > > endpoint
> > > > > > > > > > > > > today to discover the connectors that can be created by
> > > > the
> > > > > > > user. We can
> > > > > > > > > > > > > even plan to change the default for that parameter to
> > > the
> > > > > > > newly-proposed
> > > > > > > > > > > > > behavior in the next major release, which should give
> > > > people
> > > > > > > enough time to
> > > > > > > > > > > > > either adapt to the expanded response format or add the
> > > > query
> > > > > > > parameter to
> > > > > > > > > > > > > their tooling.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. The existing GET /connector-plugins endpoint doesn't
> > > > > > > contain information
> > > > > > > > > > > > > on the location of the plugin on the worker's file
> > > > system. Do
> > > > > > > you think we
> > > > > > > > > > > > > should still include this info in the new response
> > > > format?
> > > > > > > Correct me if
> > > > > > > > > > > > > I'm wrong but it seems it may have been proposed
> > > > originally to
> > > > > > > help prevent
> > > > > > > > > > > > > already-addressed bugs in Connect classloading from
> > > > striking;
> > > > > > > all else
> > > > > > > > > > > > > equal, I'd personally err on the side of leaving this
> > > > info out
> > > > > > > or at least
> > > > > > > > > > > > > reducing permitted values for it to just "classpath" or
> > > > > > > "plugin path" in
> > > > > > > > > > > > > order to avoid leaking worker file paths into the REST
> > > > API,
> > > > > > > which might
> > > > > > > > > > > > > bother super security-conscious users.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Chris
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <
> > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yes to keep compatibility we want a default
> > > > implementation
> > > > > > > for
> > > > > > > > > > > > > > Converter.configs(), I've updated the KIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding worker plugins, the use case you described
> > > > seems
> > > > > > > valuable.
> > > > > > > > > > > > > > I'd prefer not mixing worker and connector plugins on
> > > > the
> > > > > > > same
> > > > > > > > > > > > > > endpoint but I agree using /plugins and
> > > /worker-plugins
> > > > > > > could be
> > > > > > > > > > > > > > confusing.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > One alternative is to expose all connector-level
> > > > plugins via
> > > > > > > the
> > > > > > > > > > > > > > existing /connector-plugins endpoint. In that case,
> > > > we'd
> > > > > > > need to keep
> > > > > > > > > > > > > > the current JSON schema and not group plugins by
> > > type.
> > > > As
> > > > > > > the current
> > > > > > > > > > > > > > schema already has a type field for each entry, we'll
> > > > still
> > > > > > > be able to
> > > > > > > > > > > > > > tell them apart. Then we can have /worker-plugins
> > > and a
> > > > > > > relatively
> > > > > > > > > > > > > > clean API. What do you think?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Mickael
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think one potential use case for exposing
> > > > worker-level
> > > > > > > plugins is that
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > may make it easier to determine whether a worker is
> > > > set up
> > > > > > > correctly (the
> > > > > > > > > > > > > > > current alternative requires looking through log
> > > > files and
> > > > > > > can be a
> > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > tedious), and might even make it possible to
> > > > automatically
> > > > > > > identify
> > > > > > > > > > > > > > > discrepancies within a cluster by diffing the
> > > > contents of
> > > > > > > that endpoint
> > > > > > > > > > > > > > > across each worker. But I don't think this has to
> > > be
> > > > > > > addressed by the
> > > > > > > > > > > > > > > current KIP; the only thing that bothers me a
> > > little
> > > > is
> > > > > > > that "plugins" is
> > > > > > > > > > > > > > > generic and it may confuse people down the road if
> > > > we add
> > > > > > > an endpoint for
> > > > > > > > > > > > > > > worker-level plugins ("why is one just called
> > > > 'plugins'
> > > > > > > and the other one
> > > > > > > > > > > > > > > is 'worker-plugins'?"). Probably not worth blocking
> > > > on,
> > > > > > > though.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Agreed that the suggestion for improved validation
> > > > should
> > > > > > > be made on the
> > > > > > > > > > > > > > > KIP-802 thread.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I also noticed that the newly-proposed config
> > > method
> > > > for
> > > > > > > the Converter
> > > > > > > > > > > > > > > interface doesn't have a default implementation,
> > > > making it
> > > > > > > > > > > > > > > backwards-incompatible. Should we add a default
> > > > > > > implementation that
> > > > > > > > > > > > > > returns
> > > > > > > > > > > > > > > either null or an empty ConfigDef?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <
> > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. If we want to expose worker plugins, I think
> > > we
> > > > > > > should do it via a
> > > > > > > > > > > > > > > > separate endpoint. But to be honest, I'm not even
> > > > sure I
> > > > > > > see strong
> > > > > > > > > > > > > > > > use cases for exposing them as they are either
> > > > enabled
> > > > > > > or not and
> > > > > > > > > > > > > > > > can't be changed at runtime. So I'd prefer to
> > > > stick to
> > > > > > > "connector
> > > > > > > > > > > > > > > > level" plugins in this KIP. Let me now if you
> > > have
> > > > use
> > > > > > > cases, I'm open
> > > > > > > > > > > > > > > > to reconsider this choice.
> > > > > > > > > > > > > > > > I'll add that in the rejected alternatives
> > > section
> > > > for
> > > > > > > now
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2. I remembered seeing issues in the past with
> > > > multiple
> > > > > > > plugin.path
> > > > > > > > > > > > > > > > entries but I tried today and I was able to mix
> > > and
> > > > > > > match plugins from
> > > > > > > > > > > > > > > > different paths. So my bad for getting confused.
> > > > > > > > > > > > > > > > Then I agree, it makes more sense to group them
> > > by
> > > > > > > plugin type.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 3. Yes this should be covered in KIP-802:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 4. No particular reason. We can support both
> > > > formats
> > > > > > > like today. I've
> > > > > > > > > > > > > > > > updated the KIP
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Mickael
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > > > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think the increase in scope here is great and
> > > > the
> > > > > > > added value
> > > > > > > > > > > > > > certainly
> > > > > > > > > > > > > > > > > justifies the proposed changes. I have some
> > > > thoughts
> > > > > > > but overall I
> > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > direction this is going in now.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. The new /plugins endpoint is described as
> > > > > > > containing "all plugins
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > are Connectors, Transformations, Converters,
> > > > > > > HeaderConverters and
> > > > > > > > > > > > > > > > > Predicates". So essentially, it looks like we
> > > > want to
> > > > > > > expose all
> > > > > > > > > > > > > > plugins
> > > > > > > > > > > > > > > > > that are configured on a per-connector basis,
> > > but
> > > > > > > exclude plugins
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > configured on a per-worker basis (such as
> > > config
> > > > > > > providers and REST
> > > > > > > > > > > > > > > > > extensions). Do you think it may be valuable to
> > > > expose
> > > > > > > information on
> > > > > > > > > > > > > > > > > worker-level plugins as well?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2. The description for the new /plugins
> > > endpoint
> > > > also
> > > > > > > states that
> > > > > > > > > > > > > > > > "Plugins
> > > > > > > > > > > > > > > > > will be grouped by plugin.path. This will make
> > > it
> > > > > > > clear to users
> > > > > > > > > > > > > > what's
> > > > > > > > > > > > > > > > > available to use as it's not possible to use a
> > > > > > > Connector from one
> > > > > > > > > > > > > > path
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > Transformations from another.". Is this true? I
> > > > > > > thought that
> > > > > > > > > > > > > > Connect's
> > > > > > > > > > > > > > > > > classloading made it possible to package
> > > > > > > > > > > > > > > > > converters/transformations/predicates
> > > completely
> > > > > > > independently from
> > > > > > > > > > > > > > each
> > > > > > > > > > > > > > > > > other, and to reference them from
> > > > > > > also-independently-packaged
> > > > > > > > > > > > > > connectors.
> > > > > > > > > > > > > > > > > If it turns out that this is the case, could we
> > > > > > > consider
> > > > > > > > > > > > > > restructuring
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > response to be grouped by plugin type instead
> > > of
> > > > by
> > > > > > > classloader?
> > > > > > > > > > > > > > There's
> > > > > > > > > > > > > > > > > also the ungrouped format proposed in KIP-494 (
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > >
> > > >
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > which we might consider as well.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 3. I think this can be left for a follow-up KIP
> > > > if
> > > > > > > necessary, but I'm
> > > > > > > > > > > > > > > > > curious about your thoughts on adding new
> > > > validate
> > > > > > > methods to all
> > > > > > > > > > > > > > > > > connector-level plugins that can be used
> > > > similarly to
> > > > > > > how the
> > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > Connector::validate method (
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > >
> > > >
> > > https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > > is used. This would allow for plugins to
> > > perform
> > > > > > > validation that's
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > sophisticated than what the ConfigDef is
> > > capable
> > > > of,
> > > > > > > such as
> > > > > > > > > > > > > > validating
> > > > > > > > > > > > > > > > > combinations of properties like a hostname and
> > > > > > > credentials for
> > > > > > > > > > > > > > reaching
> > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > I know that at least Confluent's Avro,
> > > protobuf,
> > > > and
> > > > > > > JSON schema
> > > > > > > > > > > > > > > > converters
> > > > > > > > > > > > > > > > > would benefit from this kind of feature. It's a
> > > > little
> > > > > > > tangential to
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > KIP (which at the moment is about discovering
> > > > plugins
> > > > > > > and their
> > > > > > > > > > > > > > > > > configuration surfaces, as opposed to
> > > validating
> > > > > > > them), but I
> > > > > > > > > > > > > > figured I'd
> > > > > > > > > > > > > > > > > ask since we're going to be expanding the
> > > > Converter
> > > > > > > interface and it
> > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > useful to tackle this while we're in the
> > > > neighborhood.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 4. The description for the new
> > > > > > > /plugins/<type>/<name>/configdef
> > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > states that "Name must be the fully qualified
> > > > class
> > > > > > > name of the
> > > > > > > > > > > > > > plugin".
> > > > > > > > > > > > > > > > > Any reason not to also support aliases (e.g.,
> > > > > > > > > > > > > > "FileStreamSinkConnector"
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > "FileStreamSink" instead of
> > > > > > > > > > > > > > > > >
> > > > > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael
> > > Maison <
> > > > > > > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks all for the feedback!
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Chris,
> > > > > > > > > > > > > > > > > > I agree that fixing the current endpoint
> > > helps
> > > > a
> > > > > > > lot. Thanks for
> > > > > > > > > > > > > > > > > > raising these JIRAs and submitting a PR!
> > > > > > > > > > > > > > > > > > However thinking about the issue further, I
> > > > decided
> > > > > > > to expand the
> > > > > > > > > > > > > > > > > > scope of the KIP to cover all user-visible
> > > > plugins.
> > > > > > > > > > > > > > > > > > In practice, users want to know about all
> > > > available
> > > > > > > plugins not
> > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > connectors. This includes transformations,
> > > > > > > converters,
> > > > > > > > > > > > > > > > > > header_converters and predicates. As we also
> > > > want to
> > > > > > > retrieve
> > > > > > > > > > > > > > > > > > configdef for these too, I think it makes
> > > > sense to
> > > > > > > introduce a new
> > > > > > > > > > > > > > > > > > endpoint to do so. Alongside we obviously
> > > need
> > > > a new
> > > > > > > endpoint for
> > > > > > > > > > > > > > > > > > listing all plugins.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Gunnar,
> > > > > > > > > > > > > > > > > > I took a look at exposing valid values via
> > > the
> > > > API.
> > > > > > > I think the
> > > > > > > > > > > > > > issue
> > > > > > > > > > > > > > > > > > is that Validators don't expose a way to
> > > > retrieve
> > > > > > > valid values.
> > > > > > > > > > > > > > > > > > Changing validators will have an impact on
> > > all
> > > > > > > components so I'd
> > > > > > > > > > > > > > > > > > prefer to address this requirement in a
> > > > separate
> > > > > > > KIP. I agree this
> > > > > > > > > > > > > > > > > > would be an interesting improvement and I'd
> > > > happy to
> > > > > > > write a KIP
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > it too.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I have updated the KIP accordingly. Let me
> > > > know if
> > > > > > > you have further
> > > > > > > > > > > > > > > > > > feedback.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Mickael
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar
> > > Morling
> > > > > > > > > > > > > > > > > > <gu...@googlemail.com.invalid>
> > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I'm +1 for adding a GET endpoint for
> > > > obtaining
> > > > > > > config
> > > > > > > > > > > > > > definitions. It
> > > > > > > > > > > > > > > > > > > always felt odd to me that one has to issue
> > > > a PUT
> > > > > > > for that
> > > > > > > > > > > > > > purpose.
> > > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > > > nothing else, it'd be better in terms of
> > > > > > > discoverability of the
> > > > > > > > > > > > > > KC
> > > > > > > > > > > > > > > > REST
> > > > > > > > > > > > > > > > > > API.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > One additional feature request I'd have is
> > > to
> > > > > > > expose the valid
> > > > > > > > > > > > > > enum
> > > > > > > > > > > > > > > > > > > constants for enum-typed options. That'll
> > > > help to
> > > > > > > display the
> > > > > > > > > > > > > > values
> > > > > > > > > > > > > > > > in a
> > > > > > > > > > > > > > > > > > > drop-down or via radio buttons in a UI,
> > > give
> > > > us
> > > > > > > tab completion in
> > > > > > > > > > > > > > > > kcctl,
> > > > > > > > > > > > > > > > > > > etc.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > --Gunnar
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb
> > > > Chris
> > > > > > > Egerton
> > > > > > > > > > > > > > > > > > > <ch...@confluent.io.invalid>:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Viktor,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > It sounds like there are three major
> > > > points here
> > > > > > > in favor of a
> > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > GET
> > > > > > > > > > > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. You cannot issue a blank ("dummy")
> > > > request
> > > > > > > for sink
> > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > because a
> > > > > > > > > > > > > > > > > > > > topic list/topic regex has to be supplied
> > > > > > > (otherwise the PUT
> > > > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > > > > returns a 500 response)
> > > > > > > > > > > > > > > > > > > > 2. A dummy request still triggers custom
> > > > > > > validations by the
> > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > which may be best to avoid if we know for
> > > > sure
> > > > > > > that the config
> > > > > > > > > > > > > > > > isn't
> > > > > > > > > > > > > > > > > > worth
> > > > > > > > > > > > > > > > > > > > validating yet
> > > > > > > > > > > > > > > > > > > > 3. It's more ergonomic and intuitive to
> > > be
> > > > able
> > > > > > > to issue a GET
> > > > > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > > > > without having to give a dummy connector
> > > > config
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > With regards to 1, this is actually a bug
> > > > in
> > > > > > > Connect (
> > > > > > > > > > > > > > > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix
> > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > > > implemented and awaiting committer
> > > review (
> > > > > > > > > > > > > > > > > > > >
> > > https://github.com/apache/kafka/pull/11369).
> > > > I
> > > > > > > think it'd be
> > > > > > > > > > > > > > > > better to
> > > > > > > > > > > > > > > > > > > > focus on fixing this bug in general
> > > > instead of
> > > > > > > implementing a
> > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > REST
> > > > > > > > > > > > > > > > > > > > endpoint in order to allow people to work
> > > > around
> > > > > > > it.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > With regards to 2, this is technically
> > > > possible
> > > > > > > but I'm unsure
> > > > > > > > > > > > > > > > it'd be
> > > > > > > > > > > > > > > > > > too
> > > > > > > > > > > > > > > > > > > > common out in the wild given that most
> > > > > > > validations that could
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > expensive
> > > > > > > > > > > > > > > > > > > > would involve things like connecting to a
> > > > > > > database, checking
> > > > > > > > > > > > > > if a
> > > > > > > > > > > > > > > > cloud
> > > > > > > > > > > > > > > > > > > > storage bucket exists, etc., none of
> > > which
> > > > are
> > > > > > > possible without
> > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > configuration properties from the user
> > > (db
> > > > > > > hostname, bucket
> > > > > > > > > > > > > > name,
> > > > > > > > > > > > > > > > > > etc.).
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > With regards to 3, I do agree that it'd
> > > be
> > > > > > > easier for people
> > > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > > UIs
> > > > > > > > > > > > > > > > > > > > to have a GET API to work against. I'm
> > > > just not
> > > > > > > sure it's
> > > > > > > > > > > > > > worth the
> > > > > > > > > > > > > > > > > > > > additional implementation, testing, and
> > > > > > > maintenance burden. If
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > > > possible to issue a PUT request without
> > > > > > > unexpected 500s for
> > > > > > > > > > > > > > invalid
> > > > > > > > > > > > > > > > > > > > configs, would that suffice? AFAICT it'd
> > > > > > > basically be as
> > > > > > > > > > > > > > simple as
> > > > > > > > > > > > > > > > > > issuing
> > > > > > > > > > > > > > > > > > > > a PUT request with a dummy body
> > > consisting
> > > > of
> > > > > > > nothing except
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > class (which at this point we might even
> > > > make
> > > > > > > unnecessary and
> > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > > > automatically replace with the connector
> > > > class
> > > > > > > from the URL)
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > then
> > > > > > > > > > > > > > > > > > > > filtering the response to just grab the
> > > > > > > "definition" field of
> > > > > > > > > > > > > > each
> > > > > > > > > > > > > > > > > > element
> > > > > > > > > > > > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor
> > > > > > > Somogyi-Vass <
> > > > > > > > > > > > > > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Folks,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I too think this would be a very useful
> > > > > > > feature. Some of our
> > > > > > > > > > > > > > > > > > management
> > > > > > > > > > > > > > > > > > > > > applications would provide a wizard for
> > > > > > > creating connectors.
> > > > > > > > > > > > > > In
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > scenario the user basically would fill
> > > > out a
> > > > > > > sample
> > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > generated
> > > > > > > > > > > > > > > > > > > > > by the UI which would send it back to
> > > > Connect
> > > > > > > for validation
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > eventually
> > > > > > > > > > > > > > > > > > > > > create a new connector. The first part
> > > > of this
> > > > > > > workflow can
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > enhanced
> > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > we had an API that can return the
> > > > > > > configuration definition
> > > > > > > > > > > > > > of the
> > > > > > > > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > > > > > type of connector as the UI application
> > > > would
> > > > > > > be able to
> > > > > > > > > > > > > > > > generate a
> > > > > > > > > > > > > > > > > > > > sample
> > > > > > > > > > > > > > > > > > > > > for the user based on that (nicely
> > > drawn
> > > > > > > diagram:
> > > > > > > > > > > > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > > > > > > > > > > > The
> > > > > > > connector-plugins/{connectorType}/config/validate API
> > > > > > > > > > > > > > > > essentially
> > > > > > > > > > > > > > > > > > > > works
> > > > > > > > > > > > > > > > > > > > > and returns the data that we need,
> > > > however it
> > > > > > > is a HTTP PUT
> > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > > > > bit unintuitive for a fetch-like
> > > > functionality
> > > > > > > and also
> > > > > > > > > > > > > > > > functionally
> > > > > > > > > > > > > > > > > > > > > different as it validates the given
> > > > (dummy)
> > > > > > > request. In case
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > sink
> > > > > > > > > > > > > > > > > > > > > connectors one would need to also
> > > > provide a
> > > > > > > topic name.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > A suggestion for the KIP: I think it
> > > can
> > > > be
> > > > > > > useful to return
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > > groups and the connector class' name
> > > > similarly
> > > > > > > to the
> > > > > > > > > > > > > > validate
> > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > case any frontend needs them (and also
> > > > the
> > > > > > > response would be
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > validate API but simpler).
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Viktor
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne
> > > > Dolan <
> > > > > > > > > > > > > > > > ryannedolan@gmail.com>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think it'd be worth adding a GET
> > > > version,
> > > > > > > fwiw. Could be
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > > > > handler
> > > > > > > > > > > > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael
> > > > Maison
> > > > > > > <
> > > > > > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > You're right, you can achieve the
> > > > same
> > > > > > > functionality
> > > > > > > > > > > > > > using
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > > > > > > > > > > > In my mind it was only for
> > > > validation once
> > > > > > > you have
> > > > > > > > > > > > > > build a
> > > > > > > > > > > > > > > > > > > > > > > configuration but when used with an
> > > > empty
> > > > > > > configuration,
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > basically
> > > > > > > > > > > > > > > > > > > > > > > serves the same purpose as the
> > > > proposed
> > > > > > > new endpoint.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I think it's a bit easier to use a
> > > > GET
> > > > > > > endpoint but I
> > > > > > > > > > > > > > don't
> > > > > > > > > > > > > > > > > > think it
> > > > > > > > > > > > > > > > > > > > > > > really warrants a different
> > > endpoint.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM
> > > Chris
> > > > > > > Egerton
> > > > > > > > > > > > > > > > > > > > > > > <ch...@confluent.io.invalid>
> > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I'm wondering about the use case
> > > > here.
> > > > > > > The motivation
> > > > > > > > > > > > > > > > section
> > > > > > > > > > > > > > > > > > > > states
> > > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > "Connect does not provide a way
> > > to
> > > > see
> > > > > > > what
> > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > requires. Instead users have to
> > > go
> > > > look
> > > > > > > at the
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > documentation
> > > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > in the worst case, look directly
> > > > at the
> > > > > > > connector
> > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > code.",
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > with this KIP, "users will be
> > > able
> > > > to
> > > > > > > discover the
> > > > > > > > > > > > > > required
> > > > > > > > > > > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > > > > > > > > > > for connectors installed in a
> > > > Connect
> > > > > > > cluster" and
> > > > > > > > > > > > > > "tools
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > generate wizards for configuring
> > > > and
> > > > > > > starting
> > > > > > > > > > > > > > connectors".
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > > > > > > > > > > > endpoint not address these
> > > points?
> > > > What
> > > > > > > will the
> > > > > > > > > > > > > > > > newly-proposed
> > > > > > > > > > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > > > > > > > > allow users to do that they will
> > > > not
> > > > > > > already be able
> > > > > > > > > > > > > > to do
> > > > > > > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM
> > > > Mickael
> > > > > > > Maison <
> > > > > > > > > > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I've created KIP-769 to expose
> > > > > > > connector
> > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > definitions
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Please take a look and let me
> > > > know if
> > > > > > > you have any
> > > > > > > > > > > > > > > > feedback.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > >
> > > >
> > > >
> > >

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Mickael Maison <mi...@gmail.com>]

Hi Tom,

Thanks for the feedback.

You're right we don't have any contracts around the version field.
Like Chris I've only used the current field to find the exact source
code of connectors. Even if it's a limited use, I think that's enough
to justify adding it to all plugins and making them consistent. Work
for improving the contract can be done in a separate KIP.

Thanks,
Mickael

On Wed, Jan 19, 2022 at 7:55 PM Chris Egerton
<ch...@confluent.io.invalid> wrote:
>
> Hi all,
>
> With regards to Tom's question about possible uses of the version string,
> one of the most useful applications I've found for it so far has been
> debugging connector issues, especially based on captured logs. Using the
> version string to identify a git tag containing the exact source code for a
> connector, and then using that as a reference when poring over log files
> that usually contain class names, method names, and/or line numbers, is a
> great way to construct a timeline of events and understand how things may
> have gone wrong. I don't think we can require this as part of the contract
> for the version method, but at least identifying this use case in the docs
> for it could be useful to help connector/SMT/converter/etc. developers
> understand why they should care about what this method returns and how it
> could be beneficial to themselves and their users down the road.
>
> One other potential use case that comes up is that, if/when Connect
> supports loading multiple versions of the same plugin type on a single
> worker, the version string can be used to identify plugins that share the
> same name and type but which should be exposed individually to users. If we
> want to leave room for this possibility, perhaps part of the contract for
> the method can be that "any two releases of the same plugin but with
> different behavior should return different values from their version
> method".
>
> One final note--I've just realized that, with the introduction of a default
> implementation for the Versioned interface, we've made Connector::version
> and ConnectRestExtension::version optional to implement, whereas before it
> was required that instances of these plugins explicitly implement that
> method. Should we allow this? An alternative could be to add a new
> "PossiblyVersioned" interface (name obviously subject to change) that
> extends the current Versioned interface and contains a default
> implementation that returns the "undefined" string. The to-be-updated
> plugin interfaces could extend from this new "PossiblyVersioned" interface,
> and the plugin interfaces that extend from the current "Versioned"
> interface could be left unchanged.
>
> Cheers,
>
> Chris
>
> On Wed, Jan 19, 2022 at 10:04 AM Tom Bentley <tb...@redhat.com> wrote:
>
> > Hi Mickael,
> >
> > Thanks for the KIP. Overall I think this looks good.
> >
> > I have a small concern about the proliferation of Versioned. Currently this
> > interface has a really weak contract. There's no requirement for versions
> > to be comparable, or documentation about when versions should be changed by
> > authors of plugins which extend this interface. For example, if I release
> > version 1.2.3 of my SMT should the version be 1.2.3 or something else? Is
> > compatibility implied for different versions of my jar file which return
> > the same result from version()? Because of this it's not really clear what
> > the consumer of the version string is supposed to do with it. So I think if
> > we're going to be using Versioned more extensively it would be good to make
> > the expectations clearer, for both authors and people wanting to reason
> > about versions.
> >
> > Many thanks,
> >
> > Tom
> >
> > On Mon, 6 Dec 2021 at 09:56, Mickael Maison <mi...@gmail.com>
> > wrote:
> >
> > > Hi Arjun,
> > >
> > > Thank you for taking a look at this KIP.
> > >
> > > 1.a Yes this is exactly what this KIP is proposing.
> > >
> > > 1.b I prefer returning all plugins together instead of having different
> > > paths.
> > >
> > > 2. While we seem to agree it would be nice to expose worker plugins, I
> > > don't think we've reached a consensus on the right way to do it. For
> > > that reason, I'm dropping this feature from this KIP. We can address
> > > this issue in another KIP. I have updated the KIP.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Mon, Dec 6, 2021 at 10:15 AM Mickael Maison <mickael.maison@gmail.com
> > >
> > > wrote:
> > > >
> > > > Thanks Randall, updated.
> > > >
> > > > On Sun, Dec 5, 2021 at 6:05 AM Arjun Satish <ar...@gmail.com>
> > > wrote:
> > > > >
> > > > > hey folks,
> > > > >
> > > > > Thanks a lot for the KIP and the discussions. Here are a couple of
> > > thoughts
> > > > > (apologies if I missed some point in this thread).
> > > > >
> > > > > 1. it seems like we are changing the behavior of /connector-plugins
> > by
> > > now
> > > > > returning the list of transforms, converters as well? This would
> > break
> > > > > backward compatibility, and clients would need to track what version
> > of
> > > > > Connect workers they are querying to keep their experience
> > consistent.
> > > I
> > > > > wonder if we can keep the old behavior, with one of the following
> > > methods:
> > > > >
> > > > > 1a. We add an "expand" multi-query to the /connector-plugin endpoint
> > > like
> > > > > how it was done in KIP-465 [1]. This would preserve backward
> > > compatibility,
> > > > > as well as allow you to get consolidated output. Acceptable values
> > for
> > > the
> > > > > new param could be: "all" (to show all plugins), or
> > > > > "connectors|transforms|predicates" that show a subset of the plugin
> > > types.
> > > > >
> > > > > 1b. Query params might get crummy to extend. Alternatively, we could
> > > have
> > > > > an additional fine-grained paths for each plugin:
> > > > >
> > > > > # to list all the source and sink plugins
> > > > > /connector-plugins/connectors
> > > > >
> > > > > # to list all the SMT plugins
> > > > > /connector-plugins/transformations
> > > > >
> > > > > # to list all the predicate plugins
> > > > > /connector-plugins/predicates
> > > > >
> > > > > # to list all the converter plugins
> > > > > /connector-plugins/converters
> > > > >
> > > > > # to list all the header converter plugins
> > > > > /connector-plugins/header-converters
> > > > >
> > > > > This allows us to cleanly add new types of plugins down the road, and
> > > > > naturally extend to displaying attributes of these objects. Configs,
> > > for
> > > > > example, can be accessed with
> > > > >
> > > > >
> > >
> > http://localhost:8083/connector-plugins/transformations/Cast$Value/config
> > > > >
> > > > > A migration path (for both the suggestions above) would be to add a
> > > > > deprecation notice on the current behavior of /connector-plugins
> > > (where,
> > > > > after this PR is merged, both /connector-plugins and
> > > > > /connector-plugins/connectors return the same output). And starting
> > > from
> > > > > the next major version release, /connector-plugins redirects to
> > > > > /connector-plugins/connectors (making it super easy for clients to
> > > catch
> > > > > up).
> > > > >
> > > > > 2. Regarding /worker-plugins, I agree that it is useful to have the
> > > REST
> > > > > API inform the state of the REST extensions, and Config Providers.
> > > However,
> > > > > users of the Connect clusters will rarely need to know how a config
> > > > > provider or REST extension is configured to construct data pipelines.
> > > It's
> > > > > mostly an admin task, AFAICT. Also, we have also seen these problems
> > in
> > > > > Connect clusters where one host is somewhat differently set up than
> > > other
> > > > > (contents of plugin path, classpath, local worker configuration
> > files,
> > > jvm
> > > > > versions, GC settings) that surface as issues in Connector behavior.
> > > > > Overall, I'm wondering if we should just tackle this problem in a
> > > separate
> > > > > KIP, that evaluates all of these cases. All of this might nicely fall
> > > into
> > > > > under the /admin endpoint and allow us to expose detailed info on the
> > > state
> > > > > of the worker (maybe a /admin/worker/ that returns all of this info).
> > > > >
> > > > > Thanks,
> > > > >
> > > > > [1]
> > > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-465%3A+Add+Consolidated+Connector+Endpoint+to+Connect+REST+API
> > > > >
> > > > >
> > > > > On Sat, Dec 4, 2021 at 10:29 AM Randall Hauch <rh...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Thanks, Mickael. Can we call out these as literals? For example:
> > > > > >
> > > > > > > The response structure of the objects in the array remain
> > > unchanged, but
> > > > > > this proposal does add "converter", "header_converter" and
> > > "transformation"
> > > > > > to the allowed values for the "type" field, in addition to the
> > > existing
> > > > > > "sink" and "source" values.
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Randall
> > > > > >
> > > > > > On Sat, Dec 4, 2021 at 4:58 AM Mickael Maison <
> > > mickael.maison@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Randall,
> > > > > > >
> > > > > > > Sure. The types are sink, source, converter, header_converter,
> > > > > > > transformation and predicate. I've updated the KIP.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mickael
> > > > > > >
> > > > > > > On Fri, Dec 3, 2021 at 10:14 PM Randall Hauch <rh...@gmail.com>
> > > wrote:
> > > > > > > >
> > > > > > > > One more thing, inline below:
> > > > > > > >
> > > > > > > > Randall
> > > > > > > >
> > > > > > > > On Fri, Dec 3, 2021 at 10:38 AM Mickael Maison <
> > > > > > mickael.maison@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Thanks Randall for the feedback!
> > > > > > > > >
> > > > > > > > > 1. Done
> > > > > > > > >
> > > > > > > > > 2. Yes that makes sense. I've updated the KIP
> > > > > > > > >
> > > > > > > > > 3.a I've updated to explicitly mention the plugin can be
> > > provided
> > > > > > > > > using the fully qualified class or the alias.
> > > > > > > > > 3.b Only concrete classes will be returned. I made the
> > mistake
> > > when I
> > > > > > > > > crafted the examples by hand. Fixed
> > > > > > > >
> > > > > > > > Can we also enumerate all of the values that can be included in
> > > the
> > > > > > > > "type" field in the response from the "GET
> > > > > > > > /connector-plugins?connectorsOnly=false"?
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 4. If a plugin does not override config(), the response is an
> > > empty
> > > > > > > > > array. I've updated the KIP
> > > > > > > > >
> > > > > > > > > 5. I have updated the compatibility section with more details
> > > > > > > > >
> > > > > > > > > 6. Can you clarify what you mean? KIP-507 was for securing
> > > internal
> > > > > > > > > REST endpoints.
> > > > > > > > >
> > > > > > > > > 7. I don't think it is necessary. I think the use case Chris
> > > > > > mentioned
> > > > > > > > > could be useful for admins. I'm interested in exposing the
> > > > > > > > > connector-level plugins and their configs. I thought it was
> > > easy to
> > > > > > > > > add an API for worker plugins while working in this area but
> > > if there
> > > > > > > > > isn't a strong consensus, I'm happy to drop it.
> > > > > > > > >
> > > > > > > > > Mickael
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch <
> > rhauch@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Thanks for the KIP, Mickael. And thanks to everyone else
> > for
> > > the
> > > > > > good
> > > > > > > > > > discussion so far.
> > > > > > > > > >
> > > > > > > > > > Overall this is pretty good, but I do have a few
> > > > > > questions/comments:
> > > > > > > > > > 1. Can we include the HTTP method/verb in each of the
> > > endpoints in
> > > > > > the
> > > > > > > > > > "Public Interfaces" section?
> > > > > > > > > > 2. The sample output for `GET
> > > > > > /connector-plugins?connectorsOnly=false`
> > > > > > > > > > includes a `version` field for the connector plugin, but
> > not
> > > for
> > > > > > the
> > > > > > > > > > transform or converter plugins. Neither the
> > `Transformation`
> > > or
> > > > > > > > > > `Converter` interfaces have a version nor implement the
> > > `Versioned`
> > > > > > > > > > interface, so should we add the `Versioned` interface to
> > the
> > > > > > > > > > `Transformation` and `Converter` interfaces and add a
> > default
> > > > > > method
> > > > > > > > > > that returns "unknown"? Yes, we could do that in a separate
> > > KIP,
> > > > > > but I
> > > > > > > > > > think it's appropriate to do it in this KIP for a few
> > > reasons.
> > > > > > First,
> > > > > > > > > > we're already modifying the `Converter` interface in this
> > > KIP and
> > > > > > it
> > > > > > > > > > would be quite straightforward to also extend the
> > `Versioned`
> > > > > > > > > > interface and add a default method; then doing that for
> > > > > > > > > > `Transformation` would also be relatively straightforward.
> > > Second,
> > > > > > > > > > this KIP provides the justification for adding a version to
> > > the
> > > > > > > > > > `Transformation` and `Converter` interfaces.
> > > > > > > > > > 3. In the "Public Interfaces" section, there are two
> > > > > > inconsistencies
> > > > > > > > > > between the `GET /connector-plugins` and `GET
> > > > > > > > > > /connector-plugins/<plugin>/config` methods. Both will make
> > > it
> > > > > > > > > > difficult to use the second method, since the input to the
> > > second
> > > > > > > > > > method is not directly found in the response from the first
> > > method:
> > > > > > > > > >  a. The example for `GET
> > > /connector-plugins/Cast$Value/config` uses
> > > > > > > > > > short name, or what Connect often refers to as the alias,
> > > > > > "Cast$Value"
> > > > > > > > > > for the plugin, rather than the fully-qualified name
> > > > > > > > > > "org.apache.kafka.connect.transforms.Cast$Value". The KIP
> > > should
> > > > > > > > > > specify whether it will support both the shortened name and
> > > the
> > > > > > > > > > fully-qualified name, or whether only one of these will be
> > > > > > supported.
> > > > > > > > > > Not supporting the fully-qualified name will make it
> > > impossible to
> > > > > > use
> > > > > > > > > > the response from the first method as an input to the
> > second.
> > > > > > > > > >  a. The example for the list method returns the
> > > `Transformation`
> > > > > > base
> > > > > > > > > > and abstract classes (e.g.,
> > > > > > > > > > "org.apache.kafka.connect.transforms.Cast") rather than the
> > > > > > concrete
> > > > > > > > > > classes (e.g.,
> > > "org.apache.kafka.connect.transforms.Cast$Value" and
> > > > > > > > > > "org.apache.kafka.connect.transforms.Cast$Key"), while the
> > > `GET
> > > > > > > > > > /connector-plugins/<plugin>/config` will require the
> > concrete
> > > > > > class.
> > > > > > > > > > 4. The "Public Interfaces" section's description of the
> > `GET
> > > > > > > > > > /connector-plugins/<plugin>/config` method should probably
> > > specify
> > > > > > the
> > > > > > > > > > expected behavior for `Converter` implementations that
> > don't
> > > > > > override
> > > > > > > > > > the new `config()` method.
> > > > > > > > > > 5. The "Compatibility, Deprecation, and Migration Plan"
> > > section
> > > > > > should
> > > > > > > > > > describe whether older `Converter` implementations
> > (compiled
> > > with
> > > > > > > > > > older versions of the Connect API without Converter
> > extending
> > > > > > > > > > Configurable) will be able to be installed into newer
> > Connect
> > > > > > > > > > runtimes, and what the behavior will be for the `GET
> > > > > > > > > > /connector-plugins/<plugin>/config` method. (Although my
> > > suggestion
> > > > > > > > > > for #5 will partially address the latter, we still need to
> > > talk
> > > > > > about
> > > > > > > > > > it in terms of compatibility.)
> > > > > > > > > > 6. Can we add a small section addressing the security
> > > implications
> > > > > > of
> > > > > > > > > > these new APIs? KIP-507 [1] applied to APIs that existed at
> > > the
> > > > > > time,
> > > > > > > > > > and this KIP should at least mention how it aligns with
> > that
> > > > > > > > > > mechanism.
> > > > > > > > > > 7. Do we really think adding `GET /worker-plugins` is
> > > necessary?
> > > > > > While
> > > > > > > > > > I understand the reasoning, I'm less convinced that it's
> > > worth
> > > > > > adding
> > > > > > > > > > to the public API. First, an administrator still has to
> > > modify the
> > > > > > > > > > worker config for all workers to take advantage of that.
> > > Second,
> > > > > > > > > > adding this method increases the surface area and therefore
> > > risk of
> > > > > > > > > > additional attack vectors, despite not really being able to
> > > use the
> > > > > > > > > > resulting information in any other part of the API.
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > >
> > > > > > > > > > Randall
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Dec 1, 2021 at 7:52 AM Mickael Maison <
> > > > > > mickael.maison@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Chris,
> > > > > > > > > > >
> > > > > > > > > > > Thanks again for the feedback! I've updated the KIP based
> > > on our
> > > > > > last
> > > > > > > > > > > discussions. I've decided to include the new endpoint for
> > > worker
> > > > > > > > > > > plugins.
> > > > > > > > > > >
> > > > > > > > > > > 1. Yes I agree, it's best to gate the new behavior.
> > > > > > > > > > > 2. Yes, it was a remnant from the original proposal. I've
> > > now
> > > > > > removed
> > > > > > > > > > > the location field.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Mickael
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
> > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > >
> > > > > > > > > > > > I think that's a great idea! I especially like how we
> > can
> > > > > > establish the
> > > > > > > > > > > > expectation that any plugin type that appears in the
> > > response
> > > > > > from the GET
> > > > > > > > > > > > /connector-plugins endpoint will have a corresponding
> > GET
> > > > > > > > > > > > /connector-plugins/<type>/config endpoint, but (if we
> > > decide
> > > > > > to add them in
> > > > > > > > > > > > the future), worker plugins won't be expected to expose
> > > this
> > > > > > kind of
> > > > > > > > > > > > information and the different root path helps give a
> > > decent
> > > > > > hint about this.
> > > > > > > > > > > >
> > > > > > > > > > > > I also like the choice to return an empty ConfigDef
> > from
> > > > > > Converter::config
> > > > > > > > > > > > instead of null.
> > > > > > > > > > > >
> > > > > > > > > > > > Two things come to mind:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. We may want to gate this behind a URL query
> > parameter
> > > > > > (maybe something
> > > > > > > > > > > > like "connectorsOnly") that defaults to the old
> > behavior
> > > in
> > > > > > order to avoid
> > > > > > > > > > > > breaking existing tools such as programmatic UIs that
> > > use the
> > > > > > endpoint
> > > > > > > > > > > > today to discover the connectors that can be created by
> > > the
> > > > > > user. We can
> > > > > > > > > > > > even plan to change the default for that parameter to
> > the
> > > > > > newly-proposed
> > > > > > > > > > > > behavior in the next major release, which should give
> > > people
> > > > > > enough time to
> > > > > > > > > > > > either adapt to the expanded response format or add the
> > > query
> > > > > > parameter to
> > > > > > > > > > > > their tooling.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. The existing GET /connector-plugins endpoint doesn't
> > > > > > contain information
> > > > > > > > > > > > on the location of the plugin on the worker's file
> > > system. Do
> > > > > > you think we
> > > > > > > > > > > > should still include this info in the new response
> > > format?
> > > > > > Correct me if
> > > > > > > > > > > > I'm wrong but it seems it may have been proposed
> > > originally to
> > > > > > help prevent
> > > > > > > > > > > > already-addressed bugs in Connect classloading from
> > > striking;
> > > > > > all else
> > > > > > > > > > > > equal, I'd personally err on the side of leaving this
> > > info out
> > > > > > or at least
> > > > > > > > > > > > reducing permitted values for it to just "classpath" or
> > > > > > "plugin path" in
> > > > > > > > > > > > order to avoid leaking worker file paths into the REST
> > > API,
> > > > > > which might
> > > > > > > > > > > > bother super security-conscious users.
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > >
> > > > > > > > > > > > Chris
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <
> > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes to keep compatibility we want a default
> > > implementation
> > > > > > for
> > > > > > > > > > > > > Converter.configs(), I've updated the KIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regarding worker plugins, the use case you described
> > > seems
> > > > > > valuable.
> > > > > > > > > > > > > I'd prefer not mixing worker and connector plugins on
> > > the
> > > > > > same
> > > > > > > > > > > > > endpoint but I agree using /plugins and
> > /worker-plugins
> > > > > > could be
> > > > > > > > > > > > > confusing.
> > > > > > > > > > > > >
> > > > > > > > > > > > > One alternative is to expose all connector-level
> > > plugins via
> > > > > > the
> > > > > > > > > > > > > existing /connector-plugins endpoint. In that case,
> > > we'd
> > > > > > need to keep
> > > > > > > > > > > > > the current JSON schema and not group plugins by
> > type.
> > > As
> > > > > > the current
> > > > > > > > > > > > > schema already has a type field for each entry, we'll
> > > still
> > > > > > be able to
> > > > > > > > > > > > > tell them apart. Then we can have /worker-plugins
> > and a
> > > > > > relatively
> > > > > > > > > > > > > clean API. What do you think?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Mickael
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think one potential use case for exposing
> > > worker-level
> > > > > > plugins is that
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > may make it easier to determine whether a worker is
> > > set up
> > > > > > correctly (the
> > > > > > > > > > > > > > current alternative requires looking through log
> > > files and
> > > > > > can be a
> > > > > > > > > > > > > little
> > > > > > > > > > > > > > tedious), and might even make it possible to
> > > automatically
> > > > > > identify
> > > > > > > > > > > > > > discrepancies within a cluster by diffing the
> > > contents of
> > > > > > that endpoint
> > > > > > > > > > > > > > across each worker. But I don't think this has to
> > be
> > > > > > addressed by the
> > > > > > > > > > > > > > current KIP; the only thing that bothers me a
> > little
> > > is
> > > > > > that "plugins" is
> > > > > > > > > > > > > > generic and it may confuse people down the road if
> > > we add
> > > > > > an endpoint for
> > > > > > > > > > > > > > worker-level plugins ("why is one just called
> > > 'plugins'
> > > > > > and the other one
> > > > > > > > > > > > > > is 'worker-plugins'?"). Probably not worth blocking
> > > on,
> > > > > > though.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Agreed that the suggestion for improved validation
> > > should
> > > > > > be made on the
> > > > > > > > > > > > > > KIP-802 thread.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I also noticed that the newly-proposed config
> > method
> > > for
> > > > > > the Converter
> > > > > > > > > > > > > > interface doesn't have a default implementation,
> > > making it
> > > > > > > > > > > > > > backwards-incompatible. Should we add a default
> > > > > > implementation that
> > > > > > > > > > > > > returns
> > > > > > > > > > > > > > either null or an empty ConfigDef?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <
> > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. If we want to expose worker plugins, I think
> > we
> > > > > > should do it via a
> > > > > > > > > > > > > > > separate endpoint. But to be honest, I'm not even
> > > sure I
> > > > > > see strong
> > > > > > > > > > > > > > > use cases for exposing them as they are either
> > > enabled
> > > > > > or not and
> > > > > > > > > > > > > > > can't be changed at runtime. So I'd prefer to
> > > stick to
> > > > > > "connector
> > > > > > > > > > > > > > > level" plugins in this KIP. Let me now if you
> > have
> > > use
> > > > > > cases, I'm open
> > > > > > > > > > > > > > > to reconsider this choice.
> > > > > > > > > > > > > > > I'll add that in the rejected alternatives
> > section
> > > for
> > > > > > now
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2. I remembered seeing issues in the past with
> > > multiple
> > > > > > plugin.path
> > > > > > > > > > > > > > > entries but I tried today and I was able to mix
> > and
> > > > > > match plugins from
> > > > > > > > > > > > > > > different paths. So my bad for getting confused.
> > > > > > > > > > > > > > > Then I agree, it makes more sense to group them
> > by
> > > > > > plugin type.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 3. Yes this should be covered in KIP-802:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 4. No particular reason. We can support both
> > > formats
> > > > > > like today. I've
> > > > > > > > > > > > > > > updated the KIP
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Mickael
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think the increase in scope here is great and
> > > the
> > > > > > added value
> > > > > > > > > > > > > certainly
> > > > > > > > > > > > > > > > justifies the proposed changes. I have some
> > > thoughts
> > > > > > but overall I
> > > > > > > > > > > > > like
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > direction this is going in now.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. The new /plugins endpoint is described as
> > > > > > containing "all plugins
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > are Connectors, Transformations, Converters,
> > > > > > HeaderConverters and
> > > > > > > > > > > > > > > > Predicates". So essentially, it looks like we
> > > want to
> > > > > > expose all
> > > > > > > > > > > > > plugins
> > > > > > > > > > > > > > > > that are configured on a per-connector basis,
> > but
> > > > > > exclude plugins
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > configured on a per-worker basis (such as
> > config
> > > > > > providers and REST
> > > > > > > > > > > > > > > > extensions). Do you think it may be valuable to
> > > expose
> > > > > > information on
> > > > > > > > > > > > > > > > worker-level plugins as well?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2. The description for the new /plugins
> > endpoint
> > > also
> > > > > > states that
> > > > > > > > > > > > > > > "Plugins
> > > > > > > > > > > > > > > > will be grouped by plugin.path. This will make
> > it
> > > > > > clear to users
> > > > > > > > > > > > > what's
> > > > > > > > > > > > > > > > available to use as it's not possible to use a
> > > > > > Connector from one
> > > > > > > > > > > > > path
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > Transformations from another.". Is this true? I
> > > > > > thought that
> > > > > > > > > > > > > Connect's
> > > > > > > > > > > > > > > > classloading made it possible to package
> > > > > > > > > > > > > > > > converters/transformations/predicates
> > completely
> > > > > > independently from
> > > > > > > > > > > > > each
> > > > > > > > > > > > > > > > other, and to reference them from
> > > > > > also-independently-packaged
> > > > > > > > > > > > > connectors.
> > > > > > > > > > > > > > > > If it turns out that this is the case, could we
> > > > > > consider
> > > > > > > > > > > > > restructuring
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > response to be grouped by plugin type instead
> > of
> > > by
> > > > > > classloader?
> > > > > > > > > > > > > There's
> > > > > > > > > > > > > > > > also the ungrouped format proposed in KIP-494 (
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > which we might consider as well.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 3. I think this can be left for a follow-up KIP
> > > if
> > > > > > necessary, but I'm
> > > > > > > > > > > > > > > > curious about your thoughts on adding new
> > > validate
> > > > > > methods to all
> > > > > > > > > > > > > > > > connector-level plugins that can be used
> > > similarly to
> > > > > > how the
> > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > Connector::validate method (
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > >
> > >
> > https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > > is used. This would allow for plugins to
> > perform
> > > > > > validation that's
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > sophisticated than what the ConfigDef is
> > capable
> > > of,
> > > > > > such as
> > > > > > > > > > > > > validating
> > > > > > > > > > > > > > > > combinations of properties like a hostname and
> > > > > > credentials for
> > > > > > > > > > > > > reaching
> > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > I know that at least Confluent's Avro,
> > protobuf,
> > > and
> > > > > > JSON schema
> > > > > > > > > > > > > > > converters
> > > > > > > > > > > > > > > > would benefit from this kind of feature. It's a
> > > little
> > > > > > tangential to
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > KIP (which at the moment is about discovering
> > > plugins
> > > > > > and their
> > > > > > > > > > > > > > > > configuration surfaces, as opposed to
> > validating
> > > > > > them), but I
> > > > > > > > > > > > > figured I'd
> > > > > > > > > > > > > > > > ask since we're going to be expanding the
> > > Converter
> > > > > > interface and it
> > > > > > > > > > > > > may
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > useful to tackle this while we're in the
> > > neighborhood.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 4. The description for the new
> > > > > > /plugins/<type>/<name>/configdef
> > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > states that "Name must be the fully qualified
> > > class
> > > > > > name of the
> > > > > > > > > > > > > plugin".
> > > > > > > > > > > > > > > > Any reason not to also support aliases (e.g.,
> > > > > > > > > > > > > "FileStreamSinkConnector"
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > "FileStreamSink" instead of
> > > > > > > > > > > > > > > >
> > > > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael
> > Maison <
> > > > > > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks all for the feedback!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Chris,
> > > > > > > > > > > > > > > > > I agree that fixing the current endpoint
> > helps
> > > a
> > > > > > lot. Thanks for
> > > > > > > > > > > > > > > > > raising these JIRAs and submitting a PR!
> > > > > > > > > > > > > > > > > However thinking about the issue further, I
> > > decided
> > > > > > to expand the
> > > > > > > > > > > > > > > > > scope of the KIP to cover all user-visible
> > > plugins.
> > > > > > > > > > > > > > > > > In practice, users want to know about all
> > > available
> > > > > > plugins not
> > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > connectors. This includes transformations,
> > > > > > converters,
> > > > > > > > > > > > > > > > > header_converters and predicates. As we also
> > > want to
> > > > > > retrieve
> > > > > > > > > > > > > > > > > configdef for these too, I think it makes
> > > sense to
> > > > > > introduce a new
> > > > > > > > > > > > > > > > > endpoint to do so. Alongside we obviously
> > need
> > > a new
> > > > > > endpoint for
> > > > > > > > > > > > > > > > > listing all plugins.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Gunnar,
> > > > > > > > > > > > > > > > > I took a look at exposing valid values via
> > the
> > > API.
> > > > > > I think the
> > > > > > > > > > > > > issue
> > > > > > > > > > > > > > > > > is that Validators don't expose a way to
> > > retrieve
> > > > > > valid values.
> > > > > > > > > > > > > > > > > Changing validators will have an impact on
> > all
> > > > > > components so I'd
> > > > > > > > > > > > > > > > > prefer to address this requirement in a
> > > separate
> > > > > > KIP. I agree this
> > > > > > > > > > > > > > > > > would be an interesting improvement and I'd
> > > happy to
> > > > > > write a KIP
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > it too.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I have updated the KIP accordingly. Let me
> > > know if
> > > > > > you have further
> > > > > > > > > > > > > > > > > feedback.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Mickael
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar
> > Morling
> > > > > > > > > > > > > > > > > <gu...@googlemail.com.invalid>
> > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I'm +1 for adding a GET endpoint for
> > > obtaining
> > > > > > config
> > > > > > > > > > > > > definitions. It
> > > > > > > > > > > > > > > > > > always felt odd to me that one has to issue
> > > a PUT
> > > > > > for that
> > > > > > > > > > > > > purpose.
> > > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > > nothing else, it'd be better in terms of
> > > > > > discoverability of the
> > > > > > > > > > > > > KC
> > > > > > > > > > > > > > > REST
> > > > > > > > > > > > > > > > > API.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > One additional feature request I'd have is
> > to
> > > > > > expose the valid
> > > > > > > > > > > > > enum
> > > > > > > > > > > > > > > > > > constants for enum-typed options. That'll
> > > help to
> > > > > > display the
> > > > > > > > > > > > > values
> > > > > > > > > > > > > > > in a
> > > > > > > > > > > > > > > > > > drop-down or via radio buttons in a UI,
> > give
> > > us
> > > > > > tab completion in
> > > > > > > > > > > > > > > kcctl,
> > > > > > > > > > > > > > > > > > etc.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > --Gunnar
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb
> > > Chris
> > > > > > Egerton
> > > > > > > > > > > > > > > > > > <ch...@confluent.io.invalid>:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Viktor,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It sounds like there are three major
> > > points here
> > > > > > in favor of a
> > > > > > > > > > > > > new
> > > > > > > > > > > > > > > GET
> > > > > > > > > > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1. You cannot issue a blank ("dummy")
> > > request
> > > > > > for sink
> > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > because a
> > > > > > > > > > > > > > > > > > > topic list/topic regex has to be supplied
> > > > > > (otherwise the PUT
> > > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > > > returns a 500 response)
> > > > > > > > > > > > > > > > > > > 2. A dummy request still triggers custom
> > > > > > validations by the
> > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > which may be best to avoid if we know for
> > > sure
> > > > > > that the config
> > > > > > > > > > > > > > > isn't
> > > > > > > > > > > > > > > > > worth
> > > > > > > > > > > > > > > > > > > validating yet
> > > > > > > > > > > > > > > > > > > 3. It's more ergonomic and intuitive to
> > be
> > > able
> > > > > > to issue a GET
> > > > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > > > without having to give a dummy connector
> > > config
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > With regards to 1, this is actually a bug
> > > in
> > > > > > Connect (
> > > > > > > > > > > > > > > > > > >
> > > > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix
> > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > > implemented and awaiting committer
> > review (
> > > > > > > > > > > > > > > > > > >
> > https://github.com/apache/kafka/pull/11369).
> > > I
> > > > > > think it'd be
> > > > > > > > > > > > > > > better to
> > > > > > > > > > > > > > > > > > > focus on fixing this bug in general
> > > instead of
> > > > > > implementing a
> > > > > > > > > > > > > new
> > > > > > > > > > > > > > > REST
> > > > > > > > > > > > > > > > > > > endpoint in order to allow people to work
> > > around
> > > > > > it.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > With regards to 2, this is technically
> > > possible
> > > > > > but I'm unsure
> > > > > > > > > > > > > > > it'd be
> > > > > > > > > > > > > > > > > too
> > > > > > > > > > > > > > > > > > > common out in the wild given that most
> > > > > > validations that could
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > expensive
> > > > > > > > > > > > > > > > > > > would involve things like connecting to a
> > > > > > database, checking
> > > > > > > > > > > > > if a
> > > > > > > > > > > > > > > cloud
> > > > > > > > > > > > > > > > > > > storage bucket exists, etc., none of
> > which
> > > are
> > > > > > possible without
> > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > configuration properties from the user
> > (db
> > > > > > hostname, bucket
> > > > > > > > > > > > > name,
> > > > > > > > > > > > > > > > > etc.).
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > With regards to 3, I do agree that it'd
> > be
> > > > > > easier for people
> > > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > > UIs
> > > > > > > > > > > > > > > > > > > to have a GET API to work against. I'm
> > > just not
> > > > > > sure it's
> > > > > > > > > > > > > worth the
> > > > > > > > > > > > > > > > > > > additional implementation, testing, and
> > > > > > maintenance burden. If
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > > possible to issue a PUT request without
> > > > > > unexpected 500s for
> > > > > > > > > > > > > invalid
> > > > > > > > > > > > > > > > > > > configs, would that suffice? AFAICT it'd
> > > > > > basically be as
> > > > > > > > > > > > > simple as
> > > > > > > > > > > > > > > > > issuing
> > > > > > > > > > > > > > > > > > > a PUT request with a dummy body
> > consisting
> > > of
> > > > > > nothing except
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > class (which at this point we might even
> > > make
> > > > > > unnecessary and
> > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > > automatically replace with the connector
> > > class
> > > > > > from the URL)
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > then
> > > > > > > > > > > > > > > > > > > filtering the response to just grab the
> > > > > > "definition" field of
> > > > > > > > > > > > > each
> > > > > > > > > > > > > > > > > element
> > > > > > > > > > > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor
> > > > > > Somogyi-Vass <
> > > > > > > > > > > > > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Folks,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I too think this would be a very useful
> > > > > > feature. Some of our
> > > > > > > > > > > > > > > > > management
> > > > > > > > > > > > > > > > > > > > applications would provide a wizard for
> > > > > > creating connectors.
> > > > > > > > > > > > > In
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > scenario the user basically would fill
> > > out a
> > > > > > sample
> > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > generated
> > > > > > > > > > > > > > > > > > > > by the UI which would send it back to
> > > Connect
> > > > > > for validation
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > eventually
> > > > > > > > > > > > > > > > > > > > create a new connector. The first part
> > > of this
> > > > > > workflow can
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > enhanced
> > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > we had an API that can return the
> > > > > > configuration definition
> > > > > > > > > > > > > of the
> > > > > > > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > > > > type of connector as the UI application
> > > would
> > > > > > be able to
> > > > > > > > > > > > > > > generate a
> > > > > > > > > > > > > > > > > > > sample
> > > > > > > > > > > > > > > > > > > > for the user based on that (nicely
> > drawn
> > > > > > diagram:
> > > > > > > > > > > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > > > > > > > > > > The
> > > > > > connector-plugins/{connectorType}/config/validate API
> > > > > > > > > > > > > > > essentially
> > > > > > > > > > > > > > > > > > > works
> > > > > > > > > > > > > > > > > > > > and returns the data that we need,
> > > however it
> > > > > > is a HTTP PUT
> > > > > > > > > > > > > API
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > > > bit unintuitive for a fetch-like
> > > functionality
> > > > > > and also
> > > > > > > > > > > > > > > functionally
> > > > > > > > > > > > > > > > > > > > different as it validates the given
> > > (dummy)
> > > > > > request. In case
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > sink
> > > > > > > > > > > > > > > > > > > > connectors one would need to also
> > > provide a
> > > > > > topic name.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > A suggestion for the KIP: I think it
> > can
> > > be
> > > > > > useful to return
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > groups and the connector class' name
> > > similarly
> > > > > > to the
> > > > > > > > > > > > > validate
> > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > case any frontend needs them (and also
> > > the
> > > > > > response would be
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > validate API but simpler).
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Viktor
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne
> > > Dolan <
> > > > > > > > > > > > > > > ryannedolan@gmail.com>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think it'd be worth adding a GET
> > > version,
> > > > > > fwiw. Could be
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > > > handler
> > > > > > > > > > > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael
> > > Maison
> > > > > > <
> > > > > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > You're right, you can achieve the
> > > same
> > > > > > functionality
> > > > > > > > > > > > > using
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > > > > > > > > > > In my mind it was only for
> > > validation once
> > > > > > you have
> > > > > > > > > > > > > build a
> > > > > > > > > > > > > > > > > > > > > > configuration but when used with an
> > > empty
> > > > > > configuration,
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > basically
> > > > > > > > > > > > > > > > > > > > > > serves the same purpose as the
> > > proposed
> > > > > > new endpoint.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think it's a bit easier to use a
> > > GET
> > > > > > endpoint but I
> > > > > > > > > > > > > don't
> > > > > > > > > > > > > > > > > think it
> > > > > > > > > > > > > > > > > > > > > > really warrants a different
> > endpoint.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM
> > Chris
> > > > > > Egerton
> > > > > > > > > > > > > > > > > > > > > > <ch...@confluent.io.invalid>
> > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I'm wondering about the use case
> > > here.
> > > > > > The motivation
> > > > > > > > > > > > > > > section
> > > > > > > > > > > > > > > > > > > states
> > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > "Connect does not provide a way
> > to
> > > see
> > > > > > what
> > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > requires. Instead users have to
> > go
> > > look
> > > > > > at the
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > documentation
> > > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > in the worst case, look directly
> > > at the
> > > > > > connector
> > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > code.",
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > with this KIP, "users will be
> > able
> > > to
> > > > > > discover the
> > > > > > > > > > > > > required
> > > > > > > > > > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > > > > > > > > > for connectors installed in a
> > > Connect
> > > > > > cluster" and
> > > > > > > > > > > > > "tools
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > generate wizards for configuring
> > > and
> > > > > > starting
> > > > > > > > > > > > > connectors".
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > > > > > > > > > > endpoint not address these
> > points?
> > > What
> > > > > > will the
> > > > > > > > > > > > > > > newly-proposed
> > > > > > > > > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > > > > > > > allow users to do that they will
> > > not
> > > > > > already be able
> > > > > > > > > > > > > to do
> > > > > > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM
> > > Mickael
> > > > > > Maison <
> > > > > > > > > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I've created KIP-769 to expose
> > > > > > connector
> > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > definitions
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Please take a look and let me
> > > know if
> > > > > > you have any
> > > > > > > > > > > > > > > feedback.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > >
> > >
> > >
> >

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Chris Egerton <ch...@confluent.io.INVALID>]

Hi all,

With regards to Tom's question about possible uses of the version string,
one of the most useful applications I've found for it so far has been
debugging connector issues, especially based on captured logs. Using the
version string to identify a git tag containing the exact source code for a
connector, and then using that as a reference when poring over log files
that usually contain class names, method names, and/or line numbers, is a
great way to construct a timeline of events and understand how things may
have gone wrong. I don't think we can require this as part of the contract
for the version method, but at least identifying this use case in the docs
for it could be useful to help connector/SMT/converter/etc. developers
understand why they should care about what this method returns and how it
could be beneficial to themselves and their users down the road.

One other potential use case that comes up is that, if/when Connect
supports loading multiple versions of the same plugin type on a single
worker, the version string can be used to identify plugins that share the
same name and type but which should be exposed individually to users. If we
want to leave room for this possibility, perhaps part of the contract for
the method can be that "any two releases of the same plugin but with
different behavior should return different values from their version
method".

One final note--I've just realized that, with the introduction of a default
implementation for the Versioned interface, we've made Connector::version
and ConnectRestExtension::version optional to implement, whereas before it
was required that instances of these plugins explicitly implement that
method. Should we allow this? An alternative could be to add a new
"PossiblyVersioned" interface (name obviously subject to change) that
extends the current Versioned interface and contains a default
implementation that returns the "undefined" string. The to-be-updated
plugin interfaces could extend from this new "PossiblyVersioned" interface,
and the plugin interfaces that extend from the current "Versioned"
interface could be left unchanged.

Cheers,

Chris

On Wed, Jan 19, 2022 at 10:04 AM Tom Bentley <tb...@redhat.com> wrote:

> Hi Mickael,
>
> Thanks for the KIP. Overall I think this looks good.
>
> I have a small concern about the proliferation of Versioned. Currently this
> interface has a really weak contract. There's no requirement for versions
> to be comparable, or documentation about when versions should be changed by
> authors of plugins which extend this interface. For example, if I release
> version 1.2.3 of my SMT should the version be 1.2.3 or something else? Is
> compatibility implied for different versions of my jar file which return
> the same result from version()? Because of this it's not really clear what
> the consumer of the version string is supposed to do with it. So I think if
> we're going to be using Versioned more extensively it would be good to make
> the expectations clearer, for both authors and people wanting to reason
> about versions.
>
> Many thanks,
>
> Tom
>
> On Mon, 6 Dec 2021 at 09:56, Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Hi Arjun,
> >
> > Thank you for taking a look at this KIP.
> >
> > 1.a Yes this is exactly what this KIP is proposing.
> >
> > 1.b I prefer returning all plugins together instead of having different
> > paths.
> >
> > 2. While we seem to agree it would be nice to expose worker plugins, I
> > don't think we've reached a consensus on the right way to do it. For
> > that reason, I'm dropping this feature from this KIP. We can address
> > this issue in another KIP. I have updated the KIP.
> >
> > Thanks,
> > Mickael
> >
> > On Mon, Dec 6, 2021 at 10:15 AM Mickael Maison <mickael.maison@gmail.com
> >
> > wrote:
> > >
> > > Thanks Randall, updated.
> > >
> > > On Sun, Dec 5, 2021 at 6:05 AM Arjun Satish <ar...@gmail.com>
> > wrote:
> > > >
> > > > hey folks,
> > > >
> > > > Thanks a lot for the KIP and the discussions. Here are a couple of
> > thoughts
> > > > (apologies if I missed some point in this thread).
> > > >
> > > > 1. it seems like we are changing the behavior of /connector-plugins
> by
> > now
> > > > returning the list of transforms, converters as well? This would
> break
> > > > backward compatibility, and clients would need to track what version
> of
> > > > Connect workers they are querying to keep their experience
> consistent.
> > I
> > > > wonder if we can keep the old behavior, with one of the following
> > methods:
> > > >
> > > > 1a. We add an "expand" multi-query to the /connector-plugin endpoint
> > like
> > > > how it was done in KIP-465 [1]. This would preserve backward
> > compatibility,
> > > > as well as allow you to get consolidated output. Acceptable values
> for
> > the
> > > > new param could be: "all" (to show all plugins), or
> > > > "connectors|transforms|predicates" that show a subset of the plugin
> > types.
> > > >
> > > > 1b. Query params might get crummy to extend. Alternatively, we could
> > have
> > > > an additional fine-grained paths for each plugin:
> > > >
> > > > # to list all the source and sink plugins
> > > > /connector-plugins/connectors
> > > >
> > > > # to list all the SMT plugins
> > > > /connector-plugins/transformations
> > > >
> > > > # to list all the predicate plugins
> > > > /connector-plugins/predicates
> > > >
> > > > # to list all the converter plugins
> > > > /connector-plugins/converters
> > > >
> > > > # to list all the header converter plugins
> > > > /connector-plugins/header-converters
> > > >
> > > > This allows us to cleanly add new types of plugins down the road, and
> > > > naturally extend to displaying attributes of these objects. Configs,
> > for
> > > > example, can be accessed with
> > > >
> > > >
> >
> http://localhost:8083/connector-plugins/transformations/Cast$Value/config
> > > >
> > > > A migration path (for both the suggestions above) would be to add a
> > > > deprecation notice on the current behavior of /connector-plugins
> > (where,
> > > > after this PR is merged, both /connector-plugins and
> > > > /connector-plugins/connectors return the same output). And starting
> > from
> > > > the next major version release, /connector-plugins redirects to
> > > > /connector-plugins/connectors (making it super easy for clients to
> > catch
> > > > up).
> > > >
> > > > 2. Regarding /worker-plugins, I agree that it is useful to have the
> > REST
> > > > API inform the state of the REST extensions, and Config Providers.
> > However,
> > > > users of the Connect clusters will rarely need to know how a config
> > > > provider or REST extension is configured to construct data pipelines.
> > It's
> > > > mostly an admin task, AFAICT. Also, we have also seen these problems
> in
> > > > Connect clusters where one host is somewhat differently set up than
> > other
> > > > (contents of plugin path, classpath, local worker configuration
> files,
> > jvm
> > > > versions, GC settings) that surface as issues in Connector behavior.
> > > > Overall, I'm wondering if we should just tackle this problem in a
> > separate
> > > > KIP, that evaluates all of these cases. All of this might nicely fall
> > into
> > > > under the /admin endpoint and allow us to expose detailed info on the
> > state
> > > > of the worker (maybe a /admin/worker/ that returns all of this info).
> > > >
> > > > Thanks,
> > > >
> > > > [1]
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-465%3A+Add+Consolidated+Connector+Endpoint+to+Connect+REST+API
> > > >
> > > >
> > > > On Sat, Dec 4, 2021 at 10:29 AM Randall Hauch <rh...@gmail.com>
> > wrote:
> > > >
> > > > > Thanks, Mickael. Can we call out these as literals? For example:
> > > > >
> > > > > > The response structure of the objects in the array remain
> > unchanged, but
> > > > > this proposal does add "converter", "header_converter" and
> > "transformation"
> > > > > to the allowed values for the "type" field, in addition to the
> > existing
> > > > > "sink" and "source" values.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Randall
> > > > >
> > > > > On Sat, Dec 4, 2021 at 4:58 AM Mickael Maison <
> > mickael.maison@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Hi Randall,
> > > > > >
> > > > > > Sure. The types are sink, source, converter, header_converter,
> > > > > > transformation and predicate. I've updated the KIP.
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > > On Fri, Dec 3, 2021 at 10:14 PM Randall Hauch <rh...@gmail.com>
> > wrote:
> > > > > > >
> > > > > > > One more thing, inline below:
> > > > > > >
> > > > > > > Randall
> > > > > > >
> > > > > > > On Fri, Dec 3, 2021 at 10:38 AM Mickael Maison <
> > > > > mickael.maison@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Thanks Randall for the feedback!
> > > > > > > >
> > > > > > > > 1. Done
> > > > > > > >
> > > > > > > > 2. Yes that makes sense. I've updated the KIP
> > > > > > > >
> > > > > > > > 3.a I've updated to explicitly mention the plugin can be
> > provided
> > > > > > > > using the fully qualified class or the alias.
> > > > > > > > 3.b Only concrete classes will be returned. I made the
> mistake
> > when I
> > > > > > > > crafted the examples by hand. Fixed
> > > > > > >
> > > > > > > Can we also enumerate all of the values that can be included in
> > the
> > > > > > > "type" field in the response from the "GET
> > > > > > > /connector-plugins?connectorsOnly=false"?
> > > > > > >
> > > > > > > >
> > > > > > > > 4. If a plugin does not override config(), the response is an
> > empty
> > > > > > > > array. I've updated the KIP
> > > > > > > >
> > > > > > > > 5. I have updated the compatibility section with more details
> > > > > > > >
> > > > > > > > 6. Can you clarify what you mean? KIP-507 was for securing
> > internal
> > > > > > > > REST endpoints.
> > > > > > > >
> > > > > > > > 7. I don't think it is necessary. I think the use case Chris
> > > > > mentioned
> > > > > > > > could be useful for admins. I'm interested in exposing the
> > > > > > > > connector-level plugins and their configs. I thought it was
> > easy to
> > > > > > > > add an API for worker plugins while working in this area but
> > if there
> > > > > > > > isn't a strong consensus, I'm happy to drop it.
> > > > > > > >
> > > > > > > > Mickael
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch <
> rhauch@gmail.com
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > Thanks for the KIP, Mickael. And thanks to everyone else
> for
> > the
> > > > > good
> > > > > > > > > discussion so far.
> > > > > > > > >
> > > > > > > > > Overall this is pretty good, but I do have a few
> > > > > questions/comments:
> > > > > > > > > 1. Can we include the HTTP method/verb in each of the
> > endpoints in
> > > > > the
> > > > > > > > > "Public Interfaces" section?
> > > > > > > > > 2. The sample output for `GET
> > > > > /connector-plugins?connectorsOnly=false`
> > > > > > > > > includes a `version` field for the connector plugin, but
> not
> > for
> > > > > the
> > > > > > > > > transform or converter plugins. Neither the
> `Transformation`
> > or
> > > > > > > > > `Converter` interfaces have a version nor implement the
> > `Versioned`
> > > > > > > > > interface, so should we add the `Versioned` interface to
> the
> > > > > > > > > `Transformation` and `Converter` interfaces and add a
> default
> > > > > method
> > > > > > > > > that returns "unknown"? Yes, we could do that in a separate
> > KIP,
> > > > > but I
> > > > > > > > > think it's appropriate to do it in this KIP for a few
> > reasons.
> > > > > First,
> > > > > > > > > we're already modifying the `Converter` interface in this
> > KIP and
> > > > > it
> > > > > > > > > would be quite straightforward to also extend the
> `Versioned`
> > > > > > > > > interface and add a default method; then doing that for
> > > > > > > > > `Transformation` would also be relatively straightforward.
> > Second,
> > > > > > > > > this KIP provides the justification for adding a version to
> > the
> > > > > > > > > `Transformation` and `Converter` interfaces.
> > > > > > > > > 3. In the "Public Interfaces" section, there are two
> > > > > inconsistencies
> > > > > > > > > between the `GET /connector-plugins` and `GET
> > > > > > > > > /connector-plugins/<plugin>/config` methods. Both will make
> > it
> > > > > > > > > difficult to use the second method, since the input to the
> > second
> > > > > > > > > method is not directly found in the response from the first
> > method:
> > > > > > > > >  a. The example for `GET
> > /connector-plugins/Cast$Value/config` uses
> > > > > > > > > short name, or what Connect often refers to as the alias,
> > > > > "Cast$Value"
> > > > > > > > > for the plugin, rather than the fully-qualified name
> > > > > > > > > "org.apache.kafka.connect.transforms.Cast$Value". The KIP
> > should
> > > > > > > > > specify whether it will support both the shortened name and
> > the
> > > > > > > > > fully-qualified name, or whether only one of these will be
> > > > > supported.
> > > > > > > > > Not supporting the fully-qualified name will make it
> > impossible to
> > > > > use
> > > > > > > > > the response from the first method as an input to the
> second.
> > > > > > > > >  a. The example for the list method returns the
> > `Transformation`
> > > > > base
> > > > > > > > > and abstract classes (e.g.,
> > > > > > > > > "org.apache.kafka.connect.transforms.Cast") rather than the
> > > > > concrete
> > > > > > > > > classes (e.g.,
> > "org.apache.kafka.connect.transforms.Cast$Value" and
> > > > > > > > > "org.apache.kafka.connect.transforms.Cast$Key"), while the
> > `GET
> > > > > > > > > /connector-plugins/<plugin>/config` will require the
> concrete
> > > > > class.
> > > > > > > > > 4. The "Public Interfaces" section's description of the
> `GET
> > > > > > > > > /connector-plugins/<plugin>/config` method should probably
> > specify
> > > > > the
> > > > > > > > > expected behavior for `Converter` implementations that
> don't
> > > > > override
> > > > > > > > > the new `config()` method.
> > > > > > > > > 5. The "Compatibility, Deprecation, and Migration Plan"
> > section
> > > > > should
> > > > > > > > > describe whether older `Converter` implementations
> (compiled
> > with
> > > > > > > > > older versions of the Connect API without Converter
> extending
> > > > > > > > > Configurable) will be able to be installed into newer
> Connect
> > > > > > > > > runtimes, and what the behavior will be for the `GET
> > > > > > > > > /connector-plugins/<plugin>/config` method. (Although my
> > suggestion
> > > > > > > > > for #5 will partially address the latter, we still need to
> > talk
> > > > > about
> > > > > > > > > it in terms of compatibility.)
> > > > > > > > > 6. Can we add a small section addressing the security
> > implications
> > > > > of
> > > > > > > > > these new APIs? KIP-507 [1] applied to APIs that existed at
> > the
> > > > > time,
> > > > > > > > > and this KIP should at least mention how it aligns with
> that
> > > > > > > > > mechanism.
> > > > > > > > > 7. Do we really think adding `GET /worker-plugins` is
> > necessary?
> > > > > While
> > > > > > > > > I understand the reasoning, I'm less convinced that it's
> > worth
> > > > > adding
> > > > > > > > > to the public API. First, an administrator still has to
> > modify the
> > > > > > > > > worker config for all workers to take advantage of that.
> > Second,
> > > > > > > > > adding this method increases the surface area and therefore
> > risk of
> > > > > > > > > additional attack vectors, despite not really being able to
> > use the
> > > > > > > > > resulting information in any other part of the API.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > >
> > > > > > > > > Randall
> > > > > > > > >
> > > > > > > > > [1]
> > > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Dec 1, 2021 at 7:52 AM Mickael Maison <
> > > > > mickael.maison@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Chris,
> > > > > > > > > >
> > > > > > > > > > Thanks again for the feedback! I've updated the KIP based
> > on our
> > > > > last
> > > > > > > > > > discussions. I've decided to include the new endpoint for
> > worker
> > > > > > > > > > plugins.
> > > > > > > > > >
> > > > > > > > > > 1. Yes I agree, it's best to gate the new behavior.
> > > > > > > > > > 2. Yes, it was a remnant from the original proposal. I've
> > now
> > > > > removed
> > > > > > > > > > the location field.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Mickael
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
> > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Mickael,
> > > > > > > > > > >
> > > > > > > > > > > I think that's a great idea! I especially like how we
> can
> > > > > establish the
> > > > > > > > > > > expectation that any plugin type that appears in the
> > response
> > > > > from the GET
> > > > > > > > > > > /connector-plugins endpoint will have a corresponding
> GET
> > > > > > > > > > > /connector-plugins/<type>/config endpoint, but (if we
> > decide
> > > > > to add them in
> > > > > > > > > > > the future), worker plugins won't be expected to expose
> > this
> > > > > kind of
> > > > > > > > > > > information and the different root path helps give a
> > decent
> > > > > hint about this.
> > > > > > > > > > >
> > > > > > > > > > > I also like the choice to return an empty ConfigDef
> from
> > > > > Converter::config
> > > > > > > > > > > instead of null.
> > > > > > > > > > >
> > > > > > > > > > > Two things come to mind:
> > > > > > > > > > >
> > > > > > > > > > > 1. We may want to gate this behind a URL query
> parameter
> > > > > (maybe something
> > > > > > > > > > > like "connectorsOnly") that defaults to the old
> behavior
> > in
> > > > > order to avoid
> > > > > > > > > > > breaking existing tools such as programmatic UIs that
> > use the
> > > > > endpoint
> > > > > > > > > > > today to discover the connectors that can be created by
> > the
> > > > > user. We can
> > > > > > > > > > > even plan to change the default for that parameter to
> the
> > > > > newly-proposed
> > > > > > > > > > > behavior in the next major release, which should give
> > people
> > > > > enough time to
> > > > > > > > > > > either adapt to the expanded response format or add the
> > query
> > > > > parameter to
> > > > > > > > > > > their tooling.
> > > > > > > > > > >
> > > > > > > > > > > 2. The existing GET /connector-plugins endpoint doesn't
> > > > > contain information
> > > > > > > > > > > on the location of the plugin on the worker's file
> > system. Do
> > > > > you think we
> > > > > > > > > > > should still include this info in the new response
> > format?
> > > > > Correct me if
> > > > > > > > > > > I'm wrong but it seems it may have been proposed
> > originally to
> > > > > help prevent
> > > > > > > > > > > already-addressed bugs in Connect classloading from
> > striking;
> > > > > all else
> > > > > > > > > > > equal, I'd personally err on the side of leaving this
> > info out
> > > > > or at least
> > > > > > > > > > > reducing permitted values for it to just "classpath" or
> > > > > "plugin path" in
> > > > > > > > > > > order to avoid leaking worker file paths into the REST
> > API,
> > > > > which might
> > > > > > > > > > > bother super security-conscious users.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > Chris
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <
> > > > > mickael.maison@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > >
> > > > > > > > > > > > Yes to keep compatibility we want a default
> > implementation
> > > > > for
> > > > > > > > > > > > Converter.configs(), I've updated the KIP.
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding worker plugins, the use case you described
> > seems
> > > > > valuable.
> > > > > > > > > > > > I'd prefer not mixing worker and connector plugins on
> > the
> > > > > same
> > > > > > > > > > > > endpoint but I agree using /plugins and
> /worker-plugins
> > > > > could be
> > > > > > > > > > > > confusing.
> > > > > > > > > > > >
> > > > > > > > > > > > One alternative is to expose all connector-level
> > plugins via
> > > > > the
> > > > > > > > > > > > existing /connector-plugins endpoint. In that case,
> > we'd
> > > > > need to keep
> > > > > > > > > > > > the current JSON schema and not group plugins by
> type.
> > As
> > > > > the current
> > > > > > > > > > > > schema already has a type field for each entry, we'll
> > still
> > > > > be able to
> > > > > > > > > > > > tell them apart. Then we can have /worker-plugins
> and a
> > > > > relatively
> > > > > > > > > > > > clean API. What do you think?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Mickael
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think one potential use case for exposing
> > worker-level
> > > > > plugins is that
> > > > > > > > > > > > it
> > > > > > > > > > > > > may make it easier to determine whether a worker is
> > set up
> > > > > correctly (the
> > > > > > > > > > > > > current alternative requires looking through log
> > files and
> > > > > can be a
> > > > > > > > > > > > little
> > > > > > > > > > > > > tedious), and might even make it possible to
> > automatically
> > > > > identify
> > > > > > > > > > > > > discrepancies within a cluster by diffing the
> > contents of
> > > > > that endpoint
> > > > > > > > > > > > > across each worker. But I don't think this has to
> be
> > > > > addressed by the
> > > > > > > > > > > > > current KIP; the only thing that bothers me a
> little
> > is
> > > > > that "plugins" is
> > > > > > > > > > > > > generic and it may confuse people down the road if
> > we add
> > > > > an endpoint for
> > > > > > > > > > > > > worker-level plugins ("why is one just called
> > 'plugins'
> > > > > and the other one
> > > > > > > > > > > > > is 'worker-plugins'?"). Probably not worth blocking
> > on,
> > > > > though.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Agreed that the suggestion for improved validation
> > should
> > > > > be made on the
> > > > > > > > > > > > > KIP-802 thread.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I also noticed that the newly-proposed config
> method
> > for
> > > > > the Converter
> > > > > > > > > > > > > interface doesn't have a default implementation,
> > making it
> > > > > > > > > > > > > backwards-incompatible. Should we add a default
> > > > > implementation that
> > > > > > > > > > > > returns
> > > > > > > > > > > > > either null or an empty ConfigDef?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Chris
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <
> > > > > mickael.maison@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. If we want to expose worker plugins, I think
> we
> > > > > should do it via a
> > > > > > > > > > > > > > separate endpoint. But to be honest, I'm not even
> > sure I
> > > > > see strong
> > > > > > > > > > > > > > use cases for exposing them as they are either
> > enabled
> > > > > or not and
> > > > > > > > > > > > > > can't be changed at runtime. So I'd prefer to
> > stick to
> > > > > "connector
> > > > > > > > > > > > > > level" plugins in this KIP. Let me now if you
> have
> > use
> > > > > cases, I'm open
> > > > > > > > > > > > > > to reconsider this choice.
> > > > > > > > > > > > > > I'll add that in the rejected alternatives
> section
> > for
> > > > > now
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. I remembered seeing issues in the past with
> > multiple
> > > > > plugin.path
> > > > > > > > > > > > > > entries but I tried today and I was able to mix
> and
> > > > > match plugins from
> > > > > > > > > > > > > > different paths. So my bad for getting confused.
> > > > > > > > > > > > > > Then I agree, it makes more sense to group them
> by
> > > > > plugin type.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3. Yes this should be covered in KIP-802:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 4. No particular reason. We can support both
> > formats
> > > > > like today. I've
> > > > > > > > > > > > > > updated the KIP
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Mickael
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think the increase in scope here is great and
> > the
> > > > > added value
> > > > > > > > > > > > certainly
> > > > > > > > > > > > > > > justifies the proposed changes. I have some
> > thoughts
> > > > > but overall I
> > > > > > > > > > > > like
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > direction this is going in now.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. The new /plugins endpoint is described as
> > > > > containing "all plugins
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > are Connectors, Transformations, Converters,
> > > > > HeaderConverters and
> > > > > > > > > > > > > > > Predicates". So essentially, it looks like we
> > want to
> > > > > expose all
> > > > > > > > > > > > plugins
> > > > > > > > > > > > > > > that are configured on a per-connector basis,
> but
> > > > > exclude plugins
> > > > > > > > > > > > that
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > configured on a per-worker basis (such as
> config
> > > > > providers and REST
> > > > > > > > > > > > > > > extensions). Do you think it may be valuable to
> > expose
> > > > > information on
> > > > > > > > > > > > > > > worker-level plugins as well?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2. The description for the new /plugins
> endpoint
> > also
> > > > > states that
> > > > > > > > > > > > > > "Plugins
> > > > > > > > > > > > > > > will be grouped by plugin.path. This will make
> it
> > > > > clear to users
> > > > > > > > > > > > what's
> > > > > > > > > > > > > > > available to use as it's not possible to use a
> > > > > Connector from one
> > > > > > > > > > > > path
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > Transformations from another.". Is this true? I
> > > > > thought that
> > > > > > > > > > > > Connect's
> > > > > > > > > > > > > > > classloading made it possible to package
> > > > > > > > > > > > > > > converters/transformations/predicates
> completely
> > > > > independently from
> > > > > > > > > > > > each
> > > > > > > > > > > > > > > other, and to reference them from
> > > > > also-independently-packaged
> > > > > > > > > > > > connectors.
> > > > > > > > > > > > > > > If it turns out that this is the case, could we
> > > > > consider
> > > > > > > > > > > > restructuring
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > response to be grouped by plugin type instead
> of
> > by
> > > > > classloader?
> > > > > > > > > > > > There's
> > > > > > > > > > > > > > > also the ungrouped format proposed in KIP-494 (
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > which we might consider as well.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 3. I think this can be left for a follow-up KIP
> > if
> > > > > necessary, but I'm
> > > > > > > > > > > > > > > curious about your thoughts on adding new
> > validate
> > > > > methods to all
> > > > > > > > > > > > > > > connector-level plugins that can be used
> > similarly to
> > > > > how the
> > > > > > > > > > > > existing
> > > > > > > > > > > > > > > Connector::validate method (
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > >
> >
> https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > > > > > > > > > > > )
> > > > > > > > > > > > > > > is used. This would allow for plugins to
> perform
> > > > > validation that's
> > > > > > > > > > > > more
> > > > > > > > > > > > > > > sophisticated than what the ConfigDef is
> capable
> > of,
> > > > > such as
> > > > > > > > > > > > validating
> > > > > > > > > > > > > > > combinations of properties like a hostname and
> > > > > credentials for
> > > > > > > > > > > > reaching
> > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > I know that at least Confluent's Avro,
> protobuf,
> > and
> > > > > JSON schema
> > > > > > > > > > > > > > converters
> > > > > > > > > > > > > > > would benefit from this kind of feature. It's a
> > little
> > > > > tangential to
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > KIP (which at the moment is about discovering
> > plugins
> > > > > and their
> > > > > > > > > > > > > > > configuration surfaces, as opposed to
> validating
> > > > > them), but I
> > > > > > > > > > > > figured I'd
> > > > > > > > > > > > > > > ask since we're going to be expanding the
> > Converter
> > > > > interface and it
> > > > > > > > > > > > may
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > useful to tackle this while we're in the
> > neighborhood.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 4. The description for the new
> > > > > /plugins/<type>/<name>/configdef
> > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > states that "Name must be the fully qualified
> > class
> > > > > name of the
> > > > > > > > > > > > plugin".
> > > > > > > > > > > > > > > Any reason not to also support aliases (e.g.,
> > > > > > > > > > > > "FileStreamSinkConnector"
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > "FileStreamSink" instead of
> > > > > > > > > > > > > > >
> > > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael
> Maison <
> > > > > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks all for the feedback!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Chris,
> > > > > > > > > > > > > > > > I agree that fixing the current endpoint
> helps
> > a
> > > > > lot. Thanks for
> > > > > > > > > > > > > > > > raising these JIRAs and submitting a PR!
> > > > > > > > > > > > > > > > However thinking about the issue further, I
> > decided
> > > > > to expand the
> > > > > > > > > > > > > > > > scope of the KIP to cover all user-visible
> > plugins.
> > > > > > > > > > > > > > > > In practice, users want to know about all
> > available
> > > > > plugins not
> > > > > > > > > > > > only
> > > > > > > > > > > > > > > > connectors. This includes transformations,
> > > > > converters,
> > > > > > > > > > > > > > > > header_converters and predicates. As we also
> > want to
> > > > > retrieve
> > > > > > > > > > > > > > > > configdef for these too, I think it makes
> > sense to
> > > > > introduce a new
> > > > > > > > > > > > > > > > endpoint to do so. Alongside we obviously
> need
> > a new
> > > > > endpoint for
> > > > > > > > > > > > > > > > listing all plugins.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Gunnar,
> > > > > > > > > > > > > > > > I took a look at exposing valid values via
> the
> > API.
> > > > > I think the
> > > > > > > > > > > > issue
> > > > > > > > > > > > > > > > is that Validators don't expose a way to
> > retrieve
> > > > > valid values.
> > > > > > > > > > > > > > > > Changing validators will have an impact on
> all
> > > > > components so I'd
> > > > > > > > > > > > > > > > prefer to address this requirement in a
> > separate
> > > > > KIP. I agree this
> > > > > > > > > > > > > > > > would be an interesting improvement and I'd
> > happy to
> > > > > write a KIP
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > it too.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have updated the KIP accordingly. Let me
> > know if
> > > > > you have further
> > > > > > > > > > > > > > > > feedback.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Mickael
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar
> Morling
> > > > > > > > > > > > > > > > <gu...@googlemail.com.invalid>
> wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I'm +1 for adding a GET endpoint for
> > obtaining
> > > > > config
> > > > > > > > > > > > definitions. It
> > > > > > > > > > > > > > > > > always felt odd to me that one has to issue
> > a PUT
> > > > > for that
> > > > > > > > > > > > purpose.
> > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > > nothing else, it'd be better in terms of
> > > > > discoverability of the
> > > > > > > > > > > > KC
> > > > > > > > > > > > > > REST
> > > > > > > > > > > > > > > > API.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > One additional feature request I'd have is
> to
> > > > > expose the valid
> > > > > > > > > > > > enum
> > > > > > > > > > > > > > > > > constants for enum-typed options. That'll
> > help to
> > > > > display the
> > > > > > > > > > > > values
> > > > > > > > > > > > > > in a
> > > > > > > > > > > > > > > > > drop-down or via radio buttons in a UI,
> give
> > us
> > > > > tab completion in
> > > > > > > > > > > > > > kcctl,
> > > > > > > > > > > > > > > > > etc.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > --Gunnar
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb
> > Chris
> > > > > Egerton
> > > > > > > > > > > > > > > > > <ch...@confluent.io.invalid>:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Viktor,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It sounds like there are three major
> > points here
> > > > > in favor of a
> > > > > > > > > > > > new
> > > > > > > > > > > > > > GET
> > > > > > > > > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1. You cannot issue a blank ("dummy")
> > request
> > > > > for sink
> > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > because a
> > > > > > > > > > > > > > > > > > topic list/topic regex has to be supplied
> > > > > (otherwise the PUT
> > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > > returns a 500 response)
> > > > > > > > > > > > > > > > > > 2. A dummy request still triggers custom
> > > > > validations by the
> > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > which may be best to avoid if we know for
> > sure
> > > > > that the config
> > > > > > > > > > > > > > isn't
> > > > > > > > > > > > > > > > worth
> > > > > > > > > > > > > > > > > > validating yet
> > > > > > > > > > > > > > > > > > 3. It's more ergonomic and intuitive to
> be
> > able
> > > > > to issue a GET
> > > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > > without having to give a dummy connector
> > config
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > With regards to 1, this is actually a bug
> > in
> > > > > Connect (
> > > > > > > > > > > > > > > > > >
> > > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix
> > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > implemented and awaiting committer
> review (
> > > > > > > > > > > > > > > > > >
> https://github.com/apache/kafka/pull/11369).
> > I
> > > > > think it'd be
> > > > > > > > > > > > > > better to
> > > > > > > > > > > > > > > > > > focus on fixing this bug in general
> > instead of
> > > > > implementing a
> > > > > > > > > > > > new
> > > > > > > > > > > > > > REST
> > > > > > > > > > > > > > > > > > endpoint in order to allow people to work
> > around
> > > > > it.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > With regards to 2, this is technically
> > possible
> > > > > but I'm unsure
> > > > > > > > > > > > > > it'd be
> > > > > > > > > > > > > > > > too
> > > > > > > > > > > > > > > > > > common out in the wild given that most
> > > > > validations that could
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > expensive
> > > > > > > > > > > > > > > > > > would involve things like connecting to a
> > > > > database, checking
> > > > > > > > > > > > if a
> > > > > > > > > > > > > > cloud
> > > > > > > > > > > > > > > > > > storage bucket exists, etc., none of
> which
> > are
> > > > > possible without
> > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > configuration properties from the user
> (db
> > > > > hostname, bucket
> > > > > > > > > > > > name,
> > > > > > > > > > > > > > > > etc.).
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > With regards to 3, I do agree that it'd
> be
> > > > > easier for people
> > > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > > UIs
> > > > > > > > > > > > > > > > > > to have a GET API to work against. I'm
> > just not
> > > > > sure it's
> > > > > > > > > > > > worth the
> > > > > > > > > > > > > > > > > > additional implementation, testing, and
> > > > > maintenance burden. If
> > > > > > > > > > > > it
> > > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > possible to issue a PUT request without
> > > > > unexpected 500s for
> > > > > > > > > > > > invalid
> > > > > > > > > > > > > > > > > > configs, would that suffice? AFAICT it'd
> > > > > basically be as
> > > > > > > > > > > > simple as
> > > > > > > > > > > > > > > > issuing
> > > > > > > > > > > > > > > > > > a PUT request with a dummy body
> consisting
> > of
> > > > > nothing except
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > class (which at this point we might even
> > make
> > > > > unnecessary and
> > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > automatically replace with the connector
> > class
> > > > > from the URL)
> > > > > > > > > > > > and
> > > > > > > > > > > > > > then
> > > > > > > > > > > > > > > > > > filtering the response to just grab the
> > > > > "definition" field of
> > > > > > > > > > > > each
> > > > > > > > > > > > > > > > element
> > > > > > > > > > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor
> > > > > Somogyi-Vass <
> > > > > > > > > > > > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Folks,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I too think this would be a very useful
> > > > > feature. Some of our
> > > > > > > > > > > > > > > > management
> > > > > > > > > > > > > > > > > > > applications would provide a wizard for
> > > > > creating connectors.
> > > > > > > > > > > > In
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > scenario the user basically would fill
> > out a
> > > > > sample
> > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > generated
> > > > > > > > > > > > > > > > > > > by the UI which would send it back to
> > Connect
> > > > > for validation
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > eventually
> > > > > > > > > > > > > > > > > > > create a new connector. The first part
> > of this
> > > > > workflow can
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > > enhanced
> > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > we had an API that can return the
> > > > > configuration definition
> > > > > > > > > > > > of the
> > > > > > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > > > type of connector as the UI application
> > would
> > > > > be able to
> > > > > > > > > > > > > > generate a
> > > > > > > > > > > > > > > > > > sample
> > > > > > > > > > > > > > > > > > > for the user based on that (nicely
> drawn
> > > > > diagram:
> > > > > > > > > > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > > > > > > > > > The
> > > > > connector-plugins/{connectorType}/config/validate API
> > > > > > > > > > > > > > essentially
> > > > > > > > > > > > > > > > > > works
> > > > > > > > > > > > > > > > > > > and returns the data that we need,
> > however it
> > > > > is a HTTP PUT
> > > > > > > > > > > > API
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > > bit unintuitive for a fetch-like
> > functionality
> > > > > and also
> > > > > > > > > > > > > > functionally
> > > > > > > > > > > > > > > > > > > different as it validates the given
> > (dummy)
> > > > > request. In case
> > > > > > > > > > > > of
> > > > > > > > > > > > > > sink
> > > > > > > > > > > > > > > > > > > connectors one would need to also
> > provide a
> > > > > topic name.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > A suggestion for the KIP: I think it
> can
> > be
> > > > > useful to return
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > groups and the connector class' name
> > similarly
> > > > > to the
> > > > > > > > > > > > validate
> > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > case any frontend needs them (and also
> > the
> > > > > response would be
> > > > > > > > > > > > more
> > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > validate API but simpler).
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Viktor
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne
> > Dolan <
> > > > > > > > > > > > > > ryannedolan@gmail.com>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think it'd be worth adding a GET
> > version,
> > > > > fwiw. Could be
> > > > > > > > > > > > the
> > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > > handler
> > > > > > > > > > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael
> > Maison
> > > > > <
> > > > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > You're right, you can achieve the
> > same
> > > > > functionality
> > > > > > > > > > > > using
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > > > > > > > > > In my mind it was only for
> > validation once
> > > > > you have
> > > > > > > > > > > > build a
> > > > > > > > > > > > > > > > > > > > > configuration but when used with an
> > empty
> > > > > configuration,
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > > basically
> > > > > > > > > > > > > > > > > > > > > serves the same purpose as the
> > proposed
> > > > > new endpoint.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think it's a bit easier to use a
> > GET
> > > > > endpoint but I
> > > > > > > > > > > > don't
> > > > > > > > > > > > > > > > think it
> > > > > > > > > > > > > > > > > > > > > really warrants a different
> endpoint.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM
> Chris
> > > > > Egerton
> > > > > > > > > > > > > > > > > > > > > <ch...@confluent.io.invalid>
> wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I'm wondering about the use case
> > here.
> > > > > The motivation
> > > > > > > > > > > > > > section
> > > > > > > > > > > > > > > > > > states
> > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > "Connect does not provide a way
> to
> > see
> > > > > what
> > > > > > > > > > > > configurations
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > requires. Instead users have to
> go
> > look
> > > > > at the
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > documentation
> > > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > in the worst case, look directly
> > at the
> > > > > connector
> > > > > > > > > > > > source
> > > > > > > > > > > > > > > > code.",
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > with this KIP, "users will be
> able
> > to
> > > > > discover the
> > > > > > > > > > > > required
> > > > > > > > > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > > > > > > > > for connectors installed in a
> > Connect
> > > > > cluster" and
> > > > > > > > > > > > "tools
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > generate wizards for configuring
> > and
> > > > > starting
> > > > > > > > > > > > connectors".
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > > > > > > > > >
> > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > > > > > > > > > endpoint not address these
> points?
> > What
> > > > > will the
> > > > > > > > > > > > > > newly-proposed
> > > > > > > > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > > > > > > allow users to do that they will
> > not
> > > > > already be able
> > > > > > > > > > > > to do
> > > > > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM
> > Mickael
> > > > > Maison <
> > > > > > > > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I've created KIP-769 to expose
> > > > > connector
> > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > definitions
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Please take a look and let me
> > know if
> > > > > you have any
> > > > > > > > > > > > > > feedback.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > >
> >
> >
>

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Tom Bentley <tb...@redhat.com>]

Hi Mickael,

Thanks for the KIP. Overall I think this looks good.

I have a small concern about the proliferation of Versioned. Currently this
interface has a really weak contract. There's no requirement for versions
to be comparable, or documentation about when versions should be changed by
authors of plugins which extend this interface. For example, if I release
version 1.2.3 of my SMT should the version be 1.2.3 or something else? Is
compatibility implied for different versions of my jar file which return
the same result from version()? Because of this it's not really clear what
the consumer of the version string is supposed to do with it. So I think if
we're going to be using Versioned more extensively it would be good to make
the expectations clearer, for both authors and people wanting to reason
about versions.

Many thanks,

Tom

On Mon, 6 Dec 2021 at 09:56, Mickael Maison <mi...@gmail.com>
wrote:

> Hi Arjun,
>
> Thank you for taking a look at this KIP.
>
> 1.a Yes this is exactly what this KIP is proposing.
>
> 1.b I prefer returning all plugins together instead of having different
> paths.
>
> 2. While we seem to agree it would be nice to expose worker plugins, I
> don't think we've reached a consensus on the right way to do it. For
> that reason, I'm dropping this feature from this KIP. We can address
> this issue in another KIP. I have updated the KIP.
>
> Thanks,
> Mickael
>
> On Mon, Dec 6, 2021 at 10:15 AM Mickael Maison <mi...@gmail.com>
> wrote:
> >
> > Thanks Randall, updated.
> >
> > On Sun, Dec 5, 2021 at 6:05 AM Arjun Satish <ar...@gmail.com>
> wrote:
> > >
> > > hey folks,
> > >
> > > Thanks a lot for the KIP and the discussions. Here are a couple of
> thoughts
> > > (apologies if I missed some point in this thread).
> > >
> > > 1. it seems like we are changing the behavior of /connector-plugins by
> now
> > > returning the list of transforms, converters as well? This would break
> > > backward compatibility, and clients would need to track what version of
> > > Connect workers they are querying to keep their experience consistent.
> I
> > > wonder if we can keep the old behavior, with one of the following
> methods:
> > >
> > > 1a. We add an "expand" multi-query to the /connector-plugin endpoint
> like
> > > how it was done in KIP-465 [1]. This would preserve backward
> compatibility,
> > > as well as allow you to get consolidated output. Acceptable values for
> the
> > > new param could be: "all" (to show all plugins), or
> > > "connectors|transforms|predicates" that show a subset of the plugin
> types.
> > >
> > > 1b. Query params might get crummy to extend. Alternatively, we could
> have
> > > an additional fine-grained paths for each plugin:
> > >
> > > # to list all the source and sink plugins
> > > /connector-plugins/connectors
> > >
> > > # to list all the SMT plugins
> > > /connector-plugins/transformations
> > >
> > > # to list all the predicate plugins
> > > /connector-plugins/predicates
> > >
> > > # to list all the converter plugins
> > > /connector-plugins/converters
> > >
> > > # to list all the header converter plugins
> > > /connector-plugins/header-converters
> > >
> > > This allows us to cleanly add new types of plugins down the road, and
> > > naturally extend to displaying attributes of these objects. Configs,
> for
> > > example, can be accessed with
> > >
> > >
> http://localhost:8083/connector-plugins/transformations/Cast$Value/config
> > >
> > > A migration path (for both the suggestions above) would be to add a
> > > deprecation notice on the current behavior of /connector-plugins
> (where,
> > > after this PR is merged, both /connector-plugins and
> > > /connector-plugins/connectors return the same output). And starting
> from
> > > the next major version release, /connector-plugins redirects to
> > > /connector-plugins/connectors (making it super easy for clients to
> catch
> > > up).
> > >
> > > 2. Regarding /worker-plugins, I agree that it is useful to have the
> REST
> > > API inform the state of the REST extensions, and Config Providers.
> However,
> > > users of the Connect clusters will rarely need to know how a config
> > > provider or REST extension is configured to construct data pipelines.
> It's
> > > mostly an admin task, AFAICT. Also, we have also seen these problems in
> > > Connect clusters where one host is somewhat differently set up than
> other
> > > (contents of plugin path, classpath, local worker configuration files,
> jvm
> > > versions, GC settings) that surface as issues in Connector behavior.
> > > Overall, I'm wondering if we should just tackle this problem in a
> separate
> > > KIP, that evaluates all of these cases. All of this might nicely fall
> into
> > > under the /admin endpoint and allow us to expose detailed info on the
> state
> > > of the worker (maybe a /admin/worker/ that returns all of this info).
> > >
> > > Thanks,
> > >
> > > [1]
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-465%3A+Add+Consolidated+Connector+Endpoint+to+Connect+REST+API
> > >
> > >
> > > On Sat, Dec 4, 2021 at 10:29 AM Randall Hauch <rh...@gmail.com>
> wrote:
> > >
> > > > Thanks, Mickael. Can we call out these as literals? For example:
> > > >
> > > > > The response structure of the objects in the array remain
> unchanged, but
> > > > this proposal does add "converter", "header_converter" and
> "transformation"
> > > > to the allowed values for the "type" field, in addition to the
> existing
> > > > "sink" and "source" values.
> > > >
> > > > Best regards,
> > > >
> > > > Randall
> > > >
> > > > On Sat, Dec 4, 2021 at 4:58 AM Mickael Maison <
> mickael.maison@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hi Randall,
> > > > >
> > > > > Sure. The types are sink, source, converter, header_converter,
> > > > > transformation and predicate. I've updated the KIP.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Fri, Dec 3, 2021 at 10:14 PM Randall Hauch <rh...@gmail.com>
> wrote:
> > > > > >
> > > > > > One more thing, inline below:
> > > > > >
> > > > > > Randall
> > > > > >
> > > > > > On Fri, Dec 3, 2021 at 10:38 AM Mickael Maison <
> > > > mickael.maison@gmail.com> wrote:
> > > > > > >
> > > > > > > Thanks Randall for the feedback!
> > > > > > >
> > > > > > > 1. Done
> > > > > > >
> > > > > > > 2. Yes that makes sense. I've updated the KIP
> > > > > > >
> > > > > > > 3.a I've updated to explicitly mention the plugin can be
> provided
> > > > > > > using the fully qualified class or the alias.
> > > > > > > 3.b Only concrete classes will be returned. I made the mistake
> when I
> > > > > > > crafted the examples by hand. Fixed
> > > > > >
> > > > > > Can we also enumerate all of the values that can be included in
> the
> > > > > > "type" field in the response from the "GET
> > > > > > /connector-plugins?connectorsOnly=false"?
> > > > > >
> > > > > > >
> > > > > > > 4. If a plugin does not override config(), the response is an
> empty
> > > > > > > array. I've updated the KIP
> > > > > > >
> > > > > > > 5. I have updated the compatibility section with more details
> > > > > > >
> > > > > > > 6. Can you clarify what you mean? KIP-507 was for securing
> internal
> > > > > > > REST endpoints.
> > > > > > >
> > > > > > > 7. I don't think it is necessary. I think the use case Chris
> > > > mentioned
> > > > > > > could be useful for admins. I'm interested in exposing the
> > > > > > > connector-level plugins and their configs. I thought it was
> easy to
> > > > > > > add an API for worker plugins while working in this area but
> if there
> > > > > > > isn't a strong consensus, I'm happy to drop it.
> > > > > > >
> > > > > > > Mickael
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch <rhauch@gmail.com
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > Thanks for the KIP, Mickael. And thanks to everyone else for
> the
> > > > good
> > > > > > > > discussion so far.
> > > > > > > >
> > > > > > > > Overall this is pretty good, but I do have a few
> > > > questions/comments:
> > > > > > > > 1. Can we include the HTTP method/verb in each of the
> endpoints in
> > > > the
> > > > > > > > "Public Interfaces" section?
> > > > > > > > 2. The sample output for `GET
> > > > /connector-plugins?connectorsOnly=false`
> > > > > > > > includes a `version` field for the connector plugin, but not
> for
> > > > the
> > > > > > > > transform or converter plugins. Neither the `Transformation`
> or
> > > > > > > > `Converter` interfaces have a version nor implement the
> `Versioned`
> > > > > > > > interface, so should we add the `Versioned` interface to the
> > > > > > > > `Transformation` and `Converter` interfaces and add a default
> > > > method
> > > > > > > > that returns "unknown"? Yes, we could do that in a separate
> KIP,
> > > > but I
> > > > > > > > think it's appropriate to do it in this KIP for a few
> reasons.
> > > > First,
> > > > > > > > we're already modifying the `Converter` interface in this
> KIP and
> > > > it
> > > > > > > > would be quite straightforward to also extend the `Versioned`
> > > > > > > > interface and add a default method; then doing that for
> > > > > > > > `Transformation` would also be relatively straightforward.
> Second,
> > > > > > > > this KIP provides the justification for adding a version to
> the
> > > > > > > > `Transformation` and `Converter` interfaces.
> > > > > > > > 3. In the "Public Interfaces" section, there are two
> > > > inconsistencies
> > > > > > > > between the `GET /connector-plugins` and `GET
> > > > > > > > /connector-plugins/<plugin>/config` methods. Both will make
> it
> > > > > > > > difficult to use the second method, since the input to the
> second
> > > > > > > > method is not directly found in the response from the first
> method:
> > > > > > > >  a. The example for `GET
> /connector-plugins/Cast$Value/config` uses
> > > > > > > > short name, or what Connect often refers to as the alias,
> > > > "Cast$Value"
> > > > > > > > for the plugin, rather than the fully-qualified name
> > > > > > > > "org.apache.kafka.connect.transforms.Cast$Value". The KIP
> should
> > > > > > > > specify whether it will support both the shortened name and
> the
> > > > > > > > fully-qualified name, or whether only one of these will be
> > > > supported.
> > > > > > > > Not supporting the fully-qualified name will make it
> impossible to
> > > > use
> > > > > > > > the response from the first method as an input to the second.
> > > > > > > >  a. The example for the list method returns the
> `Transformation`
> > > > base
> > > > > > > > and abstract classes (e.g.,
> > > > > > > > "org.apache.kafka.connect.transforms.Cast") rather than the
> > > > concrete
> > > > > > > > classes (e.g.,
> "org.apache.kafka.connect.transforms.Cast$Value" and
> > > > > > > > "org.apache.kafka.connect.transforms.Cast$Key"), while the
> `GET
> > > > > > > > /connector-plugins/<plugin>/config` will require the concrete
> > > > class.
> > > > > > > > 4. The "Public Interfaces" section's description of the `GET
> > > > > > > > /connector-plugins/<plugin>/config` method should probably
> specify
> > > > the
> > > > > > > > expected behavior for `Converter` implementations that don't
> > > > override
> > > > > > > > the new `config()` method.
> > > > > > > > 5. The "Compatibility, Deprecation, and Migration Plan"
> section
> > > > should
> > > > > > > > describe whether older `Converter` implementations (compiled
> with
> > > > > > > > older versions of the Connect API without Converter extending
> > > > > > > > Configurable) will be able to be installed into newer Connect
> > > > > > > > runtimes, and what the behavior will be for the `GET
> > > > > > > > /connector-plugins/<plugin>/config` method. (Although my
> suggestion
> > > > > > > > for #5 will partially address the latter, we still need to
> talk
> > > > about
> > > > > > > > it in terms of compatibility.)
> > > > > > > > 6. Can we add a small section addressing the security
> implications
> > > > of
> > > > > > > > these new APIs? KIP-507 [1] applied to APIs that existed at
> the
> > > > time,
> > > > > > > > and this KIP should at least mention how it aligns with that
> > > > > > > > mechanism.
> > > > > > > > 7. Do we really think adding `GET /worker-plugins` is
> necessary?
> > > > While
> > > > > > > > I understand the reasoning, I'm less convinced that it's
> worth
> > > > adding
> > > > > > > > to the public API. First, an administrator still has to
> modify the
> > > > > > > > worker config for all workers to take advantage of that.
> Second,
> > > > > > > > adding this method increases the surface area and therefore
> risk of
> > > > > > > > additional attack vectors, despite not really being able to
> use the
> > > > > > > > resulting information in any other part of the API.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > > Randall
> > > > > > > >
> > > > > > > > [1]
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Dec 1, 2021 at 7:52 AM Mickael Maison <
> > > > mickael.maison@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Hi Chris,
> > > > > > > > >
> > > > > > > > > Thanks again for the feedback! I've updated the KIP based
> on our
> > > > last
> > > > > > > > > discussions. I've decided to include the new endpoint for
> worker
> > > > > > > > > plugins.
> > > > > > > > >
> > > > > > > > > 1. Yes I agree, it's best to gate the new behavior.
> > > > > > > > > 2. Yes, it was a remnant from the original proposal. I've
> now
> > > > removed
> > > > > > > > > the location field.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Mickael
> > > > > > > > >
> > > > > > > > > On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
> > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Mickael,
> > > > > > > > > >
> > > > > > > > > > I think that's a great idea! I especially like how we can
> > > > establish the
> > > > > > > > > > expectation that any plugin type that appears in the
> response
> > > > from the GET
> > > > > > > > > > /connector-plugins endpoint will have a corresponding GET
> > > > > > > > > > /connector-plugins/<type>/config endpoint, but (if we
> decide
> > > > to add them in
> > > > > > > > > > the future), worker plugins won't be expected to expose
> this
> > > > kind of
> > > > > > > > > > information and the different root path helps give a
> decent
> > > > hint about this.
> > > > > > > > > >
> > > > > > > > > > I also like the choice to return an empty ConfigDef from
> > > > Converter::config
> > > > > > > > > > instead of null.
> > > > > > > > > >
> > > > > > > > > > Two things come to mind:
> > > > > > > > > >
> > > > > > > > > > 1. We may want to gate this behind a URL query parameter
> > > > (maybe something
> > > > > > > > > > like "connectorsOnly") that defaults to the old behavior
> in
> > > > order to avoid
> > > > > > > > > > breaking existing tools such as programmatic UIs that
> use the
> > > > endpoint
> > > > > > > > > > today to discover the connectors that can be created by
> the
> > > > user. We can
> > > > > > > > > > even plan to change the default for that parameter to the
> > > > newly-proposed
> > > > > > > > > > behavior in the next major release, which should give
> people
> > > > enough time to
> > > > > > > > > > either adapt to the expanded response format or add the
> query
> > > > parameter to
> > > > > > > > > > their tooling.
> > > > > > > > > >
> > > > > > > > > > 2. The existing GET /connector-plugins endpoint doesn't
> > > > contain information
> > > > > > > > > > on the location of the plugin on the worker's file
> system. Do
> > > > you think we
> > > > > > > > > > should still include this info in the new response
> format?
> > > > Correct me if
> > > > > > > > > > I'm wrong but it seems it may have been proposed
> originally to
> > > > help prevent
> > > > > > > > > > already-addressed bugs in Connect classloading from
> striking;
> > > > all else
> > > > > > > > > > equal, I'd personally err on the side of leaving this
> info out
> > > > or at least
> > > > > > > > > > reducing permitted values for it to just "classpath" or
> > > > "plugin path" in
> > > > > > > > > > order to avoid leaking worker file paths into the REST
> API,
> > > > which might
> > > > > > > > > > bother super security-conscious users.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > >
> > > > > > > > > > Chris
> > > > > > > > > >
> > > > > > > > > > On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <
> > > > mickael.maison@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Chris,
> > > > > > > > > > >
> > > > > > > > > > > Yes to keep compatibility we want a default
> implementation
> > > > for
> > > > > > > > > > > Converter.configs(), I've updated the KIP.
> > > > > > > > > > >
> > > > > > > > > > > Regarding worker plugins, the use case you described
> seems
> > > > valuable.
> > > > > > > > > > > I'd prefer not mixing worker and connector plugins on
> the
> > > > same
> > > > > > > > > > > endpoint but I agree using /plugins and /worker-plugins
> > > > could be
> > > > > > > > > > > confusing.
> > > > > > > > > > >
> > > > > > > > > > > One alternative is to expose all connector-level
> plugins via
> > > > the
> > > > > > > > > > > existing /connector-plugins endpoint. In that case,
> we'd
> > > > need to keep
> > > > > > > > > > > the current JSON schema and not group plugins by type.
> As
> > > > the current
> > > > > > > > > > > schema already has a type field for each entry, we'll
> still
> > > > be able to
> > > > > > > > > > > tell them apart. Then we can have /worker-plugins and a
> > > > relatively
> > > > > > > > > > > clean API. What do you think?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Mickael
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > >
> > > > > > > > > > > > I think one potential use case for exposing
> worker-level
> > > > plugins is that
> > > > > > > > > > > it
> > > > > > > > > > > > may make it easier to determine whether a worker is
> set up
> > > > correctly (the
> > > > > > > > > > > > current alternative requires looking through log
> files and
> > > > can be a
> > > > > > > > > > > little
> > > > > > > > > > > > tedious), and might even make it possible to
> automatically
> > > > identify
> > > > > > > > > > > > discrepancies within a cluster by diffing the
> contents of
> > > > that endpoint
> > > > > > > > > > > > across each worker. But I don't think this has to be
> > > > addressed by the
> > > > > > > > > > > > current KIP; the only thing that bothers me a little
> is
> > > > that "plugins" is
> > > > > > > > > > > > generic and it may confuse people down the road if
> we add
> > > > an endpoint for
> > > > > > > > > > > > worker-level plugins ("why is one just called
> 'plugins'
> > > > and the other one
> > > > > > > > > > > > is 'worker-plugins'?"). Probably not worth blocking
> on,
> > > > though.
> > > > > > > > > > > >
> > > > > > > > > > > > Agreed that the suggestion for improved validation
> should
> > > > be made on the
> > > > > > > > > > > > KIP-802 thread.
> > > > > > > > > > > >
> > > > > > > > > > > > I also noticed that the newly-proposed config method
> for
> > > > the Converter
> > > > > > > > > > > > interface doesn't have a default implementation,
> making it
> > > > > > > > > > > > backwards-incompatible. Should we add a default
> > > > implementation that
> > > > > > > > > > > returns
> > > > > > > > > > > > either null or an empty ConfigDef?
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > >
> > > > > > > > > > > > Chris
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <
> > > > mickael.maison@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. If we want to expose worker plugins, I think we
> > > > should do it via a
> > > > > > > > > > > > > separate endpoint. But to be honest, I'm not even
> sure I
> > > > see strong
> > > > > > > > > > > > > use cases for exposing them as they are either
> enabled
> > > > or not and
> > > > > > > > > > > > > can't be changed at runtime. So I'd prefer to
> stick to
> > > > "connector
> > > > > > > > > > > > > level" plugins in this KIP. Let me now if you have
> use
> > > > cases, I'm open
> > > > > > > > > > > > > to reconsider this choice.
> > > > > > > > > > > > > I'll add that in the rejected alternatives section
> for
> > > > now
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. I remembered seeing issues in the past with
> multiple
> > > > plugin.path
> > > > > > > > > > > > > entries but I tried today and I was able to mix and
> > > > match plugins from
> > > > > > > > > > > > > different paths. So my bad for getting confused.
> > > > > > > > > > > > > Then I agree, it makes more sense to group them by
> > > > plugin type.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3. Yes this should be covered in KIP-802:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > > > > > > > > > > >
> > > > > > > > > > > > > 4. No particular reason. We can support both
> formats
> > > > like today. I've
> > > > > > > > > > > > > updated the KIP
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Mickael
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think the increase in scope here is great and
> the
> > > > added value
> > > > > > > > > > > certainly
> > > > > > > > > > > > > > justifies the proposed changes. I have some
> thoughts
> > > > but overall I
> > > > > > > > > > > like
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > direction this is going in now.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. The new /plugins endpoint is described as
> > > > containing "all plugins
> > > > > > > > > > > that
> > > > > > > > > > > > > > are Connectors, Transformations, Converters,
> > > > HeaderConverters and
> > > > > > > > > > > > > > Predicates". So essentially, it looks like we
> want to
> > > > expose all
> > > > > > > > > > > plugins
> > > > > > > > > > > > > > that are configured on a per-connector basis, but
> > > > exclude plugins
> > > > > > > > > > > that
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > configured on a per-worker basis (such as config
> > > > providers and REST
> > > > > > > > > > > > > > extensions). Do you think it may be valuable to
> expose
> > > > information on
> > > > > > > > > > > > > > worker-level plugins as well?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2. The description for the new /plugins endpoint
> also
> > > > states that
> > > > > > > > > > > > > "Plugins
> > > > > > > > > > > > > > will be grouped by plugin.path. This will make it
> > > > clear to users
> > > > > > > > > > > what's
> > > > > > > > > > > > > > available to use as it's not possible to use a
> > > > Connector from one
> > > > > > > > > > > path
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > Transformations from another.". Is this true? I
> > > > thought that
> > > > > > > > > > > Connect's
> > > > > > > > > > > > > > classloading made it possible to package
> > > > > > > > > > > > > > converters/transformations/predicates completely
> > > > independently from
> > > > > > > > > > > each
> > > > > > > > > > > > > > other, and to reference them from
> > > > also-independently-packaged
> > > > > > > > > > > connectors.
> > > > > > > > > > > > > > If it turns out that this is the case, could we
> > > > consider
> > > > > > > > > > > restructuring
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > response to be grouped by plugin type instead of
> by
> > > > classloader?
> > > > > > > > > > > There's
> > > > > > > > > > > > > > also the ungrouped format proposed in KIP-494 (
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > > > > > > > > > > )
> > > > > > > > > > > > > > which we might consider as well.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3. I think this can be left for a follow-up KIP
> if
> > > > necessary, but I'm
> > > > > > > > > > > > > > curious about your thoughts on adding new
> validate
> > > > methods to all
> > > > > > > > > > > > > > connector-level plugins that can be used
> similarly to
> > > > how the
> > > > > > > > > > > existing
> > > > > > > > > > > > > > Connector::validate method (
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > >
> https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > > > > > > > > > > )
> > > > > > > > > > > > > > is used. This would allow for plugins to perform
> > > > validation that's
> > > > > > > > > > > more
> > > > > > > > > > > > > > sophisticated than what the ConfigDef is capable
> of,
> > > > such as
> > > > > > > > > > > validating
> > > > > > > > > > > > > > combinations of properties like a hostname and
> > > > credentials for
> > > > > > > > > > > reaching
> > > > > > > > > > > > > it.
> > > > > > > > > > > > > > I know that at least Confluent's Avro, protobuf,
> and
> > > > JSON schema
> > > > > > > > > > > > > converters
> > > > > > > > > > > > > > would benefit from this kind of feature. It's a
> little
> > > > tangential to
> > > > > > > > > > > this
> > > > > > > > > > > > > > KIP (which at the moment is about discovering
> plugins
> > > > and their
> > > > > > > > > > > > > > configuration surfaces, as opposed to validating
> > > > them), but I
> > > > > > > > > > > figured I'd
> > > > > > > > > > > > > > ask since we're going to be expanding the
> Converter
> > > > interface and it
> > > > > > > > > > > may
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > useful to tackle this while we're in the
> neighborhood.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 4. The description for the new
> > > > /plugins/<type>/<name>/configdef
> > > > > > > > > > > endpoint
> > > > > > > > > > > > > > states that "Name must be the fully qualified
> class
> > > > name of the
> > > > > > > > > > > plugin".
> > > > > > > > > > > > > > Any reason not to also support aliases (e.g.,
> > > > > > > > > > > "FileStreamSinkConnector"
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > "FileStreamSink" instead of
> > > > > > > > > > > > > >
> > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <
> > > > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks all for the feedback!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Chris,
> > > > > > > > > > > > > > > I agree that fixing the current endpoint helps
> a
> > > > lot. Thanks for
> > > > > > > > > > > > > > > raising these JIRAs and submitting a PR!
> > > > > > > > > > > > > > > However thinking about the issue further, I
> decided
> > > > to expand the
> > > > > > > > > > > > > > > scope of the KIP to cover all user-visible
> plugins.
> > > > > > > > > > > > > > > In practice, users want to know about all
> available
> > > > plugins not
> > > > > > > > > > > only
> > > > > > > > > > > > > > > connectors. This includes transformations,
> > > > converters,
> > > > > > > > > > > > > > > header_converters and predicates. As we also
> want to
> > > > retrieve
> > > > > > > > > > > > > > > configdef for these too, I think it makes
> sense to
> > > > introduce a new
> > > > > > > > > > > > > > > endpoint to do so. Alongside we obviously need
> a new
> > > > endpoint for
> > > > > > > > > > > > > > > listing all plugins.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Gunnar,
> > > > > > > > > > > > > > > I took a look at exposing valid values via the
> API.
> > > > I think the
> > > > > > > > > > > issue
> > > > > > > > > > > > > > > is that Validators don't expose a way to
> retrieve
> > > > valid values.
> > > > > > > > > > > > > > > Changing validators will have an impact on all
> > > > components so I'd
> > > > > > > > > > > > > > > prefer to address this requirement in a
> separate
> > > > KIP. I agree this
> > > > > > > > > > > > > > > would be an interesting improvement and I'd
> happy to
> > > > write a KIP
> > > > > > > > > > > for
> > > > > > > > > > > > > > > it too.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I have updated the KIP accordingly. Let me
> know if
> > > > you have further
> > > > > > > > > > > > > > > feedback.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Mickael
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > > > > > > > > > > > > > > <gu...@googlemail.com.invalid> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'm +1 for adding a GET endpoint for
> obtaining
> > > > config
> > > > > > > > > > > definitions. It
> > > > > > > > > > > > > > > > always felt odd to me that one has to issue
> a PUT
> > > > for that
> > > > > > > > > > > purpose.
> > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > nothing else, it'd be better in terms of
> > > > discoverability of the
> > > > > > > > > > > KC
> > > > > > > > > > > > > REST
> > > > > > > > > > > > > > > API.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > One additional feature request I'd have is to
> > > > expose the valid
> > > > > > > > > > > enum
> > > > > > > > > > > > > > > > constants for enum-typed options. That'll
> help to
> > > > display the
> > > > > > > > > > > values
> > > > > > > > > > > > > in a
> > > > > > > > > > > > > > > > drop-down or via radio buttons in a UI, give
> us
> > > > tab completion in
> > > > > > > > > > > > > kcctl,
> > > > > > > > > > > > > > > > etc.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --Gunnar
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb
> Chris
> > > > Egerton
> > > > > > > > > > > > > > > > <ch...@confluent.io.invalid>:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Viktor,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It sounds like there are three major
> points here
> > > > in favor of a
> > > > > > > > > > > new
> > > > > > > > > > > > > GET
> > > > > > > > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1. You cannot issue a blank ("dummy")
> request
> > > > for sink
> > > > > > > > > > > connectors
> > > > > > > > > > > > > > > because a
> > > > > > > > > > > > > > > > > topic list/topic regex has to be supplied
> > > > (otherwise the PUT
> > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > returns a 500 response)
> > > > > > > > > > > > > > > > > 2. A dummy request still triggers custom
> > > > validations by the
> > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > which may be best to avoid if we know for
> sure
> > > > that the config
> > > > > > > > > > > > > isn't
> > > > > > > > > > > > > > > worth
> > > > > > > > > > > > > > > > > validating yet
> > > > > > > > > > > > > > > > > 3. It's more ergonomic and intuitive to be
> able
> > > > to issue a GET
> > > > > > > > > > > > > request
> > > > > > > > > > > > > > > > > without having to give a dummy connector
> config
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > With regards to 1, this is actually a bug
> in
> > > > Connect (
> > > > > > > > > > > > > > > > >
> > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix
> > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > > > > > > > > > https://github.com/apache/kafka/pull/11369).
> I
> > > > think it'd be
> > > > > > > > > > > > > better to
> > > > > > > > > > > > > > > > > focus on fixing this bug in general
> instead of
> > > > implementing a
> > > > > > > > > > > new
> > > > > > > > > > > > > REST
> > > > > > > > > > > > > > > > > endpoint in order to allow people to work
> around
> > > > it.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > With regards to 2, this is technically
> possible
> > > > but I'm unsure
> > > > > > > > > > > > > it'd be
> > > > > > > > > > > > > > > too
> > > > > > > > > > > > > > > > > common out in the wild given that most
> > > > validations that could
> > > > > > > > > > > be
> > > > > > > > > > > > > > > expensive
> > > > > > > > > > > > > > > > > would involve things like connecting to a
> > > > database, checking
> > > > > > > > > > > if a
> > > > > > > > > > > > > cloud
> > > > > > > > > > > > > > > > > storage bucket exists, etc., none of which
> are
> > > > possible without
> > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > configuration properties from the user (db
> > > > hostname, bucket
> > > > > > > > > > > name,
> > > > > > > > > > > > > > > etc.).
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > With regards to 3, I do agree that it'd be
> > > > easier for people
> > > > > > > > > > > > > designing
> > > > > > > > > > > > > > > UIs
> > > > > > > > > > > > > > > > > to have a GET API to work against. I'm
> just not
> > > > sure it's
> > > > > > > > > > > worth the
> > > > > > > > > > > > > > > > > additional implementation, testing, and
> > > > maintenance burden. If
> > > > > > > > > > > it
> > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > possible to issue a PUT request without
> > > > unexpected 500s for
> > > > > > > > > > > invalid
> > > > > > > > > > > > > > > > > configs, would that suffice? AFAICT it'd
> > > > basically be as
> > > > > > > > > > > simple as
> > > > > > > > > > > > > > > issuing
> > > > > > > > > > > > > > > > > a PUT request with a dummy body consisting
> of
> > > > nothing except
> > > > > > > > > > > the
> > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > class (which at this point we might even
> make
> > > > unnecessary and
> > > > > > > > > > > just
> > > > > > > > > > > > > > > > > automatically replace with the connector
> class
> > > > from the URL)
> > > > > > > > > > > and
> > > > > > > > > > > > > then
> > > > > > > > > > > > > > > > > filtering the response to just grab the
> > > > "definition" field of
> > > > > > > > > > > each
> > > > > > > > > > > > > > > element
> > > > > > > > > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor
> > > > Somogyi-Vass <
> > > > > > > > > > > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Folks,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I too think this would be a very useful
> > > > feature. Some of our
> > > > > > > > > > > > > > > management
> > > > > > > > > > > > > > > > > > applications would provide a wizard for
> > > > creating connectors.
> > > > > > > > > > > In
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > scenario the user basically would fill
> out a
> > > > sample
> > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > generated
> > > > > > > > > > > > > > > > > > by the UI which would send it back to
> Connect
> > > > for validation
> > > > > > > > > > > and
> > > > > > > > > > > > > > > > > eventually
> > > > > > > > > > > > > > > > > > create a new connector. The first part
> of this
> > > > workflow can
> > > > > > > > > > > be
> > > > > > > > > > > > > > > enhanced
> > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > we had an API that can return the
> > > > configuration definition
> > > > > > > > > > > of the
> > > > > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > > type of connector as the UI application
> would
> > > > be able to
> > > > > > > > > > > > > generate a
> > > > > > > > > > > > > > > > > sample
> > > > > > > > > > > > > > > > > > for the user based on that (nicely drawn
> > > > diagram:
> > > > > > > > > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > > > > > > > > The
> > > > connector-plugins/{connectorType}/config/validate API
> > > > > > > > > > > > > essentially
> > > > > > > > > > > > > > > > > works
> > > > > > > > > > > > > > > > > > and returns the data that we need,
> however it
> > > > is a HTTP PUT
> > > > > > > > > > > API
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > bit unintuitive for a fetch-like
> functionality
> > > > and also
> > > > > > > > > > > > > functionally
> > > > > > > > > > > > > > > > > > different as it validates the given
> (dummy)
> > > > request. In case
> > > > > > > > > > > of
> > > > > > > > > > > > > sink
> > > > > > > > > > > > > > > > > > connectors one would need to also
> provide a
> > > > topic name.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > A suggestion for the KIP: I think it can
> be
> > > > useful to return
> > > > > > > > > > > the
> > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > groups and the connector class' name
> similarly
> > > > to the
> > > > > > > > > > > validate
> > > > > > > > > > > > > API
> > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > case any frontend needs them (and also
> the
> > > > response would be
> > > > > > > > > > > more
> > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > validate API but simpler).
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Viktor
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne
> Dolan <
> > > > > > > > > > > > > ryannedolan@gmail.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think it'd be worth adding a GET
> version,
> > > > fwiw. Could be
> > > > > > > > > > > the
> > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > handler
> > > > > > > > > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael
> Maison
> > > > <
> > > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > You're right, you can achieve the
> same
> > > > functionality
> > > > > > > > > > > using
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > > > > > > > > In my mind it was only for
> validation once
> > > > you have
> > > > > > > > > > > build a
> > > > > > > > > > > > > > > > > > > > configuration but when used with an
> empty
> > > > configuration,
> > > > > > > > > > > it
> > > > > > > > > > > > > > > basically
> > > > > > > > > > > > > > > > > > > > serves the same purpose as the
> proposed
> > > > new endpoint.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think it's a bit easier to use a
> GET
> > > > endpoint but I
> > > > > > > > > > > don't
> > > > > > > > > > > > > > > think it
> > > > > > > > > > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris
> > > > Egerton
> > > > > > > > > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I'm wondering about the use case
> here.
> > > > The motivation
> > > > > > > > > > > > > section
> > > > > > > > > > > > > > > > > states
> > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > "Connect does not provide a way to
> see
> > > > what
> > > > > > > > > > > configurations
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > requires. Instead users have to go
> look
> > > > at the
> > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > documentation
> > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > in the worst case, look directly
> at the
> > > > connector
> > > > > > > > > > > source
> > > > > > > > > > > > > > > code.",
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > with this KIP, "users will be able
> to
> > > > discover the
> > > > > > > > > > > required
> > > > > > > > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > > > > > > > for connectors installed in a
> Connect
> > > > cluster" and
> > > > > > > > > > > "tools
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > generate wizards for configuring
> and
> > > > starting
> > > > > > > > > > > connectors".
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > > > > > > > >
> > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > > > > > > > > endpoint not address these points?
> What
> > > > will the
> > > > > > > > > > > > > newly-proposed
> > > > > > > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > > > > > allow users to do that they will
> not
> > > > already be able
> > > > > > > > > > > to do
> > > > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM
> Mickael
> > > > Maison <
> > > > > > > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I've created KIP-769 to expose
> > > > connector
> > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > definitions
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Please take a look and let me
> know if
> > > > you have any
> > > > > > > > > > > > > feedback.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > >
>
>

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Mickael Maison <mi...@gmail.com>]

Hi Arjun,

Thank you for taking a look at this KIP.

1.a Yes this is exactly what this KIP is proposing.

1.b I prefer returning all plugins together instead of having different paths.

2. While we seem to agree it would be nice to expose worker plugins, I
don't think we've reached a consensus on the right way to do it. For
that reason, I'm dropping this feature from this KIP. We can address
this issue in another KIP. I have updated the KIP.

Thanks,
Mickael

On Mon, Dec 6, 2021 at 10:15 AM Mickael Maison <mi...@gmail.com> wrote:
>
> Thanks Randall, updated.
>
> On Sun, Dec 5, 2021 at 6:05 AM Arjun Satish <ar...@gmail.com> wrote:
> >
> > hey folks,
> >
> > Thanks a lot for the KIP and the discussions. Here are a couple of thoughts
> > (apologies if I missed some point in this thread).
> >
> > 1. it seems like we are changing the behavior of /connector-plugins by now
> > returning the list of transforms, converters as well? This would break
> > backward compatibility, and clients would need to track what version of
> > Connect workers they are querying to keep their experience consistent. I
> > wonder if we can keep the old behavior, with one of the following methods:
> >
> > 1a. We add an "expand" multi-query to the /connector-plugin endpoint like
> > how it was done in KIP-465 [1]. This would preserve backward compatibility,
> > as well as allow you to get consolidated output. Acceptable values for the
> > new param could be: "all" (to show all plugins), or
> > "connectors|transforms|predicates" that show a subset of the plugin types.
> >
> > 1b. Query params might get crummy to extend. Alternatively, we could have
> > an additional fine-grained paths for each plugin:
> >
> > # to list all the source and sink plugins
> > /connector-plugins/connectors
> >
> > # to list all the SMT plugins
> > /connector-plugins/transformations
> >
> > # to list all the predicate plugins
> > /connector-plugins/predicates
> >
> > # to list all the converter plugins
> > /connector-plugins/converters
> >
> > # to list all the header converter plugins
> > /connector-plugins/header-converters
> >
> > This allows us to cleanly add new types of plugins down the road, and
> > naturally extend to displaying attributes of these objects. Configs, for
> > example, can be accessed with
> >
> > http://localhost:8083/connector-plugins/transformations/Cast$Value/config
> >
> > A migration path (for both the suggestions above) would be to add a
> > deprecation notice on the current behavior of /connector-plugins (where,
> > after this PR is merged, both /connector-plugins and
> > /connector-plugins/connectors return the same output). And starting from
> > the next major version release, /connector-plugins redirects to
> > /connector-plugins/connectors (making it super easy for clients to catch
> > up).
> >
> > 2. Regarding /worker-plugins, I agree that it is useful to have the REST
> > API inform the state of the REST extensions, and Config Providers. However,
> > users of the Connect clusters will rarely need to know how a config
> > provider or REST extension is configured to construct data pipelines. It's
> > mostly an admin task, AFAICT. Also, we have also seen these problems in
> > Connect clusters where one host is somewhat differently set up than other
> > (contents of plugin path, classpath, local worker configuration files, jvm
> > versions, GC settings) that surface as issues in Connector behavior.
> > Overall, I'm wondering if we should just tackle this problem in a separate
> > KIP, that evaluates all of these cases. All of this might nicely fall into
> > under the /admin endpoint and allow us to expose detailed info on the state
> > of the worker (maybe a /admin/worker/ that returns all of this info).
> >
> > Thanks,
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-465%3A+Add+Consolidated+Connector+Endpoint+to+Connect+REST+API
> >
> >
> > On Sat, Dec 4, 2021 at 10:29 AM Randall Hauch <rh...@gmail.com> wrote:
> >
> > > Thanks, Mickael. Can we call out these as literals? For example:
> > >
> > > > The response structure of the objects in the array remain unchanged, but
> > > this proposal does add "converter", "header_converter" and "transformation"
> > > to the allowed values for the "type" field, in addition to the existing
> > > "sink" and "source" values.
> > >
> > > Best regards,
> > >
> > > Randall
> > >
> > > On Sat, Dec 4, 2021 at 4:58 AM Mickael Maison <mi...@gmail.com>
> > > wrote:
> > > >
> > > > Hi Randall,
> > > >
> > > > Sure. The types are sink, source, converter, header_converter,
> > > > transformation and predicate. I've updated the KIP.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Fri, Dec 3, 2021 at 10:14 PM Randall Hauch <rh...@gmail.com> wrote:
> > > > >
> > > > > One more thing, inline below:
> > > > >
> > > > > Randall
> > > > >
> > > > > On Fri, Dec 3, 2021 at 10:38 AM Mickael Maison <
> > > mickael.maison@gmail.com> wrote:
> > > > > >
> > > > > > Thanks Randall for the feedback!
> > > > > >
> > > > > > 1. Done
> > > > > >
> > > > > > 2. Yes that makes sense. I've updated the KIP
> > > > > >
> > > > > > 3.a I've updated to explicitly mention the plugin can be provided
> > > > > > using the fully qualified class or the alias.
> > > > > > 3.b Only concrete classes will be returned. I made the mistake when I
> > > > > > crafted the examples by hand. Fixed
> > > > >
> > > > > Can we also enumerate all of the values that can be included in the
> > > > > "type" field in the response from the "GET
> > > > > /connector-plugins?connectorsOnly=false"?
> > > > >
> > > > > >
> > > > > > 4. If a plugin does not override config(), the response is an empty
> > > > > > array. I've updated the KIP
> > > > > >
> > > > > > 5. I have updated the compatibility section with more details
> > > > > >
> > > > > > 6. Can you clarify what you mean? KIP-507 was for securing internal
> > > > > > REST endpoints.
> > > > > >
> > > > > > 7. I don't think it is necessary. I think the use case Chris
> > > mentioned
> > > > > > could be useful for admins. I'm interested in exposing the
> > > > > > connector-level plugins and their configs. I thought it was easy to
> > > > > > add an API for worker plugins while working in this area but if there
> > > > > > isn't a strong consensus, I'm happy to drop it.
> > > > > >
> > > > > > Mickael
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch <rh...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > Thanks for the KIP, Mickael. And thanks to everyone else for the
> > > good
> > > > > > > discussion so far.
> > > > > > >
> > > > > > > Overall this is pretty good, but I do have a few
> > > questions/comments:
> > > > > > > 1. Can we include the HTTP method/verb in each of the endpoints in
> > > the
> > > > > > > "Public Interfaces" section?
> > > > > > > 2. The sample output for `GET
> > > /connector-plugins?connectorsOnly=false`
> > > > > > > includes a `version` field for the connector plugin, but not for
> > > the
> > > > > > > transform or converter plugins. Neither the `Transformation` or
> > > > > > > `Converter` interfaces have a version nor implement the `Versioned`
> > > > > > > interface, so should we add the `Versioned` interface to the
> > > > > > > `Transformation` and `Converter` interfaces and add a default
> > > method
> > > > > > > that returns "unknown"? Yes, we could do that in a separate KIP,
> > > but I
> > > > > > > think it's appropriate to do it in this KIP for a few reasons.
> > > First,
> > > > > > > we're already modifying the `Converter` interface in this KIP and
> > > it
> > > > > > > would be quite straightforward to also extend the `Versioned`
> > > > > > > interface and add a default method; then doing that for
> > > > > > > `Transformation` would also be relatively straightforward. Second,
> > > > > > > this KIP provides the justification for adding a version to the
> > > > > > > `Transformation` and `Converter` interfaces.
> > > > > > > 3. In the "Public Interfaces" section, there are two
> > > inconsistencies
> > > > > > > between the `GET /connector-plugins` and `GET
> > > > > > > /connector-plugins/<plugin>/config` methods. Both will make it
> > > > > > > difficult to use the second method, since the input to the second
> > > > > > > method is not directly found in the response from the first method:
> > > > > > >  a. The example for `GET /connector-plugins/Cast$Value/config` uses
> > > > > > > short name, or what Connect often refers to as the alias,
> > > "Cast$Value"
> > > > > > > for the plugin, rather than the fully-qualified name
> > > > > > > "org.apache.kafka.connect.transforms.Cast$Value". The KIP should
> > > > > > > specify whether it will support both the shortened name and the
> > > > > > > fully-qualified name, or whether only one of these will be
> > > supported.
> > > > > > > Not supporting the fully-qualified name will make it impossible to
> > > use
> > > > > > > the response from the first method as an input to the second.
> > > > > > >  a. The example for the list method returns the `Transformation`
> > > base
> > > > > > > and abstract classes (e.g.,
> > > > > > > "org.apache.kafka.connect.transforms.Cast") rather than the
> > > concrete
> > > > > > > classes (e.g., "org.apache.kafka.connect.transforms.Cast$Value" and
> > > > > > > "org.apache.kafka.connect.transforms.Cast$Key"), while the `GET
> > > > > > > /connector-plugins/<plugin>/config` will require the concrete
> > > class.
> > > > > > > 4. The "Public Interfaces" section's description of the `GET
> > > > > > > /connector-plugins/<plugin>/config` method should probably specify
> > > the
> > > > > > > expected behavior for `Converter` implementations that don't
> > > override
> > > > > > > the new `config()` method.
> > > > > > > 5. The "Compatibility, Deprecation, and Migration Plan" section
> > > should
> > > > > > > describe whether older `Converter` implementations (compiled with
> > > > > > > older versions of the Connect API without Converter extending
> > > > > > > Configurable) will be able to be installed into newer Connect
> > > > > > > runtimes, and what the behavior will be for the `GET
> > > > > > > /connector-plugins/<plugin>/config` method. (Although my suggestion
> > > > > > > for #5 will partially address the latter, we still need to talk
> > > about
> > > > > > > it in terms of compatibility.)
> > > > > > > 6. Can we add a small section addressing the security implications
> > > of
> > > > > > > these new APIs? KIP-507 [1] applied to APIs that existed at the
> > > time,
> > > > > > > and this KIP should at least mention how it aligns with that
> > > > > > > mechanism.
> > > > > > > 7. Do we really think adding `GET /worker-plugins` is necessary?
> > > While
> > > > > > > I understand the reasoning, I'm less convinced that it's worth
> > > adding
> > > > > > > to the public API. First, an administrator still has to modify the
> > > > > > > worker config for all workers to take advantage of that. Second,
> > > > > > > adding this method increases the surface area and therefore risk of
> > > > > > > additional attack vectors, despite not really being able to use the
> > > > > > > resulting information in any other part of the API.
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > > Randall
> > > > > > >
> > > > > > > [1]
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Dec 1, 2021 at 7:52 AM Mickael Maison <
> > > mickael.maison@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Hi Chris,
> > > > > > > >
> > > > > > > > Thanks again for the feedback! I've updated the KIP based on our
> > > last
> > > > > > > > discussions. I've decided to include the new endpoint for worker
> > > > > > > > plugins.
> > > > > > > >
> > > > > > > > 1. Yes I agree, it's best to gate the new behavior.
> > > > > > > > 2. Yes, it was a remnant from the original proposal. I've now
> > > removed
> > > > > > > > the location field.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mickael
> > > > > > > >
> > > > > > > > On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
> > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > >
> > > > > > > > > Hi Mickael,
> > > > > > > > >
> > > > > > > > > I think that's a great idea! I especially like how we can
> > > establish the
> > > > > > > > > expectation that any plugin type that appears in the response
> > > from the GET
> > > > > > > > > /connector-plugins endpoint will have a corresponding GET
> > > > > > > > > /connector-plugins/<type>/config endpoint, but (if we decide
> > > to add them in
> > > > > > > > > the future), worker plugins won't be expected to expose this
> > > kind of
> > > > > > > > > information and the different root path helps give a decent
> > > hint about this.
> > > > > > > > >
> > > > > > > > > I also like the choice to return an empty ConfigDef from
> > > Converter::config
> > > > > > > > > instead of null.
> > > > > > > > >
> > > > > > > > > Two things come to mind:
> > > > > > > > >
> > > > > > > > > 1. We may want to gate this behind a URL query parameter
> > > (maybe something
> > > > > > > > > like "connectorsOnly") that defaults to the old behavior in
> > > order to avoid
> > > > > > > > > breaking existing tools such as programmatic UIs that use the
> > > endpoint
> > > > > > > > > today to discover the connectors that can be created by the
> > > user. We can
> > > > > > > > > even plan to change the default for that parameter to the
> > > newly-proposed
> > > > > > > > > behavior in the next major release, which should give people
> > > enough time to
> > > > > > > > > either adapt to the expanded response format or add the query
> > > parameter to
> > > > > > > > > their tooling.
> > > > > > > > >
> > > > > > > > > 2. The existing GET /connector-plugins endpoint doesn't
> > > contain information
> > > > > > > > > on the location of the plugin on the worker's file system. Do
> > > you think we
> > > > > > > > > should still include this info in the new response format?
> > > Correct me if
> > > > > > > > > I'm wrong but it seems it may have been proposed originally to
> > > help prevent
> > > > > > > > > already-addressed bugs in Connect classloading from striking;
> > > all else
> > > > > > > > > equal, I'd personally err on the side of leaving this info out
> > > or at least
> > > > > > > > > reducing permitted values for it to just "classpath" or
> > > "plugin path" in
> > > > > > > > > order to avoid leaking worker file paths into the REST API,
> > > which might
> > > > > > > > > bother super security-conscious users.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > > On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <
> > > mickael.maison@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Chris,
> > > > > > > > > >
> > > > > > > > > > Yes to keep compatibility we want a default implementation
> > > for
> > > > > > > > > > Converter.configs(), I've updated the KIP.
> > > > > > > > > >
> > > > > > > > > > Regarding worker plugins, the use case you described seems
> > > valuable.
> > > > > > > > > > I'd prefer not mixing worker and connector plugins on the
> > > same
> > > > > > > > > > endpoint but I agree using /plugins and /worker-plugins
> > > could be
> > > > > > > > > > confusing.
> > > > > > > > > >
> > > > > > > > > > One alternative is to expose all connector-level plugins via
> > > the
> > > > > > > > > > existing /connector-plugins endpoint. In that case, we'd
> > > need to keep
> > > > > > > > > > the current JSON schema and not group plugins by type. As
> > > the current
> > > > > > > > > > schema already has a type field for each entry, we'll still
> > > be able to
> > > > > > > > > > tell them apart. Then we can have /worker-plugins and a
> > > relatively
> > > > > > > > > > clean API. What do you think?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Mickael
> > > > > > > > > >
> > > > > > > > > > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Mickael,
> > > > > > > > > > >
> > > > > > > > > > > I think one potential use case for exposing worker-level
> > > plugins is that
> > > > > > > > > > it
> > > > > > > > > > > may make it easier to determine whether a worker is set up
> > > correctly (the
> > > > > > > > > > > current alternative requires looking through log files and
> > > can be a
> > > > > > > > > > little
> > > > > > > > > > > tedious), and might even make it possible to automatically
> > > identify
> > > > > > > > > > > discrepancies within a cluster by diffing the contents of
> > > that endpoint
> > > > > > > > > > > across each worker. But I don't think this has to be
> > > addressed by the
> > > > > > > > > > > current KIP; the only thing that bothers me a little is
> > > that "plugins" is
> > > > > > > > > > > generic and it may confuse people down the road if we add
> > > an endpoint for
> > > > > > > > > > > worker-level plugins ("why is one just called 'plugins'
> > > and the other one
> > > > > > > > > > > is 'worker-plugins'?"). Probably not worth blocking on,
> > > though.
> > > > > > > > > > >
> > > > > > > > > > > Agreed that the suggestion for improved validation should
> > > be made on the
> > > > > > > > > > > KIP-802 thread.
> > > > > > > > > > >
> > > > > > > > > > > I also noticed that the newly-proposed config method for
> > > the Converter
> > > > > > > > > > > interface doesn't have a default implementation, making it
> > > > > > > > > > > backwards-incompatible. Should we add a default
> > > implementation that
> > > > > > > > > > returns
> > > > > > > > > > > either null or an empty ConfigDef?
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > Chris
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <
> > > mickael.maison@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > >
> > > > > > > > > > > > 1. If we want to expose worker plugins, I think we
> > > should do it via a
> > > > > > > > > > > > separate endpoint. But to be honest, I'm not even sure I
> > > see strong
> > > > > > > > > > > > use cases for exposing them as they are either enabled
> > > or not and
> > > > > > > > > > > > can't be changed at runtime. So I'd prefer to stick to
> > > "connector
> > > > > > > > > > > > level" plugins in this KIP. Let me now if you have use
> > > cases, I'm open
> > > > > > > > > > > > to reconsider this choice.
> > > > > > > > > > > > I'll add that in the rejected alternatives section for
> > > now
> > > > > > > > > > > >
> > > > > > > > > > > > 2. I remembered seeing issues in the past with multiple
> > > plugin.path
> > > > > > > > > > > > entries but I tried today and I was able to mix and
> > > match plugins from
> > > > > > > > > > > > different paths. So my bad for getting confused.
> > > > > > > > > > > > Then I agree, it makes more sense to group them by
> > > plugin type.
> > > > > > > > > > > >
> > > > > > > > > > > > 3. Yes this should be covered in KIP-802:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > > > > > > > > > >
> > > > > > > > > > > > 4. No particular reason. We can support both formats
> > > like today. I've
> > > > > > > > > > > > updated the KIP
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Mickael
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think the increase in scope here is great and the
> > > added value
> > > > > > > > > > certainly
> > > > > > > > > > > > > justifies the proposed changes. I have some thoughts
> > > but overall I
> > > > > > > > > > like
> > > > > > > > > > > > the
> > > > > > > > > > > > > direction this is going in now.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. The new /plugins endpoint is described as
> > > containing "all plugins
> > > > > > > > > > that
> > > > > > > > > > > > > are Connectors, Transformations, Converters,
> > > HeaderConverters and
> > > > > > > > > > > > > Predicates". So essentially, it looks like we want to
> > > expose all
> > > > > > > > > > plugins
> > > > > > > > > > > > > that are configured on a per-connector basis, but
> > > exclude plugins
> > > > > > > > > > that
> > > > > > > > > > > > are
> > > > > > > > > > > > > configured on a per-worker basis (such as config
> > > providers and REST
> > > > > > > > > > > > > extensions). Do you think it may be valuable to expose
> > > information on
> > > > > > > > > > > > > worker-level plugins as well?
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. The description for the new /plugins endpoint also
> > > states that
> > > > > > > > > > > > "Plugins
> > > > > > > > > > > > > will be grouped by plugin.path. This will make it
> > > clear to users
> > > > > > > > > > what's
> > > > > > > > > > > > > available to use as it's not possible to use a
> > > Connector from one
> > > > > > > > > > path
> > > > > > > > > > > > with
> > > > > > > > > > > > > Transformations from another.". Is this true? I
> > > thought that
> > > > > > > > > > Connect's
> > > > > > > > > > > > > classloading made it possible to package
> > > > > > > > > > > > > converters/transformations/predicates completely
> > > independently from
> > > > > > > > > > each
> > > > > > > > > > > > > other, and to reference them from
> > > also-independently-packaged
> > > > > > > > > > connectors.
> > > > > > > > > > > > > If it turns out that this is the case, could we
> > > consider
> > > > > > > > > > restructuring
> > > > > > > > > > > > the
> > > > > > > > > > > > > response to be grouped by plugin type instead of by
> > > classloader?
> > > > > > > > > > There's
> > > > > > > > > > > > > also the ungrouped format proposed in KIP-494 (
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > > > > > > > > > )
> > > > > > > > > > > > > which we might consider as well.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3. I think this can be left for a follow-up KIP if
> > > necessary, but I'm
> > > > > > > > > > > > > curious about your thoughts on adding new validate
> > > methods to all
> > > > > > > > > > > > > connector-level plugins that can be used similarly to
> > > how the
> > > > > > > > > > existing
> > > > > > > > > > > > > Connector::validate method (
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > > > > > > > > > )
> > > > > > > > > > > > > is used. This would allow for plugins to perform
> > > validation that's
> > > > > > > > > > more
> > > > > > > > > > > > > sophisticated than what the ConfigDef is capable of,
> > > such as
> > > > > > > > > > validating
> > > > > > > > > > > > > combinations of properties like a hostname and
> > > credentials for
> > > > > > > > > > reaching
> > > > > > > > > > > > it.
> > > > > > > > > > > > > I know that at least Confluent's Avro, protobuf, and
> > > JSON schema
> > > > > > > > > > > > converters
> > > > > > > > > > > > > would benefit from this kind of feature. It's a little
> > > tangential to
> > > > > > > > > > this
> > > > > > > > > > > > > KIP (which at the moment is about discovering plugins
> > > and their
> > > > > > > > > > > > > configuration surfaces, as opposed to validating
> > > them), but I
> > > > > > > > > > figured I'd
> > > > > > > > > > > > > ask since we're going to be expanding the Converter
> > > interface and it
> > > > > > > > > > may
> > > > > > > > > > > > be
> > > > > > > > > > > > > useful to tackle this while we're in the neighborhood.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 4. The description for the new
> > > /plugins/<type>/<name>/configdef
> > > > > > > > > > endpoint
> > > > > > > > > > > > > states that "Name must be the fully qualified class
> > > name of the
> > > > > > > > > > plugin".
> > > > > > > > > > > > > Any reason not to also support aliases (e.g.,
> > > > > > > > > > "FileStreamSinkConnector"
> > > > > > > > > > > > or
> > > > > > > > > > > > > "FileStreamSink" instead of
> > > > > > > > > > > > >
> > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Chris
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <
> > > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks all for the feedback!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Chris,
> > > > > > > > > > > > > > I agree that fixing the current endpoint helps a
> > > lot. Thanks for
> > > > > > > > > > > > > > raising these JIRAs and submitting a PR!
> > > > > > > > > > > > > > However thinking about the issue further, I decided
> > > to expand the
> > > > > > > > > > > > > > scope of the KIP to cover all user-visible plugins.
> > > > > > > > > > > > > > In practice, users want to know about all available
> > > plugins not
> > > > > > > > > > only
> > > > > > > > > > > > > > connectors. This includes transformations,
> > > converters,
> > > > > > > > > > > > > > header_converters and predicates. As we also want to
> > > retrieve
> > > > > > > > > > > > > > configdef for these too, I think it makes sense to
> > > introduce a new
> > > > > > > > > > > > > > endpoint to do so. Alongside we obviously need a new
> > > endpoint for
> > > > > > > > > > > > > > listing all plugins.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Gunnar,
> > > > > > > > > > > > > > I took a look at exposing valid values via the API.
> > > I think the
> > > > > > > > > > issue
> > > > > > > > > > > > > > is that Validators don't expose a way to retrieve
> > > valid values.
> > > > > > > > > > > > > > Changing validators will have an impact on all
> > > components so I'd
> > > > > > > > > > > > > > prefer to address this requirement in a separate
> > > KIP. I agree this
> > > > > > > > > > > > > > would be an interesting improvement and I'd happy to
> > > write a KIP
> > > > > > > > > > for
> > > > > > > > > > > > > > it too.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have updated the KIP accordingly. Let me know if
> > > you have further
> > > > > > > > > > > > > > feedback.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Mickael
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > > > > > > > > > > > > > <gu...@googlemail.com.invalid> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm +1 for adding a GET endpoint for obtaining
> > > config
> > > > > > > > > > definitions. It
> > > > > > > > > > > > > > > always felt odd to me that one has to issue a PUT
> > > for that
> > > > > > > > > > purpose.
> > > > > > > > > > > > If
> > > > > > > > > > > > > > > nothing else, it'd be better in terms of
> > > discoverability of the
> > > > > > > > > > KC
> > > > > > > > > > > > REST
> > > > > > > > > > > > > > API.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > One additional feature request I'd have is to
> > > expose the valid
> > > > > > > > > > enum
> > > > > > > > > > > > > > > constants for enum-typed options. That'll help to
> > > display the
> > > > > > > > > > values
> > > > > > > > > > > > in a
> > > > > > > > > > > > > > > drop-down or via radio buttons in a UI, give us
> > > tab completion in
> > > > > > > > > > > > kcctl,
> > > > > > > > > > > > > > > etc.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --Gunnar
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris
> > > Egerton
> > > > > > > > > > > > > > > <ch...@confluent.io.invalid>:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Viktor,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It sounds like there are three major points here
> > > in favor of a
> > > > > > > > > > new
> > > > > > > > > > > > GET
> > > > > > > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1. You cannot issue a blank ("dummy") request
> > > for sink
> > > > > > > > > > connectors
> > > > > > > > > > > > > > because a
> > > > > > > > > > > > > > > > topic list/topic regex has to be supplied
> > > (otherwise the PUT
> > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > returns a 500 response)
> > > > > > > > > > > > > > > > 2. A dummy request still triggers custom
> > > validations by the
> > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > which may be best to avoid if we know for sure
> > > that the config
> > > > > > > > > > > > isn't
> > > > > > > > > > > > > > worth
> > > > > > > > > > > > > > > > validating yet
> > > > > > > > > > > > > > > > 3. It's more ergonomic and intuitive to be able
> > > to issue a GET
> > > > > > > > > > > > request
> > > > > > > > > > > > > > > > without having to give a dummy connector config
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > With regards to 1, this is actually a bug in
> > > Connect (
> > > > > > > > > > > > > > > >
> > > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix
> > > > > > > > > > > > already
> > > > > > > > > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > > > > > > > > https://github.com/apache/kafka/pull/11369). I
> > > think it'd be
> > > > > > > > > > > > better to
> > > > > > > > > > > > > > > > focus on fixing this bug in general instead of
> > > implementing a
> > > > > > > > > > new
> > > > > > > > > > > > REST
> > > > > > > > > > > > > > > > endpoint in order to allow people to work around
> > > it.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > With regards to 2, this is technically possible
> > > but I'm unsure
> > > > > > > > > > > > it'd be
> > > > > > > > > > > > > > too
> > > > > > > > > > > > > > > > common out in the wild given that most
> > > validations that could
> > > > > > > > > > be
> > > > > > > > > > > > > > expensive
> > > > > > > > > > > > > > > > would involve things like connecting to a
> > > database, checking
> > > > > > > > > > if a
> > > > > > > > > > > > cloud
> > > > > > > > > > > > > > > > storage bucket exists, etc., none of which are
> > > possible without
> > > > > > > > > > > > some
> > > > > > > > > > > > > > > > configuration properties from the user (db
> > > hostname, bucket
> > > > > > > > > > name,
> > > > > > > > > > > > > > etc.).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > With regards to 3, I do agree that it'd be
> > > easier for people
> > > > > > > > > > > > designing
> > > > > > > > > > > > > > UIs
> > > > > > > > > > > > > > > > to have a GET API to work against. I'm just not
> > > sure it's
> > > > > > > > > > worth the
> > > > > > > > > > > > > > > > additional implementation, testing, and
> > > maintenance burden. If
> > > > > > > > > > it
> > > > > > > > > > > > were
> > > > > > > > > > > > > > > > possible to issue a PUT request without
> > > unexpected 500s for
> > > > > > > > > > invalid
> > > > > > > > > > > > > > > > configs, would that suffice? AFAICT it'd
> > > basically be as
> > > > > > > > > > simple as
> > > > > > > > > > > > > > issuing
> > > > > > > > > > > > > > > > a PUT request with a dummy body consisting of
> > > nothing except
> > > > > > > > > > the
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > class (which at this point we might even make
> > > unnecessary and
> > > > > > > > > > just
> > > > > > > > > > > > > > > > automatically replace with the connector class
> > > from the URL)
> > > > > > > > > > and
> > > > > > > > > > > > then
> > > > > > > > > > > > > > > > filtering the response to just grab the
> > > "definition" field of
> > > > > > > > > > each
> > > > > > > > > > > > > > element
> > > > > > > > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor
> > > Somogyi-Vass <
> > > > > > > > > > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Folks,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I too think this would be a very useful
> > > feature. Some of our
> > > > > > > > > > > > > > management
> > > > > > > > > > > > > > > > > applications would provide a wizard for
> > > creating connectors.
> > > > > > > > > > In
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > scenario the user basically would fill out a
> > > sample
> > > > > > > > > > configuration
> > > > > > > > > > > > > > > > generated
> > > > > > > > > > > > > > > > > by the UI which would send it back to Connect
> > > for validation
> > > > > > > > > > and
> > > > > > > > > > > > > > > > eventually
> > > > > > > > > > > > > > > > > create a new connector. The first part of this
> > > workflow can
> > > > > > > > > > be
> > > > > > > > > > > > > > enhanced
> > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > we had an API that can return the
> > > configuration definition
> > > > > > > > > > of the
> > > > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > type of connector as the UI application would
> > > be able to
> > > > > > > > > > > > generate a
> > > > > > > > > > > > > > > > sample
> > > > > > > > > > > > > > > > > for the user based on that (nicely drawn
> > > diagram:
> > > > > > > > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > > > > > > > The
> > > connector-plugins/{connectorType}/config/validate API
> > > > > > > > > > > > essentially
> > > > > > > > > > > > > > > > works
> > > > > > > > > > > > > > > > > and returns the data that we need, however it
> > > is a HTTP PUT
> > > > > > > > > > API
> > > > > > > > > > > > that
> > > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > bit unintuitive for a fetch-like functionality
> > > and also
> > > > > > > > > > > > functionally
> > > > > > > > > > > > > > > > > different as it validates the given (dummy)
> > > request. In case
> > > > > > > > > > of
> > > > > > > > > > > > sink
> > > > > > > > > > > > > > > > > connectors one would need to also provide a
> > > topic name.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > A suggestion for the KIP: I think it can be
> > > useful to return
> > > > > > > > > > the
> > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > groups and the connector class' name similarly
> > > to the
> > > > > > > > > > validate
> > > > > > > > > > > > API
> > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > case any frontend needs them (and also the
> > > response would be
> > > > > > > > > > more
> > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > validate API but simpler).
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Viktor
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > > > > > > > > ryannedolan@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think it'd be worth adding a GET version,
> > > fwiw. Could be
> > > > > > > > > > the
> > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > handler
> > > > > > > > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison
> > > <
> > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > You're right, you can achieve the same
> > > functionality
> > > > > > > > > > using
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > > > > > > > In my mind it was only for validation once
> > > you have
> > > > > > > > > > build a
> > > > > > > > > > > > > > > > > > > configuration but when used with an empty
> > > configuration,
> > > > > > > > > > it
> > > > > > > > > > > > > > basically
> > > > > > > > > > > > > > > > > > > serves the same purpose as the proposed
> > > new endpoint.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think it's a bit easier to use a GET
> > > endpoint but I
> > > > > > > > > > don't
> > > > > > > > > > > > > > think it
> > > > > > > > > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris
> > > Egerton
> > > > > > > > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I'm wondering about the use case here.
> > > The motivation
> > > > > > > > > > > > section
> > > > > > > > > > > > > > > > states
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > "Connect does not provide a way to see
> > > what
> > > > > > > > > > configurations
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > requires. Instead users have to go look
> > > at the
> > > > > > > > > > connector
> > > > > > > > > > > > > > > > > documentation
> > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > in the worst case, look directly at the
> > > connector
> > > > > > > > > > source
> > > > > > > > > > > > > > code.",
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > with this KIP, "users will be able to
> > > discover the
> > > > > > > > > > required
> > > > > > > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > > > > > > for connectors installed in a Connect
> > > cluster" and
> > > > > > > > > > "tools
> > > > > > > > > > > > will
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > generate wizards for configuring and
> > > starting
> > > > > > > > > > connectors".
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > > > > > > >
> > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > > > > > > > endpoint not address these points? What
> > > will the
> > > > > > > > > > > > newly-proposed
> > > > > > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > > > > allow users to do that they will not
> > > already be able
> > > > > > > > > > to do
> > > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael
> > > Maison <
> > > > > > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I've created KIP-769 to expose
> > > connector
> > > > > > > > > > configuration
> > > > > > > > > > > > > > > > definitions
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Please take a look and let me know if
> > > you have any
> > > > > > > > > > > > feedback.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > >

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Mickael Maison <mi...@gmail.com>]

Thanks Randall, updated.

On Sun, Dec 5, 2021 at 6:05 AM Arjun Satish <ar...@gmail.com> wrote:
>
> hey folks,
>
> Thanks a lot for the KIP and the discussions. Here are a couple of thoughts
> (apologies if I missed some point in this thread).
>
> 1. it seems like we are changing the behavior of /connector-plugins by now
> returning the list of transforms, converters as well? This would break
> backward compatibility, and clients would need to track what version of
> Connect workers they are querying to keep their experience consistent. I
> wonder if we can keep the old behavior, with one of the following methods:
>
> 1a. We add an "expand" multi-query to the /connector-plugin endpoint like
> how it was done in KIP-465 [1]. This would preserve backward compatibility,
> as well as allow you to get consolidated output. Acceptable values for the
> new param could be: "all" (to show all plugins), or
> "connectors|transforms|predicates" that show a subset of the plugin types.
>
> 1b. Query params might get crummy to extend. Alternatively, we could have
> an additional fine-grained paths for each plugin:
>
> # to list all the source and sink plugins
> /connector-plugins/connectors
>
> # to list all the SMT plugins
> /connector-plugins/transformations
>
> # to list all the predicate plugins
> /connector-plugins/predicates
>
> # to list all the converter plugins
> /connector-plugins/converters
>
> # to list all the header converter plugins
> /connector-plugins/header-converters
>
> This allows us to cleanly add new types of plugins down the road, and
> naturally extend to displaying attributes of these objects. Configs, for
> example, can be accessed with
>
> http://localhost:8083/connector-plugins/transformations/Cast$Value/config
>
> A migration path (for both the suggestions above) would be to add a
> deprecation notice on the current behavior of /connector-plugins (where,
> after this PR is merged, both /connector-plugins and
> /connector-plugins/connectors return the same output). And starting from
> the next major version release, /connector-plugins redirects to
> /connector-plugins/connectors (making it super easy for clients to catch
> up).
>
> 2. Regarding /worker-plugins, I agree that it is useful to have the REST
> API inform the state of the REST extensions, and Config Providers. However,
> users of the Connect clusters will rarely need to know how a config
> provider or REST extension is configured to construct data pipelines. It's
> mostly an admin task, AFAICT. Also, we have also seen these problems in
> Connect clusters where one host is somewhat differently set up than other
> (contents of plugin path, classpath, local worker configuration files, jvm
> versions, GC settings) that surface as issues in Connector behavior.
> Overall, I'm wondering if we should just tackle this problem in a separate
> KIP, that evaluates all of these cases. All of this might nicely fall into
> under the /admin endpoint and allow us to expose detailed info on the state
> of the worker (maybe a /admin/worker/ that returns all of this info).
>
> Thanks,
>
> [1]
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-465%3A+Add+Consolidated+Connector+Endpoint+to+Connect+REST+API
>
>
> On Sat, Dec 4, 2021 at 10:29 AM Randall Hauch <rh...@gmail.com> wrote:
>
> > Thanks, Mickael. Can we call out these as literals? For example:
> >
> > > The response structure of the objects in the array remain unchanged, but
> > this proposal does add "converter", "header_converter" and "transformation"
> > to the allowed values for the "type" field, in addition to the existing
> > "sink" and "source" values.
> >
> > Best regards,
> >
> > Randall
> >
> > On Sat, Dec 4, 2021 at 4:58 AM Mickael Maison <mi...@gmail.com>
> > wrote:
> > >
> > > Hi Randall,
> > >
> > > Sure. The types are sink, source, converter, header_converter,
> > > transformation and predicate. I've updated the KIP.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Fri, Dec 3, 2021 at 10:14 PM Randall Hauch <rh...@gmail.com> wrote:
> > > >
> > > > One more thing, inline below:
> > > >
> > > > Randall
> > > >
> > > > On Fri, Dec 3, 2021 at 10:38 AM Mickael Maison <
> > mickael.maison@gmail.com> wrote:
> > > > >
> > > > > Thanks Randall for the feedback!
> > > > >
> > > > > 1. Done
> > > > >
> > > > > 2. Yes that makes sense. I've updated the KIP
> > > > >
> > > > > 3.a I've updated to explicitly mention the plugin can be provided
> > > > > using the fully qualified class or the alias.
> > > > > 3.b Only concrete classes will be returned. I made the mistake when I
> > > > > crafted the examples by hand. Fixed
> > > >
> > > > Can we also enumerate all of the values that can be included in the
> > > > "type" field in the response from the "GET
> > > > /connector-plugins?connectorsOnly=false"?
> > > >
> > > > >
> > > > > 4. If a plugin does not override config(), the response is an empty
> > > > > array. I've updated the KIP
> > > > >
> > > > > 5. I have updated the compatibility section with more details
> > > > >
> > > > > 6. Can you clarify what you mean? KIP-507 was for securing internal
> > > > > REST endpoints.
> > > > >
> > > > > 7. I don't think it is necessary. I think the use case Chris
> > mentioned
> > > > > could be useful for admins. I'm interested in exposing the
> > > > > connector-level plugins and their configs. I thought it was easy to
> > > > > add an API for worker plugins while working in this area but if there
> > > > > isn't a strong consensus, I'm happy to drop it.
> > > > >
> > > > > Mickael
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch <rh...@gmail.com>
> > wrote:
> > > > > >
> > > > > > Thanks for the KIP, Mickael. And thanks to everyone else for the
> > good
> > > > > > discussion so far.
> > > > > >
> > > > > > Overall this is pretty good, but I do have a few
> > questions/comments:
> > > > > > 1. Can we include the HTTP method/verb in each of the endpoints in
> > the
> > > > > > "Public Interfaces" section?
> > > > > > 2. The sample output for `GET
> > /connector-plugins?connectorsOnly=false`
> > > > > > includes a `version` field for the connector plugin, but not for
> > the
> > > > > > transform or converter plugins. Neither the `Transformation` or
> > > > > > `Converter` interfaces have a version nor implement the `Versioned`
> > > > > > interface, so should we add the `Versioned` interface to the
> > > > > > `Transformation` and `Converter` interfaces and add a default
> > method
> > > > > > that returns "unknown"? Yes, we could do that in a separate KIP,
> > but I
> > > > > > think it's appropriate to do it in this KIP for a few reasons.
> > First,
> > > > > > we're already modifying the `Converter` interface in this KIP and
> > it
> > > > > > would be quite straightforward to also extend the `Versioned`
> > > > > > interface and add a default method; then doing that for
> > > > > > `Transformation` would also be relatively straightforward. Second,
> > > > > > this KIP provides the justification for adding a version to the
> > > > > > `Transformation` and `Converter` interfaces.
> > > > > > 3. In the "Public Interfaces" section, there are two
> > inconsistencies
> > > > > > between the `GET /connector-plugins` and `GET
> > > > > > /connector-plugins/<plugin>/config` methods. Both will make it
> > > > > > difficult to use the second method, since the input to the second
> > > > > > method is not directly found in the response from the first method:
> > > > > >  a. The example for `GET /connector-plugins/Cast$Value/config` uses
> > > > > > short name, or what Connect often refers to as the alias,
> > "Cast$Value"
> > > > > > for the plugin, rather than the fully-qualified name
> > > > > > "org.apache.kafka.connect.transforms.Cast$Value". The KIP should
> > > > > > specify whether it will support both the shortened name and the
> > > > > > fully-qualified name, or whether only one of these will be
> > supported.
> > > > > > Not supporting the fully-qualified name will make it impossible to
> > use
> > > > > > the response from the first method as an input to the second.
> > > > > >  a. The example for the list method returns the `Transformation`
> > base
> > > > > > and abstract classes (e.g.,
> > > > > > "org.apache.kafka.connect.transforms.Cast") rather than the
> > concrete
> > > > > > classes (e.g., "org.apache.kafka.connect.transforms.Cast$Value" and
> > > > > > "org.apache.kafka.connect.transforms.Cast$Key"), while the `GET
> > > > > > /connector-plugins/<plugin>/config` will require the concrete
> > class.
> > > > > > 4. The "Public Interfaces" section's description of the `GET
> > > > > > /connector-plugins/<plugin>/config` method should probably specify
> > the
> > > > > > expected behavior for `Converter` implementations that don't
> > override
> > > > > > the new `config()` method.
> > > > > > 5. The "Compatibility, Deprecation, and Migration Plan" section
> > should
> > > > > > describe whether older `Converter` implementations (compiled with
> > > > > > older versions of the Connect API without Converter extending
> > > > > > Configurable) will be able to be installed into newer Connect
> > > > > > runtimes, and what the behavior will be for the `GET
> > > > > > /connector-plugins/<plugin>/config` method. (Although my suggestion
> > > > > > for #5 will partially address the latter, we still need to talk
> > about
> > > > > > it in terms of compatibility.)
> > > > > > 6. Can we add a small section addressing the security implications
> > of
> > > > > > these new APIs? KIP-507 [1] applied to APIs that existed at the
> > time,
> > > > > > and this KIP should at least mention how it aligns with that
> > > > > > mechanism.
> > > > > > 7. Do we really think adding `GET /worker-plugins` is necessary?
> > While
> > > > > > I understand the reasoning, I'm less convinced that it's worth
> > adding
> > > > > > to the public API. First, an administrator still has to modify the
> > > > > > worker config for all workers to take advantage of that. Second,
> > > > > > adding this method increases the surface area and therefore risk of
> > > > > > additional attack vectors, despite not really being able to use the
> > > > > > resulting information in any other part of the API.
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Randall
> > > > > >
> > > > > > [1]
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
> > > > > >
> > > > > >
> > > > > > On Wed, Dec 1, 2021 at 7:52 AM Mickael Maison <
> > mickael.maison@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi Chris,
> > > > > > >
> > > > > > > Thanks again for the feedback! I've updated the KIP based on our
> > last
> > > > > > > discussions. I've decided to include the new endpoint for worker
> > > > > > > plugins.
> > > > > > >
> > > > > > > 1. Yes I agree, it's best to gate the new behavior.
> > > > > > > 2. Yes, it was a remnant from the original proposal. I've now
> > removed
> > > > > > > the location field.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mickael
> > > > > > >
> > > > > > > On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
> > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > >
> > > > > > > > Hi Mickael,
> > > > > > > >
> > > > > > > > I think that's a great idea! I especially like how we can
> > establish the
> > > > > > > > expectation that any plugin type that appears in the response
> > from the GET
> > > > > > > > /connector-plugins endpoint will have a corresponding GET
> > > > > > > > /connector-plugins/<type>/config endpoint, but (if we decide
> > to add them in
> > > > > > > > the future), worker plugins won't be expected to expose this
> > kind of
> > > > > > > > information and the different root path helps give a decent
> > hint about this.
> > > > > > > >
> > > > > > > > I also like the choice to return an empty ConfigDef from
> > Converter::config
> > > > > > > > instead of null.
> > > > > > > >
> > > > > > > > Two things come to mind:
> > > > > > > >
> > > > > > > > 1. We may want to gate this behind a URL query parameter
> > (maybe something
> > > > > > > > like "connectorsOnly") that defaults to the old behavior in
> > order to avoid
> > > > > > > > breaking existing tools such as programmatic UIs that use the
> > endpoint
> > > > > > > > today to discover the connectors that can be created by the
> > user. We can
> > > > > > > > even plan to change the default for that parameter to the
> > newly-proposed
> > > > > > > > behavior in the next major release, which should give people
> > enough time to
> > > > > > > > either adapt to the expanded response format or add the query
> > parameter to
> > > > > > > > their tooling.
> > > > > > > >
> > > > > > > > 2. The existing GET /connector-plugins endpoint doesn't
> > contain information
> > > > > > > > on the location of the plugin on the worker's file system. Do
> > you think we
> > > > > > > > should still include this info in the new response format?
> > Correct me if
> > > > > > > > I'm wrong but it seems it may have been proposed originally to
> > help prevent
> > > > > > > > already-addressed bugs in Connect classloading from striking;
> > all else
> > > > > > > > equal, I'd personally err on the side of leaving this info out
> > or at least
> > > > > > > > reducing permitted values for it to just "classpath" or
> > "plugin path" in
> > > > > > > > order to avoid leaking worker file paths into the REST API,
> > which might
> > > > > > > > bother super security-conscious users.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <
> > mickael.maison@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Chris,
> > > > > > > > >
> > > > > > > > > Yes to keep compatibility we want a default implementation
> > for
> > > > > > > > > Converter.configs(), I've updated the KIP.
> > > > > > > > >
> > > > > > > > > Regarding worker plugins, the use case you described seems
> > valuable.
> > > > > > > > > I'd prefer not mixing worker and connector plugins on the
> > same
> > > > > > > > > endpoint but I agree using /plugins and /worker-plugins
> > could be
> > > > > > > > > confusing.
> > > > > > > > >
> > > > > > > > > One alternative is to expose all connector-level plugins via
> > the
> > > > > > > > > existing /connector-plugins endpoint. In that case, we'd
> > need to keep
> > > > > > > > > the current JSON schema and not group plugins by type. As
> > the current
> > > > > > > > > schema already has a type field for each entry, we'll still
> > be able to
> > > > > > > > > tell them apart. Then we can have /worker-plugins and a
> > relatively
> > > > > > > > > clean API. What do you think?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Mickael
> > > > > > > > >
> > > > > > > > > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Mickael,
> > > > > > > > > >
> > > > > > > > > > I think one potential use case for exposing worker-level
> > plugins is that
> > > > > > > > > it
> > > > > > > > > > may make it easier to determine whether a worker is set up
> > correctly (the
> > > > > > > > > > current alternative requires looking through log files and
> > can be a
> > > > > > > > > little
> > > > > > > > > > tedious), and might even make it possible to automatically
> > identify
> > > > > > > > > > discrepancies within a cluster by diffing the contents of
> > that endpoint
> > > > > > > > > > across each worker. But I don't think this has to be
> > addressed by the
> > > > > > > > > > current KIP; the only thing that bothers me a little is
> > that "plugins" is
> > > > > > > > > > generic and it may confuse people down the road if we add
> > an endpoint for
> > > > > > > > > > worker-level plugins ("why is one just called 'plugins'
> > and the other one
> > > > > > > > > > is 'worker-plugins'?"). Probably not worth blocking on,
> > though.
> > > > > > > > > >
> > > > > > > > > > Agreed that the suggestion for improved validation should
> > be made on the
> > > > > > > > > > KIP-802 thread.
> > > > > > > > > >
> > > > > > > > > > I also noticed that the newly-proposed config method for
> > the Converter
> > > > > > > > > > interface doesn't have a default implementation, making it
> > > > > > > > > > backwards-incompatible. Should we add a default
> > implementation that
> > > > > > > > > returns
> > > > > > > > > > either null or an empty ConfigDef?
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > >
> > > > > > > > > > Chris
> > > > > > > > > >
> > > > > > > > > > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <
> > mickael.maison@gmail.com
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Chris,
> > > > > > > > > > >
> > > > > > > > > > > 1. If we want to expose worker plugins, I think we
> > should do it via a
> > > > > > > > > > > separate endpoint. But to be honest, I'm not even sure I
> > see strong
> > > > > > > > > > > use cases for exposing them as they are either enabled
> > or not and
> > > > > > > > > > > can't be changed at runtime. So I'd prefer to stick to
> > "connector
> > > > > > > > > > > level" plugins in this KIP. Let me now if you have use
> > cases, I'm open
> > > > > > > > > > > to reconsider this choice.
> > > > > > > > > > > I'll add that in the rejected alternatives section for
> > now
> > > > > > > > > > >
> > > > > > > > > > > 2. I remembered seeing issues in the past with multiple
> > plugin.path
> > > > > > > > > > > entries but I tried today and I was able to mix and
> > match plugins from
> > > > > > > > > > > different paths. So my bad for getting confused.
> > > > > > > > > > > Then I agree, it makes more sense to group them by
> > plugin type.
> > > > > > > > > > >
> > > > > > > > > > > 3. Yes this should be covered in KIP-802:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > > > > > > > > >
> > > > > > > > > > > 4. No particular reason. We can support both formats
> > like today. I've
> > > > > > > > > > > updated the KIP
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Mickael
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > >
> > > > > > > > > > > > I think the increase in scope here is great and the
> > added value
> > > > > > > > > certainly
> > > > > > > > > > > > justifies the proposed changes. I have some thoughts
> > but overall I
> > > > > > > > > like
> > > > > > > > > > > the
> > > > > > > > > > > > direction this is going in now.
> > > > > > > > > > > >
> > > > > > > > > > > > 1. The new /plugins endpoint is described as
> > containing "all plugins
> > > > > > > > > that
> > > > > > > > > > > > are Connectors, Transformations, Converters,
> > HeaderConverters and
> > > > > > > > > > > > Predicates". So essentially, it looks like we want to
> > expose all
> > > > > > > > > plugins
> > > > > > > > > > > > that are configured on a per-connector basis, but
> > exclude plugins
> > > > > > > > > that
> > > > > > > > > > > are
> > > > > > > > > > > > configured on a per-worker basis (such as config
> > providers and REST
> > > > > > > > > > > > extensions). Do you think it may be valuable to expose
> > information on
> > > > > > > > > > > > worker-level plugins as well?
> > > > > > > > > > > >
> > > > > > > > > > > > 2. The description for the new /plugins endpoint also
> > states that
> > > > > > > > > > > "Plugins
> > > > > > > > > > > > will be grouped by plugin.path. This will make it
> > clear to users
> > > > > > > > > what's
> > > > > > > > > > > > available to use as it's not possible to use a
> > Connector from one
> > > > > > > > > path
> > > > > > > > > > > with
> > > > > > > > > > > > Transformations from another.". Is this true? I
> > thought that
> > > > > > > > > Connect's
> > > > > > > > > > > > classloading made it possible to package
> > > > > > > > > > > > converters/transformations/predicates completely
> > independently from
> > > > > > > > > each
> > > > > > > > > > > > other, and to reference them from
> > also-independently-packaged
> > > > > > > > > connectors.
> > > > > > > > > > > > If it turns out that this is the case, could we
> > consider
> > > > > > > > > restructuring
> > > > > > > > > > > the
> > > > > > > > > > > > response to be grouped by plugin type instead of by
> > classloader?
> > > > > > > > > There's
> > > > > > > > > > > > also the ungrouped format proposed in KIP-494 (
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > > > > > > > > )
> > > > > > > > > > > > which we might consider as well.
> > > > > > > > > > > >
> > > > > > > > > > > > 3. I think this can be left for a follow-up KIP if
> > necessary, but I'm
> > > > > > > > > > > > curious about your thoughts on adding new validate
> > methods to all
> > > > > > > > > > > > connector-level plugins that can be used similarly to
> > how the
> > > > > > > > > existing
> > > > > > > > > > > > Connector::validate method (
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > > > > > > > > )
> > > > > > > > > > > > is used. This would allow for plugins to perform
> > validation that's
> > > > > > > > > more
> > > > > > > > > > > > sophisticated than what the ConfigDef is capable of,
> > such as
> > > > > > > > > validating
> > > > > > > > > > > > combinations of properties like a hostname and
> > credentials for
> > > > > > > > > reaching
> > > > > > > > > > > it.
> > > > > > > > > > > > I know that at least Confluent's Avro, protobuf, and
> > JSON schema
> > > > > > > > > > > converters
> > > > > > > > > > > > would benefit from this kind of feature. It's a little
> > tangential to
> > > > > > > > > this
> > > > > > > > > > > > KIP (which at the moment is about discovering plugins
> > and their
> > > > > > > > > > > > configuration surfaces, as opposed to validating
> > them), but I
> > > > > > > > > figured I'd
> > > > > > > > > > > > ask since we're going to be expanding the Converter
> > interface and it
> > > > > > > > > may
> > > > > > > > > > > be
> > > > > > > > > > > > useful to tackle this while we're in the neighborhood.
> > > > > > > > > > > >
> > > > > > > > > > > > 4. The description for the new
> > /plugins/<type>/<name>/configdef
> > > > > > > > > endpoint
> > > > > > > > > > > > states that "Name must be the fully qualified class
> > name of the
> > > > > > > > > plugin".
> > > > > > > > > > > > Any reason not to also support aliases (e.g.,
> > > > > > > > > "FileStreamSinkConnector"
> > > > > > > > > > > or
> > > > > > > > > > > > "FileStreamSink" instead of
> > > > > > > > > > > >
> > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > >
> > > > > > > > > > > > Chris
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <
> > > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks all for the feedback!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Chris,
> > > > > > > > > > > > > I agree that fixing the current endpoint helps a
> > lot. Thanks for
> > > > > > > > > > > > > raising these JIRAs and submitting a PR!
> > > > > > > > > > > > > However thinking about the issue further, I decided
> > to expand the
> > > > > > > > > > > > > scope of the KIP to cover all user-visible plugins.
> > > > > > > > > > > > > In practice, users want to know about all available
> > plugins not
> > > > > > > > > only
> > > > > > > > > > > > > connectors. This includes transformations,
> > converters,
> > > > > > > > > > > > > header_converters and predicates. As we also want to
> > retrieve
> > > > > > > > > > > > > configdef for these too, I think it makes sense to
> > introduce a new
> > > > > > > > > > > > > endpoint to do so. Alongside we obviously need a new
> > endpoint for
> > > > > > > > > > > > > listing all plugins.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Gunnar,
> > > > > > > > > > > > > I took a look at exposing valid values via the API.
> > I think the
> > > > > > > > > issue
> > > > > > > > > > > > > is that Validators don't expose a way to retrieve
> > valid values.
> > > > > > > > > > > > > Changing validators will have an impact on all
> > components so I'd
> > > > > > > > > > > > > prefer to address this requirement in a separate
> > KIP. I agree this
> > > > > > > > > > > > > would be an interesting improvement and I'd happy to
> > write a KIP
> > > > > > > > > for
> > > > > > > > > > > > > it too.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have updated the KIP accordingly. Let me know if
> > you have further
> > > > > > > > > > > > > feedback.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Mickael
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > > > > > > > > > > > > <gu...@googlemail.com.invalid> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'm +1 for adding a GET endpoint for obtaining
> > config
> > > > > > > > > definitions. It
> > > > > > > > > > > > > > always felt odd to me that one has to issue a PUT
> > for that
> > > > > > > > > purpose.
> > > > > > > > > > > If
> > > > > > > > > > > > > > nothing else, it'd be better in terms of
> > discoverability of the
> > > > > > > > > KC
> > > > > > > > > > > REST
> > > > > > > > > > > > > API.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > One additional feature request I'd have is to
> > expose the valid
> > > > > > > > > enum
> > > > > > > > > > > > > > constants for enum-typed options. That'll help to
> > display the
> > > > > > > > > values
> > > > > > > > > > > in a
> > > > > > > > > > > > > > drop-down or via radio buttons in a UI, give us
> > tab completion in
> > > > > > > > > > > kcctl,
> > > > > > > > > > > > > > etc.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --Gunnar
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris
> > Egerton
> > > > > > > > > > > > > > <ch...@confluent.io.invalid>:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Viktor,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It sounds like there are three major points here
> > in favor of a
> > > > > > > > > new
> > > > > > > > > > > GET
> > > > > > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. You cannot issue a blank ("dummy") request
> > for sink
> > > > > > > > > connectors
> > > > > > > > > > > > > because a
> > > > > > > > > > > > > > > topic list/topic regex has to be supplied
> > (otherwise the PUT
> > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > returns a 500 response)
> > > > > > > > > > > > > > > 2. A dummy request still triggers custom
> > validations by the
> > > > > > > > > > > connector,
> > > > > > > > > > > > > > > which may be best to avoid if we know for sure
> > that the config
> > > > > > > > > > > isn't
> > > > > > > > > > > > > worth
> > > > > > > > > > > > > > > validating yet
> > > > > > > > > > > > > > > 3. It's more ergonomic and intuitive to be able
> > to issue a GET
> > > > > > > > > > > request
> > > > > > > > > > > > > > > without having to give a dummy connector config
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > With regards to 1, this is actually a bug in
> > Connect (
> > > > > > > > > > > > > > >
> > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix
> > > > > > > > > > > already
> > > > > > > > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > > > > > > > https://github.com/apache/kafka/pull/11369). I
> > think it'd be
> > > > > > > > > > > better to
> > > > > > > > > > > > > > > focus on fixing this bug in general instead of
> > implementing a
> > > > > > > > > new
> > > > > > > > > > > REST
> > > > > > > > > > > > > > > endpoint in order to allow people to work around
> > it.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > With regards to 2, this is technically possible
> > but I'm unsure
> > > > > > > > > > > it'd be
> > > > > > > > > > > > > too
> > > > > > > > > > > > > > > common out in the wild given that most
> > validations that could
> > > > > > > > > be
> > > > > > > > > > > > > expensive
> > > > > > > > > > > > > > > would involve things like connecting to a
> > database, checking
> > > > > > > > > if a
> > > > > > > > > > > cloud
> > > > > > > > > > > > > > > storage bucket exists, etc., none of which are
> > possible without
> > > > > > > > > > > some
> > > > > > > > > > > > > > > configuration properties from the user (db
> > hostname, bucket
> > > > > > > > > name,
> > > > > > > > > > > > > etc.).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > With regards to 3, I do agree that it'd be
> > easier for people
> > > > > > > > > > > designing
> > > > > > > > > > > > > UIs
> > > > > > > > > > > > > > > to have a GET API to work against. I'm just not
> > sure it's
> > > > > > > > > worth the
> > > > > > > > > > > > > > > additional implementation, testing, and
> > maintenance burden. If
> > > > > > > > > it
> > > > > > > > > > > were
> > > > > > > > > > > > > > > possible to issue a PUT request without
> > unexpected 500s for
> > > > > > > > > invalid
> > > > > > > > > > > > > > > configs, would that suffice? AFAICT it'd
> > basically be as
> > > > > > > > > simple as
> > > > > > > > > > > > > issuing
> > > > > > > > > > > > > > > a PUT request with a dummy body consisting of
> > nothing except
> > > > > > > > > the
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > class (which at this point we might even make
> > unnecessary and
> > > > > > > > > just
> > > > > > > > > > > > > > > automatically replace with the connector class
> > from the URL)
> > > > > > > > > and
> > > > > > > > > > > then
> > > > > > > > > > > > > > > filtering the response to just grab the
> > "definition" field of
> > > > > > > > > each
> > > > > > > > > > > > > element
> > > > > > > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor
> > Somogyi-Vass <
> > > > > > > > > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Folks,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I too think this would be a very useful
> > feature. Some of our
> > > > > > > > > > > > > management
> > > > > > > > > > > > > > > > applications would provide a wizard for
> > creating connectors.
> > > > > > > > > In
> > > > > > > > > > > this
> > > > > > > > > > > > > > > > scenario the user basically would fill out a
> > sample
> > > > > > > > > configuration
> > > > > > > > > > > > > > > generated
> > > > > > > > > > > > > > > > by the UI which would send it back to Connect
> > for validation
> > > > > > > > > and
> > > > > > > > > > > > > > > eventually
> > > > > > > > > > > > > > > > create a new connector. The first part of this
> > workflow can
> > > > > > > > > be
> > > > > > > > > > > > > enhanced
> > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > we had an API that can return the
> > configuration definition
> > > > > > > > > of the
> > > > > > > > > > > > > given
> > > > > > > > > > > > > > > > type of connector as the UI application would
> > be able to
> > > > > > > > > > > generate a
> > > > > > > > > > > > > > > sample
> > > > > > > > > > > > > > > > for the user based on that (nicely drawn
> > diagram:
> > > > > > > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > > > > > > The
> > connector-plugins/{connectorType}/config/validate API
> > > > > > > > > > > essentially
> > > > > > > > > > > > > > > works
> > > > > > > > > > > > > > > > and returns the data that we need, however it
> > is a HTTP PUT
> > > > > > > > > API
> > > > > > > > > > > that
> > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > bit unintuitive for a fetch-like functionality
> > and also
> > > > > > > > > > > functionally
> > > > > > > > > > > > > > > > different as it validates the given (dummy)
> > request. In case
> > > > > > > > > of
> > > > > > > > > > > sink
> > > > > > > > > > > > > > > > connectors one would need to also provide a
> > topic name.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > A suggestion for the KIP: I think it can be
> > useful to return
> > > > > > > > > the
> > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > groups and the connector class' name similarly
> > to the
> > > > > > > > > validate
> > > > > > > > > > > API
> > > > > > > > > > > > > just
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > case any frontend needs them (and also the
> > response would be
> > > > > > > > > more
> > > > > > > > > > > > > like
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > validate API but simpler).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Viktor
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > > > > > > > ryannedolan@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think it'd be worth adding a GET version,
> > fwiw. Could be
> > > > > > > > > the
> > > > > > > > > > > same
> > > > > > > > > > > > > > > > handler
> > > > > > > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison
> > <
> > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > You're right, you can achieve the same
> > functionality
> > > > > > > > > using
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > > > > > > In my mind it was only for validation once
> > you have
> > > > > > > > > build a
> > > > > > > > > > > > > > > > > > configuration but when used with an empty
> > configuration,
> > > > > > > > > it
> > > > > > > > > > > > > basically
> > > > > > > > > > > > > > > > > > serves the same purpose as the proposed
> > new endpoint.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think it's a bit easier to use a GET
> > endpoint but I
> > > > > > > > > don't
> > > > > > > > > > > > > think it
> > > > > > > > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris
> > Egerton
> > > > > > > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I'm wondering about the use case here.
> > The motivation
> > > > > > > > > > > section
> > > > > > > > > > > > > > > states
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > "Connect does not provide a way to see
> > what
> > > > > > > > > configurations
> > > > > > > > > > > a
> > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > requires. Instead users have to go look
> > at the
> > > > > > > > > connector
> > > > > > > > > > > > > > > > documentation
> > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > in the worst case, look directly at the
> > connector
> > > > > > > > > source
> > > > > > > > > > > > > code.",
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > with this KIP, "users will be able to
> > discover the
> > > > > > > > > required
> > > > > > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > > > > > for connectors installed in a Connect
> > cluster" and
> > > > > > > > > "tools
> > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > generate wizards for configuring and
> > starting
> > > > > > > > > connectors".
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > > > > > >
> > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > > > > > > endpoint not address these points? What
> > will the
> > > > > > > > > > > newly-proposed
> > > > > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > > > allow users to do that they will not
> > already be able
> > > > > > > > > to do
> > > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael
> > Maison <
> > > > > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I've created KIP-769 to expose
> > connector
> > > > > > > > > configuration
> > > > > > > > > > > > > > > definitions
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Please take a look and let me know if
> > you have any
> > > > > > > > > > > feedback.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> >

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Arjun Satish <ar...@gmail.com>]

hey folks,

Thanks a lot for the KIP and the discussions. Here are a couple of thoughts
(apologies if I missed some point in this thread).

1. it seems like we are changing the behavior of /connector-plugins by now
returning the list of transforms, converters as well? This would break
backward compatibility, and clients would need to track what version of
Connect workers they are querying to keep their experience consistent. I
wonder if we can keep the old behavior, with one of the following methods:

1a. We add an "expand" multi-query to the /connector-plugin endpoint like
how it was done in KIP-465 [1]. This would preserve backward compatibility,
as well as allow you to get consolidated output. Acceptable values for the
new param could be: "all" (to show all plugins), or
"connectors|transforms|predicates" that show a subset of the plugin types.

1b. Query params might get crummy to extend. Alternatively, we could have
an additional fine-grained paths for each plugin:

# to list all the source and sink plugins
/connector-plugins/connectors

# to list all the SMT plugins
/connector-plugins/transformations

# to list all the predicate plugins
/connector-plugins/predicates

# to list all the converter plugins
/connector-plugins/converters

# to list all the header converter plugins
/connector-plugins/header-converters

This allows us to cleanly add new types of plugins down the road, and
naturally extend to displaying attributes of these objects. Configs, for
example, can be accessed with

http://localhost:8083/connector-plugins/transformations/Cast$Value/config

A migration path (for both the suggestions above) would be to add a
deprecation notice on the current behavior of /connector-plugins (where,
after this PR is merged, both /connector-plugins and
/connector-plugins/connectors return the same output). And starting from
the next major version release, /connector-plugins redirects to
/connector-plugins/connectors (making it super easy for clients to catch
up).

2. Regarding /worker-plugins, I agree that it is useful to have the REST
API inform the state of the REST extensions, and Config Providers. However,
users of the Connect clusters will rarely need to know how a config
provider or REST extension is configured to construct data pipelines. It's
mostly an admin task, AFAICT. Also, we have also seen these problems in
Connect clusters where one host is somewhat differently set up than other
(contents of plugin path, classpath, local worker configuration files, jvm
versions, GC settings) that surface as issues in Connector behavior.
Overall, I'm wondering if we should just tackle this problem in a separate
KIP, that evaluates all of these cases. All of this might nicely fall into
under the /admin endpoint and allow us to expose detailed info on the state
of the worker (maybe a /admin/worker/ that returns all of this info).

Thanks,

[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-465%3A+Add+Consolidated+Connector+Endpoint+to+Connect+REST+API


On Sat, Dec 4, 2021 at 10:29 AM Randall Hauch <rh...@gmail.com> wrote:

> Thanks, Mickael. Can we call out these as literals? For example:
>
> > The response structure of the objects in the array remain unchanged, but
> this proposal does add "converter", "header_converter" and "transformation"
> to the allowed values for the "type" field, in addition to the existing
> "sink" and "source" values.
>
> Best regards,
>
> Randall
>
> On Sat, Dec 4, 2021 at 4:58 AM Mickael Maison <mi...@gmail.com>
> wrote:
> >
> > Hi Randall,
> >
> > Sure. The types are sink, source, converter, header_converter,
> > transformation and predicate. I've updated the KIP.
> >
> > Thanks,
> > Mickael
> >
> > On Fri, Dec 3, 2021 at 10:14 PM Randall Hauch <rh...@gmail.com> wrote:
> > >
> > > One more thing, inline below:
> > >
> > > Randall
> > >
> > > On Fri, Dec 3, 2021 at 10:38 AM Mickael Maison <
> mickael.maison@gmail.com> wrote:
> > > >
> > > > Thanks Randall for the feedback!
> > > >
> > > > 1. Done
> > > >
> > > > 2. Yes that makes sense. I've updated the KIP
> > > >
> > > > 3.a I've updated to explicitly mention the plugin can be provided
> > > > using the fully qualified class or the alias.
> > > > 3.b Only concrete classes will be returned. I made the mistake when I
> > > > crafted the examples by hand. Fixed
> > >
> > > Can we also enumerate all of the values that can be included in the
> > > "type" field in the response from the "GET
> > > /connector-plugins?connectorsOnly=false"?
> > >
> > > >
> > > > 4. If a plugin does not override config(), the response is an empty
> > > > array. I've updated the KIP
> > > >
> > > > 5. I have updated the compatibility section with more details
> > > >
> > > > 6. Can you clarify what you mean? KIP-507 was for securing internal
> > > > REST endpoints.
> > > >
> > > > 7. I don't think it is necessary. I think the use case Chris
> mentioned
> > > > could be useful for admins. I'm interested in exposing the
> > > > connector-level plugins and their configs. I thought it was easy to
> > > > add an API for worker plugins while working in this area but if there
> > > > isn't a strong consensus, I'm happy to drop it.
> > > >
> > > > Mickael
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch <rh...@gmail.com>
> wrote:
> > > > >
> > > > > Thanks for the KIP, Mickael. And thanks to everyone else for the
> good
> > > > > discussion so far.
> > > > >
> > > > > Overall this is pretty good, but I do have a few
> questions/comments:
> > > > > 1. Can we include the HTTP method/verb in each of the endpoints in
> the
> > > > > "Public Interfaces" section?
> > > > > 2. The sample output for `GET
> /connector-plugins?connectorsOnly=false`
> > > > > includes a `version` field for the connector plugin, but not for
> the
> > > > > transform or converter plugins. Neither the `Transformation` or
> > > > > `Converter` interfaces have a version nor implement the `Versioned`
> > > > > interface, so should we add the `Versioned` interface to the
> > > > > `Transformation` and `Converter` interfaces and add a default
> method
> > > > > that returns "unknown"? Yes, we could do that in a separate KIP,
> but I
> > > > > think it's appropriate to do it in this KIP for a few reasons.
> First,
> > > > > we're already modifying the `Converter` interface in this KIP and
> it
> > > > > would be quite straightforward to also extend the `Versioned`
> > > > > interface and add a default method; then doing that for
> > > > > `Transformation` would also be relatively straightforward. Second,
> > > > > this KIP provides the justification for adding a version to the
> > > > > `Transformation` and `Converter` interfaces.
> > > > > 3. In the "Public Interfaces" section, there are two
> inconsistencies
> > > > > between the `GET /connector-plugins` and `GET
> > > > > /connector-plugins/<plugin>/config` methods. Both will make it
> > > > > difficult to use the second method, since the input to the second
> > > > > method is not directly found in the response from the first method:
> > > > >  a. The example for `GET /connector-plugins/Cast$Value/config` uses
> > > > > short name, or what Connect often refers to as the alias,
> "Cast$Value"
> > > > > for the plugin, rather than the fully-qualified name
> > > > > "org.apache.kafka.connect.transforms.Cast$Value". The KIP should
> > > > > specify whether it will support both the shortened name and the
> > > > > fully-qualified name, or whether only one of these will be
> supported.
> > > > > Not supporting the fully-qualified name will make it impossible to
> use
> > > > > the response from the first method as an input to the second.
> > > > >  a. The example for the list method returns the `Transformation`
> base
> > > > > and abstract classes (e.g.,
> > > > > "org.apache.kafka.connect.transforms.Cast") rather than the
> concrete
> > > > > classes (e.g., "org.apache.kafka.connect.transforms.Cast$Value" and
> > > > > "org.apache.kafka.connect.transforms.Cast$Key"), while the `GET
> > > > > /connector-plugins/<plugin>/config` will require the concrete
> class.
> > > > > 4. The "Public Interfaces" section's description of the `GET
> > > > > /connector-plugins/<plugin>/config` method should probably specify
> the
> > > > > expected behavior for `Converter` implementations that don't
> override
> > > > > the new `config()` method.
> > > > > 5. The "Compatibility, Deprecation, and Migration Plan" section
> should
> > > > > describe whether older `Converter` implementations (compiled with
> > > > > older versions of the Connect API without Converter extending
> > > > > Configurable) will be able to be installed into newer Connect
> > > > > runtimes, and what the behavior will be for the `GET
> > > > > /connector-plugins/<plugin>/config` method. (Although my suggestion
> > > > > for #5 will partially address the latter, we still need to talk
> about
> > > > > it in terms of compatibility.)
> > > > > 6. Can we add a small section addressing the security implications
> of
> > > > > these new APIs? KIP-507 [1] applied to APIs that existed at the
> time,
> > > > > and this KIP should at least mention how it aligns with that
> > > > > mechanism.
> > > > > 7. Do we really think adding `GET /worker-plugins` is necessary?
> While
> > > > > I understand the reasoning, I'm less convinced that it's worth
> adding
> > > > > to the public API. First, an administrator still has to modify the
> > > > > worker config for all workers to take advantage of that. Second,
> > > > > adding this method increases the surface area and therefore risk of
> > > > > additional attack vectors, despite not really being able to use the
> > > > > resulting information in any other part of the API.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Randall
> > > > >
> > > > > [1]
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
> > > > >
> > > > >
> > > > > On Wed, Dec 1, 2021 at 7:52 AM Mickael Maison <
> mickael.maison@gmail.com> wrote:
> > > > > >
> > > > > > Hi Chris,
> > > > > >
> > > > > > Thanks again for the feedback! I've updated the KIP based on our
> last
> > > > > > discussions. I've decided to include the new endpoint for worker
> > > > > > plugins.
> > > > > >
> > > > > > 1. Yes I agree, it's best to gate the new behavior.
> > > > > > 2. Yes, it was a remnant from the original proposal. I've now
> removed
> > > > > > the location field.
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > > On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
> > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > >
> > > > > > > Hi Mickael,
> > > > > > >
> > > > > > > I think that's a great idea! I especially like how we can
> establish the
> > > > > > > expectation that any plugin type that appears in the response
> from the GET
> > > > > > > /connector-plugins endpoint will have a corresponding GET
> > > > > > > /connector-plugins/<type>/config endpoint, but (if we decide
> to add them in
> > > > > > > the future), worker plugins won't be expected to expose this
> kind of
> > > > > > > information and the different root path helps give a decent
> hint about this.
> > > > > > >
> > > > > > > I also like the choice to return an empty ConfigDef from
> Converter::config
> > > > > > > instead of null.
> > > > > > >
> > > > > > > Two things come to mind:
> > > > > > >
> > > > > > > 1. We may want to gate this behind a URL query parameter
> (maybe something
> > > > > > > like "connectorsOnly") that defaults to the old behavior in
> order to avoid
> > > > > > > breaking existing tools such as programmatic UIs that use the
> endpoint
> > > > > > > today to discover the connectors that can be created by the
> user. We can
> > > > > > > even plan to change the default for that parameter to the
> newly-proposed
> > > > > > > behavior in the next major release, which should give people
> enough time to
> > > > > > > either adapt to the expanded response format or add the query
> parameter to
> > > > > > > their tooling.
> > > > > > >
> > > > > > > 2. The existing GET /connector-plugins endpoint doesn't
> contain information
> > > > > > > on the location of the plugin on the worker's file system. Do
> you think we
> > > > > > > should still include this info in the new response format?
> Correct me if
> > > > > > > I'm wrong but it seems it may have been proposed originally to
> help prevent
> > > > > > > already-addressed bugs in Connect classloading from striking;
> all else
> > > > > > > equal, I'd personally err on the side of leaving this info out
> or at least
> > > > > > > reducing permitted values for it to just "classpath" or
> "plugin path" in
> > > > > > > order to avoid leaking worker file paths into the REST API,
> which might
> > > > > > > bother super security-conscious users.
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <
> mickael.maison@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Chris,
> > > > > > > >
> > > > > > > > Yes to keep compatibility we want a default implementation
> for
> > > > > > > > Converter.configs(), I've updated the KIP.
> > > > > > > >
> > > > > > > > Regarding worker plugins, the use case you described seems
> valuable.
> > > > > > > > I'd prefer not mixing worker and connector plugins on the
> same
> > > > > > > > endpoint but I agree using /plugins and /worker-plugins
> could be
> > > > > > > > confusing.
> > > > > > > >
> > > > > > > > One alternative is to expose all connector-level plugins via
> the
> > > > > > > > existing /connector-plugins endpoint. In that case, we'd
> need to keep
> > > > > > > > the current JSON schema and not group plugins by type. As
> the current
> > > > > > > > schema already has a type field for each entry, we'll still
> be able to
> > > > > > > > tell them apart. Then we can have /worker-plugins and a
> relatively
> > > > > > > > clean API. What do you think?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mickael
> > > > > > > >
> > > > > > > > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > >
> > > > > > > > > Hi Mickael,
> > > > > > > > >
> > > > > > > > > I think one potential use case for exposing worker-level
> plugins is that
> > > > > > > > it
> > > > > > > > > may make it easier to determine whether a worker is set up
> correctly (the
> > > > > > > > > current alternative requires looking through log files and
> can be a
> > > > > > > > little
> > > > > > > > > tedious), and might even make it possible to automatically
> identify
> > > > > > > > > discrepancies within a cluster by diffing the contents of
> that endpoint
> > > > > > > > > across each worker. But I don't think this has to be
> addressed by the
> > > > > > > > > current KIP; the only thing that bothers me a little is
> that "plugins" is
> > > > > > > > > generic and it may confuse people down the road if we add
> an endpoint for
> > > > > > > > > worker-level plugins ("why is one just called 'plugins'
> and the other one
> > > > > > > > > is 'worker-plugins'?"). Probably not worth blocking on,
> though.
> > > > > > > > >
> > > > > > > > > Agreed that the suggestion for improved validation should
> be made on the
> > > > > > > > > KIP-802 thread.
> > > > > > > > >
> > > > > > > > > I also noticed that the newly-proposed config method for
> the Converter
> > > > > > > > > interface doesn't have a default implementation, making it
> > > > > > > > > backwards-incompatible. Should we add a default
> implementation that
> > > > > > > > returns
> > > > > > > > > either null or an empty ConfigDef?
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <
> mickael.maison@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Chris,
> > > > > > > > > >
> > > > > > > > > > 1. If we want to expose worker plugins, I think we
> should do it via a
> > > > > > > > > > separate endpoint. But to be honest, I'm not even sure I
> see strong
> > > > > > > > > > use cases for exposing them as they are either enabled
> or not and
> > > > > > > > > > can't be changed at runtime. So I'd prefer to stick to
> "connector
> > > > > > > > > > level" plugins in this KIP. Let me now if you have use
> cases, I'm open
> > > > > > > > > > to reconsider this choice.
> > > > > > > > > > I'll add that in the rejected alternatives section for
> now
> > > > > > > > > >
> > > > > > > > > > 2. I remembered seeing issues in the past with multiple
> plugin.path
> > > > > > > > > > entries but I tried today and I was able to mix and
> match plugins from
> > > > > > > > > > different paths. So my bad for getting confused.
> > > > > > > > > > Then I agree, it makes more sense to group them by
> plugin type.
> > > > > > > > > >
> > > > > > > > > > 3. Yes this should be covered in KIP-802:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > > > > > > > >
> > > > > > > > > > 4. No particular reason. We can support both formats
> like today. I've
> > > > > > > > > > updated the KIP
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Mickael
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Mickael,
> > > > > > > > > > >
> > > > > > > > > > > I think the increase in scope here is great and the
> added value
> > > > > > > > certainly
> > > > > > > > > > > justifies the proposed changes. I have some thoughts
> but overall I
> > > > > > > > like
> > > > > > > > > > the
> > > > > > > > > > > direction this is going in now.
> > > > > > > > > > >
> > > > > > > > > > > 1. The new /plugins endpoint is described as
> containing "all plugins
> > > > > > > > that
> > > > > > > > > > > are Connectors, Transformations, Converters,
> HeaderConverters and
> > > > > > > > > > > Predicates". So essentially, it looks like we want to
> expose all
> > > > > > > > plugins
> > > > > > > > > > > that are configured on a per-connector basis, but
> exclude plugins
> > > > > > > > that
> > > > > > > > > > are
> > > > > > > > > > > configured on a per-worker basis (such as config
> providers and REST
> > > > > > > > > > > extensions). Do you think it may be valuable to expose
> information on
> > > > > > > > > > > worker-level plugins as well?
> > > > > > > > > > >
> > > > > > > > > > > 2. The description for the new /plugins endpoint also
> states that
> > > > > > > > > > "Plugins
> > > > > > > > > > > will be grouped by plugin.path. This will make it
> clear to users
> > > > > > > > what's
> > > > > > > > > > > available to use as it's not possible to use a
> Connector from one
> > > > > > > > path
> > > > > > > > > > with
> > > > > > > > > > > Transformations from another.". Is this true? I
> thought that
> > > > > > > > Connect's
> > > > > > > > > > > classloading made it possible to package
> > > > > > > > > > > converters/transformations/predicates completely
> independently from
> > > > > > > > each
> > > > > > > > > > > other, and to reference them from
> also-independently-packaged
> > > > > > > > connectors.
> > > > > > > > > > > If it turns out that this is the case, could we
> consider
> > > > > > > > restructuring
> > > > > > > > > > the
> > > > > > > > > > > response to be grouped by plugin type instead of by
> classloader?
> > > > > > > > There's
> > > > > > > > > > > also the ungrouped format proposed in KIP-494 (
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > > > > > > > )
> > > > > > > > > > > which we might consider as well.
> > > > > > > > > > >
> > > > > > > > > > > 3. I think this can be left for a follow-up KIP if
> necessary, but I'm
> > > > > > > > > > > curious about your thoughts on adding new validate
> methods to all
> > > > > > > > > > > connector-level plugins that can be used similarly to
> how the
> > > > > > > > existing
> > > > > > > > > > > Connector::validate method (
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > > > > > > > )
> > > > > > > > > > > is used. This would allow for plugins to perform
> validation that's
> > > > > > > > more
> > > > > > > > > > > sophisticated than what the ConfigDef is capable of,
> such as
> > > > > > > > validating
> > > > > > > > > > > combinations of properties like a hostname and
> credentials for
> > > > > > > > reaching
> > > > > > > > > > it.
> > > > > > > > > > > I know that at least Confluent's Avro, protobuf, and
> JSON schema
> > > > > > > > > > converters
> > > > > > > > > > > would benefit from this kind of feature. It's a little
> tangential to
> > > > > > > > this
> > > > > > > > > > > KIP (which at the moment is about discovering plugins
> and their
> > > > > > > > > > > configuration surfaces, as opposed to validating
> them), but I
> > > > > > > > figured I'd
> > > > > > > > > > > ask since we're going to be expanding the Converter
> interface and it
> > > > > > > > may
> > > > > > > > > > be
> > > > > > > > > > > useful to tackle this while we're in the neighborhood.
> > > > > > > > > > >
> > > > > > > > > > > 4. The description for the new
> /plugins/<type>/<name>/configdef
> > > > > > > > endpoint
> > > > > > > > > > > states that "Name must be the fully qualified class
> name of the
> > > > > > > > plugin".
> > > > > > > > > > > Any reason not to also support aliases (e.g.,
> > > > > > > > "FileStreamSinkConnector"
> > > > > > > > > > or
> > > > > > > > > > > "FileStreamSink" instead of
> > > > > > > > > > >
> "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > Chris
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <
> > > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks all for the feedback!
> > > > > > > > > > > >
> > > > > > > > > > > > Chris,
> > > > > > > > > > > > I agree that fixing the current endpoint helps a
> lot. Thanks for
> > > > > > > > > > > > raising these JIRAs and submitting a PR!
> > > > > > > > > > > > However thinking about the issue further, I decided
> to expand the
> > > > > > > > > > > > scope of the KIP to cover all user-visible plugins.
> > > > > > > > > > > > In practice, users want to know about all available
> plugins not
> > > > > > > > only
> > > > > > > > > > > > connectors. This includes transformations,
> converters,
> > > > > > > > > > > > header_converters and predicates. As we also want to
> retrieve
> > > > > > > > > > > > configdef for these too, I think it makes sense to
> introduce a new
> > > > > > > > > > > > endpoint to do so. Alongside we obviously need a new
> endpoint for
> > > > > > > > > > > > listing all plugins.
> > > > > > > > > > > >
> > > > > > > > > > > > Gunnar,
> > > > > > > > > > > > I took a look at exposing valid values via the API.
> I think the
> > > > > > > > issue
> > > > > > > > > > > > is that Validators don't expose a way to retrieve
> valid values.
> > > > > > > > > > > > Changing validators will have an impact on all
> components so I'd
> > > > > > > > > > > > prefer to address this requirement in a separate
> KIP. I agree this
> > > > > > > > > > > > would be an interesting improvement and I'd happy to
> write a KIP
> > > > > > > > for
> > > > > > > > > > > > it too.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I have updated the KIP accordingly. Let me know if
> you have further
> > > > > > > > > > > > feedback.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Mickael
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > > > > > > > > > > > <gu...@googlemail.com.invalid> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm +1 for adding a GET endpoint for obtaining
> config
> > > > > > > > definitions. It
> > > > > > > > > > > > > always felt odd to me that one has to issue a PUT
> for that
> > > > > > > > purpose.
> > > > > > > > > > If
> > > > > > > > > > > > > nothing else, it'd be better in terms of
> discoverability of the
> > > > > > > > KC
> > > > > > > > > > REST
> > > > > > > > > > > > API.
> > > > > > > > > > > > >
> > > > > > > > > > > > > One additional feature request I'd have is to
> expose the valid
> > > > > > > > enum
> > > > > > > > > > > > > constants for enum-typed options. That'll help to
> display the
> > > > > > > > values
> > > > > > > > > > in a
> > > > > > > > > > > > > drop-down or via radio buttons in a UI, give us
> tab completion in
> > > > > > > > > > kcctl,
> > > > > > > > > > > > > etc.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > >
> > > > > > > > > > > > > --Gunnar
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris
> Egerton
> > > > > > > > > > > > > <ch...@confluent.io.invalid>:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Viktor,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It sounds like there are three major points here
> in favor of a
> > > > > > > > new
> > > > > > > > > > GET
> > > > > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. You cannot issue a blank ("dummy") request
> for sink
> > > > > > > > connectors
> > > > > > > > > > > > because a
> > > > > > > > > > > > > > topic list/topic regex has to be supplied
> (otherwise the PUT
> > > > > > > > > > endpoint
> > > > > > > > > > > > > > returns a 500 response)
> > > > > > > > > > > > > > 2. A dummy request still triggers custom
> validations by the
> > > > > > > > > > connector,
> > > > > > > > > > > > > > which may be best to avoid if we know for sure
> that the config
> > > > > > > > > > isn't
> > > > > > > > > > > > worth
> > > > > > > > > > > > > > validating yet
> > > > > > > > > > > > > > 3. It's more ergonomic and intuitive to be able
> to issue a GET
> > > > > > > > > > request
> > > > > > > > > > > > > > without having to give a dummy connector config
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > With regards to 1, this is actually a bug in
> Connect (
> > > > > > > > > > > > > >
> https://issues.apache.org/jira/browse/KAFKA-13327) with a fix
> > > > > > > > > > already
> > > > > > > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > > > > > > https://github.com/apache/kafka/pull/11369). I
> think it'd be
> > > > > > > > > > better to
> > > > > > > > > > > > > > focus on fixing this bug in general instead of
> implementing a
> > > > > > > > new
> > > > > > > > > > REST
> > > > > > > > > > > > > > endpoint in order to allow people to work around
> it.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > With regards to 2, this is technically possible
> but I'm unsure
> > > > > > > > > > it'd be
> > > > > > > > > > > > too
> > > > > > > > > > > > > > common out in the wild given that most
> validations that could
> > > > > > > > be
> > > > > > > > > > > > expensive
> > > > > > > > > > > > > > would involve things like connecting to a
> database, checking
> > > > > > > > if a
> > > > > > > > > > cloud
> > > > > > > > > > > > > > storage bucket exists, etc., none of which are
> possible without
> > > > > > > > > > some
> > > > > > > > > > > > > > configuration properties from the user (db
> hostname, bucket
> > > > > > > > name,
> > > > > > > > > > > > etc.).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > With regards to 3, I do agree that it'd be
> easier for people
> > > > > > > > > > designing
> > > > > > > > > > > > UIs
> > > > > > > > > > > > > > to have a GET API to work against. I'm just not
> sure it's
> > > > > > > > worth the
> > > > > > > > > > > > > > additional implementation, testing, and
> maintenance burden. If
> > > > > > > > it
> > > > > > > > > > were
> > > > > > > > > > > > > > possible to issue a PUT request without
> unexpected 500s for
> > > > > > > > invalid
> > > > > > > > > > > > > > configs, would that suffice? AFAICT it'd
> basically be as
> > > > > > > > simple as
> > > > > > > > > > > > issuing
> > > > > > > > > > > > > > a PUT request with a dummy body consisting of
> nothing except
> > > > > > > > the
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > class (which at this point we might even make
> unnecessary and
> > > > > > > > just
> > > > > > > > > > > > > > automatically replace with the connector class
> from the URL)
> > > > > > > > and
> > > > > > > > > > then
> > > > > > > > > > > > > > filtering the response to just grab the
> "definition" field of
> > > > > > > > each
> > > > > > > > > > > > element
> > > > > > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor
> Somogyi-Vass <
> > > > > > > > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Folks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I too think this would be a very useful
> feature. Some of our
> > > > > > > > > > > > management
> > > > > > > > > > > > > > > applications would provide a wizard for
> creating connectors.
> > > > > > > > In
> > > > > > > > > > this
> > > > > > > > > > > > > > > scenario the user basically would fill out a
> sample
> > > > > > > > configuration
> > > > > > > > > > > > > > generated
> > > > > > > > > > > > > > > by the UI which would send it back to Connect
> for validation
> > > > > > > > and
> > > > > > > > > > > > > > eventually
> > > > > > > > > > > > > > > create a new connector. The first part of this
> workflow can
> > > > > > > > be
> > > > > > > > > > > > enhanced
> > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > we had an API that can return the
> configuration definition
> > > > > > > > of the
> > > > > > > > > > > > given
> > > > > > > > > > > > > > > type of connector as the UI application would
> be able to
> > > > > > > > > > generate a
> > > > > > > > > > > > > > sample
> > > > > > > > > > > > > > > for the user based on that (nicely drawn
> diagram:
> > > > > > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > > > > > The
> connector-plugins/{connectorType}/config/validate API
> > > > > > > > > > essentially
> > > > > > > > > > > > > > works
> > > > > > > > > > > > > > > and returns the data that we need, however it
> is a HTTP PUT
> > > > > > > > API
> > > > > > > > > > that
> > > > > > > > > > > > is a
> > > > > > > > > > > > > > > bit unintuitive for a fetch-like functionality
> and also
> > > > > > > > > > functionally
> > > > > > > > > > > > > > > different as it validates the given (dummy)
> request. In case
> > > > > > > > of
> > > > > > > > > > sink
> > > > > > > > > > > > > > > connectors one would need to also provide a
> topic name.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > A suggestion for the KIP: I think it can be
> useful to return
> > > > > > > > the
> > > > > > > > > > > > config
> > > > > > > > > > > > > > > groups and the connector class' name similarly
> to the
> > > > > > > > validate
> > > > > > > > > > API
> > > > > > > > > > > > just
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > case any frontend needs them (and also the
> response would be
> > > > > > > > more
> > > > > > > > > > > > like
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > validate API but simpler).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Viktor
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > > > > > > ryannedolan@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think it'd be worth adding a GET version,
> fwiw. Could be
> > > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > > > > > > handler
> > > > > > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison
> <
> > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > You're right, you can achieve the same
> functionality
> > > > > > > > using
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > > > > > In my mind it was only for validation once
> you have
> > > > > > > > build a
> > > > > > > > > > > > > > > > > configuration but when used with an empty
> configuration,
> > > > > > > > it
> > > > > > > > > > > > basically
> > > > > > > > > > > > > > > > > serves the same purpose as the proposed
> new endpoint.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think it's a bit easier to use a GET
> endpoint but I
> > > > > > > > don't
> > > > > > > > > > > > think it
> > > > > > > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris
> Egerton
> > > > > > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I'm wondering about the use case here.
> The motivation
> > > > > > > > > > section
> > > > > > > > > > > > > > states
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > "Connect does not provide a way to see
> what
> > > > > > > > configurations
> > > > > > > > > > a
> > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > requires. Instead users have to go look
> at the
> > > > > > > > connector
> > > > > > > > > > > > > > > documentation
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > in the worst case, look directly at the
> connector
> > > > > > > > source
> > > > > > > > > > > > code.",
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > with this KIP, "users will be able to
> discover the
> > > > > > > > required
> > > > > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > > > > for connectors installed in a Connect
> cluster" and
> > > > > > > > "tools
> > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > generate wizards for configuring and
> starting
> > > > > > > > connectors".
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > > > > >
> /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > > > > > endpoint not address these points? What
> will the
> > > > > > > > > > newly-proposed
> > > > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > > allow users to do that they will not
> already be able
> > > > > > > > to do
> > > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael
> Maison <
> > > > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I've created KIP-769 to expose
> connector
> > > > > > > > configuration
> > > > > > > > > > > > > > definitions
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Please take a look and let me know if
> you have any
> > > > > > > > > > feedback.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
>

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Randall Hauch <rh...@gmail.com>]

Thanks, Mickael. Can we call out these as literals? For example:

> The response structure of the objects in the array remain unchanged, but this proposal does add "converter", "header_converter" and "transformation" to the allowed values for the "type" field, in addition to the existing "sink" and "source" values.

Best regards,

Randall

On Sat, Dec 4, 2021 at 4:58 AM Mickael Maison <mi...@gmail.com> wrote:
>
> Hi Randall,
>
> Sure. The types are sink, source, converter, header_converter,
> transformation and predicate. I've updated the KIP.
>
> Thanks,
> Mickael
>
> On Fri, Dec 3, 2021 at 10:14 PM Randall Hauch <rh...@gmail.com> wrote:
> >
> > One more thing, inline below:
> >
> > Randall
> >
> > On Fri, Dec 3, 2021 at 10:38 AM Mickael Maison <mi...@gmail.com> wrote:
> > >
> > > Thanks Randall for the feedback!
> > >
> > > 1. Done
> > >
> > > 2. Yes that makes sense. I've updated the KIP
> > >
> > > 3.a I've updated to explicitly mention the plugin can be provided
> > > using the fully qualified class or the alias.
> > > 3.b Only concrete classes will be returned. I made the mistake when I
> > > crafted the examples by hand. Fixed
> >
> > Can we also enumerate all of the values that can be included in the
> > "type" field in the response from the "GET
> > /connector-plugins?connectorsOnly=false"?
> >
> > >
> > > 4. If a plugin does not override config(), the response is an empty
> > > array. I've updated the KIP
> > >
> > > 5. I have updated the compatibility section with more details
> > >
> > > 6. Can you clarify what you mean? KIP-507 was for securing internal
> > > REST endpoints.
> > >
> > > 7. I don't think it is necessary. I think the use case Chris mentioned
> > > could be useful for admins. I'm interested in exposing the
> > > connector-level plugins and their configs. I thought it was easy to
> > > add an API for worker plugins while working in this area but if there
> > > isn't a strong consensus, I'm happy to drop it.
> > >
> > > Mickael
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch <rh...@gmail.com> wrote:
> > > >
> > > > Thanks for the KIP, Mickael. And thanks to everyone else for the good
> > > > discussion so far.
> > > >
> > > > Overall this is pretty good, but I do have a few questions/comments:
> > > > 1. Can we include the HTTP method/verb in each of the endpoints in the
> > > > "Public Interfaces" section?
> > > > 2. The sample output for `GET /connector-plugins?connectorsOnly=false`
> > > > includes a `version` field for the connector plugin, but not for the
> > > > transform or converter plugins. Neither the `Transformation` or
> > > > `Converter` interfaces have a version nor implement the `Versioned`
> > > > interface, so should we add the `Versioned` interface to the
> > > > `Transformation` and `Converter` interfaces and add a default method
> > > > that returns "unknown"? Yes, we could do that in a separate KIP, but I
> > > > think it's appropriate to do it in this KIP for a few reasons. First,
> > > > we're already modifying the `Converter` interface in this KIP and it
> > > > would be quite straightforward to also extend the `Versioned`
> > > > interface and add a default method; then doing that for
> > > > `Transformation` would also be relatively straightforward. Second,
> > > > this KIP provides the justification for adding a version to the
> > > > `Transformation` and `Converter` interfaces.
> > > > 3. In the "Public Interfaces" section, there are two inconsistencies
> > > > between the `GET /connector-plugins` and `GET
> > > > /connector-plugins/<plugin>/config` methods. Both will make it
> > > > difficult to use the second method, since the input to the second
> > > > method is not directly found in the response from the first method:
> > > >  a. The example for `GET /connector-plugins/Cast$Value/config` uses
> > > > short name, or what Connect often refers to as the alias, "Cast$Value"
> > > > for the plugin, rather than the fully-qualified name
> > > > "org.apache.kafka.connect.transforms.Cast$Value". The KIP should
> > > > specify whether it will support both the shortened name and the
> > > > fully-qualified name, or whether only one of these will be supported.
> > > > Not supporting the fully-qualified name will make it impossible to use
> > > > the response from the first method as an input to the second.
> > > >  a. The example for the list method returns the `Transformation` base
> > > > and abstract classes (e.g.,
> > > > "org.apache.kafka.connect.transforms.Cast") rather than the concrete
> > > > classes (e.g., "org.apache.kafka.connect.transforms.Cast$Value" and
> > > > "org.apache.kafka.connect.transforms.Cast$Key"), while the `GET
> > > > /connector-plugins/<plugin>/config` will require the concrete class.
> > > > 4. The "Public Interfaces" section's description of the `GET
> > > > /connector-plugins/<plugin>/config` method should probably specify the
> > > > expected behavior for `Converter` implementations that don't override
> > > > the new `config()` method.
> > > > 5. The "Compatibility, Deprecation, and Migration Plan" section should
> > > > describe whether older `Converter` implementations (compiled with
> > > > older versions of the Connect API without Converter extending
> > > > Configurable) will be able to be installed into newer Connect
> > > > runtimes, and what the behavior will be for the `GET
> > > > /connector-plugins/<plugin>/config` method. (Although my suggestion
> > > > for #5 will partially address the latter, we still need to talk about
> > > > it in terms of compatibility.)
> > > > 6. Can we add a small section addressing the security implications of
> > > > these new APIs? KIP-507 [1] applied to APIs that existed at the time,
> > > > and this KIP should at least mention how it aligns with that
> > > > mechanism.
> > > > 7. Do we really think adding `GET /worker-plugins` is necessary? While
> > > > I understand the reasoning, I'm less convinced that it's worth adding
> > > > to the public API. First, an administrator still has to modify the
> > > > worker config for all workers to take advantage of that. Second,
> > > > adding this method increases the surface area and therefore risk of
> > > > additional attack vectors, despite not really being able to use the
> > > > resulting information in any other part of the API.
> > > >
> > > > Best regards,
> > > >
> > > > Randall
> > > >
> > > > [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
> > > >
> > > >
> > > > On Wed, Dec 1, 2021 at 7:52 AM Mickael Maison <mi...@gmail.com> wrote:
> > > > >
> > > > > Hi Chris,
> > > > >
> > > > > Thanks again for the feedback! I've updated the KIP based on our last
> > > > > discussions. I've decided to include the new endpoint for worker
> > > > > plugins.
> > > > >
> > > > > 1. Yes I agree, it's best to gate the new behavior.
> > > > > 2. Yes, it was a remnant from the original proposal. I've now removed
> > > > > the location field.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
> > > > > <ch...@confluent.io.invalid> wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > I think that's a great idea! I especially like how we can establish the
> > > > > > expectation that any plugin type that appears in the response from the GET
> > > > > > /connector-plugins endpoint will have a corresponding GET
> > > > > > /connector-plugins/<type>/config endpoint, but (if we decide to add them in
> > > > > > the future), worker plugins won't be expected to expose this kind of
> > > > > > information and the different root path helps give a decent hint about this.
> > > > > >
> > > > > > I also like the choice to return an empty ConfigDef from Converter::config
> > > > > > instead of null.
> > > > > >
> > > > > > Two things come to mind:
> > > > > >
> > > > > > 1. We may want to gate this behind a URL query parameter (maybe something
> > > > > > like "connectorsOnly") that defaults to the old behavior in order to avoid
> > > > > > breaking existing tools such as programmatic UIs that use the endpoint
> > > > > > today to discover the connectors that can be created by the user. We can
> > > > > > even plan to change the default for that parameter to the newly-proposed
> > > > > > behavior in the next major release, which should give people enough time to
> > > > > > either adapt to the expanded response format or add the query parameter to
> > > > > > their tooling.
> > > > > >
> > > > > > 2. The existing GET /connector-plugins endpoint doesn't contain information
> > > > > > on the location of the plugin on the worker's file system. Do you think we
> > > > > > should still include this info in the new response format? Correct me if
> > > > > > I'm wrong but it seems it may have been proposed originally to help prevent
> > > > > > already-addressed bugs in Connect classloading from striking; all else
> > > > > > equal, I'd personally err on the side of leaving this info out or at least
> > > > > > reducing permitted values for it to just "classpath" or "plugin path" in
> > > > > > order to avoid leaking worker file paths into the REST API, which might
> > > > > > bother super security-conscious users.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <mi...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Chris,
> > > > > > >
> > > > > > > Yes to keep compatibility we want a default implementation for
> > > > > > > Converter.configs(), I've updated the KIP.
> > > > > > >
> > > > > > > Regarding worker plugins, the use case you described seems valuable.
> > > > > > > I'd prefer not mixing worker and connector plugins on the same
> > > > > > > endpoint but I agree using /plugins and /worker-plugins could be
> > > > > > > confusing.
> > > > > > >
> > > > > > > One alternative is to expose all connector-level plugins via the
> > > > > > > existing /connector-plugins endpoint. In that case, we'd need to keep
> > > > > > > the current JSON schema and not group plugins by type. As the current
> > > > > > > schema already has a type field for each entry, we'll still be able to
> > > > > > > tell them apart. Then we can have /worker-plugins and a relatively
> > > > > > > clean API. What do you think?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mickael
> > > > > > >
> > > > > > > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > >
> > > > > > > > Hi Mickael,
> > > > > > > >
> > > > > > > > I think one potential use case for exposing worker-level plugins is that
> > > > > > > it
> > > > > > > > may make it easier to determine whether a worker is set up correctly (the
> > > > > > > > current alternative requires looking through log files and can be a
> > > > > > > little
> > > > > > > > tedious), and might even make it possible to automatically identify
> > > > > > > > discrepancies within a cluster by diffing the contents of that endpoint
> > > > > > > > across each worker. But I don't think this has to be addressed by the
> > > > > > > > current KIP; the only thing that bothers me a little is that "plugins" is
> > > > > > > > generic and it may confuse people down the road if we add an endpoint for
> > > > > > > > worker-level plugins ("why is one just called 'plugins' and the other one
> > > > > > > > is 'worker-plugins'?"). Probably not worth blocking on, though.
> > > > > > > >
> > > > > > > > Agreed that the suggestion for improved validation should be made on the
> > > > > > > > KIP-802 thread.
> > > > > > > >
> > > > > > > > I also noticed that the newly-proposed config method for the Converter
> > > > > > > > interface doesn't have a default implementation, making it
> > > > > > > > backwards-incompatible. Should we add a default implementation that
> > > > > > > returns
> > > > > > > > either null or an empty ConfigDef?
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <mickael.maison@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Chris,
> > > > > > > > >
> > > > > > > > > 1. If we want to expose worker plugins, I think we should do it via a
> > > > > > > > > separate endpoint. But to be honest, I'm not even sure I see strong
> > > > > > > > > use cases for exposing them as they are either enabled or not and
> > > > > > > > > can't be changed at runtime. So I'd prefer to stick to "connector
> > > > > > > > > level" plugins in this KIP. Let me now if you have use cases, I'm open
> > > > > > > > > to reconsider this choice.
> > > > > > > > > I'll add that in the rejected alternatives section for now
> > > > > > > > >
> > > > > > > > > 2. I remembered seeing issues in the past with multiple plugin.path
> > > > > > > > > entries but I tried today and I was able to mix and match plugins from
> > > > > > > > > different paths. So my bad for getting confused.
> > > > > > > > > Then I agree, it makes more sense to group them by plugin type.
> > > > > > > > >
> > > > > > > > > 3. Yes this should be covered in KIP-802:
> > > > > > > > >
> > > > > > > > >
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > > > > > > >
> > > > > > > > > 4. No particular reason. We can support both formats like today. I've
> > > > > > > > > updated the KIP
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Mickael
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Mickael,
> > > > > > > > > >
> > > > > > > > > > I think the increase in scope here is great and the added value
> > > > > > > certainly
> > > > > > > > > > justifies the proposed changes. I have some thoughts but overall I
> > > > > > > like
> > > > > > > > > the
> > > > > > > > > > direction this is going in now.
> > > > > > > > > >
> > > > > > > > > > 1. The new /plugins endpoint is described as containing "all plugins
> > > > > > > that
> > > > > > > > > > are Connectors, Transformations, Converters, HeaderConverters and
> > > > > > > > > > Predicates". So essentially, it looks like we want to expose all
> > > > > > > plugins
> > > > > > > > > > that are configured on a per-connector basis, but exclude plugins
> > > > > > > that
> > > > > > > > > are
> > > > > > > > > > configured on a per-worker basis (such as config providers and REST
> > > > > > > > > > extensions). Do you think it may be valuable to expose information on
> > > > > > > > > > worker-level plugins as well?
> > > > > > > > > >
> > > > > > > > > > 2. The description for the new /plugins endpoint also states that
> > > > > > > > > "Plugins
> > > > > > > > > > will be grouped by plugin.path. This will make it clear to users
> > > > > > > what's
> > > > > > > > > > available to use as it's not possible to use a Connector from one
> > > > > > > path
> > > > > > > > > with
> > > > > > > > > > Transformations from another.". Is this true? I thought that
> > > > > > > Connect's
> > > > > > > > > > classloading made it possible to package
> > > > > > > > > > converters/transformations/predicates completely independently from
> > > > > > > each
> > > > > > > > > > other, and to reference them from also-independently-packaged
> > > > > > > connectors.
> > > > > > > > > > If it turns out that this is the case, could we consider
> > > > > > > restructuring
> > > > > > > > > the
> > > > > > > > > > response to be grouped by plugin type instead of by classloader?
> > > > > > > There's
> > > > > > > > > > also the ungrouped format proposed in KIP-494 (
> > > > > > > > > >
> > > > > > > > >
> > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > > > > > > )
> > > > > > > > > > which we might consider as well.
> > > > > > > > > >
> > > > > > > > > > 3. I think this can be left for a follow-up KIP if necessary, but I'm
> > > > > > > > > > curious about your thoughts on adding new validate methods to all
> > > > > > > > > > connector-level plugins that can be used similarly to how the
> > > > > > > existing
> > > > > > > > > > Connector::validate method (
> > > > > > > > > >
> > > > > > > > >
> > > > > > > https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > > > > > > )
> > > > > > > > > > is used. This would allow for plugins to perform validation that's
> > > > > > > more
> > > > > > > > > > sophisticated than what the ConfigDef is capable of, such as
> > > > > > > validating
> > > > > > > > > > combinations of properties like a hostname and credentials for
> > > > > > > reaching
> > > > > > > > > it.
> > > > > > > > > > I know that at least Confluent's Avro, protobuf, and JSON schema
> > > > > > > > > converters
> > > > > > > > > > would benefit from this kind of feature. It's a little tangential to
> > > > > > > this
> > > > > > > > > > KIP (which at the moment is about discovering plugins and their
> > > > > > > > > > configuration surfaces, as opposed to validating them), but I
> > > > > > > figured I'd
> > > > > > > > > > ask since we're going to be expanding the Converter interface and it
> > > > > > > may
> > > > > > > > > be
> > > > > > > > > > useful to tackle this while we're in the neighborhood.
> > > > > > > > > >
> > > > > > > > > > 4. The description for the new /plugins/<type>/<name>/configdef
> > > > > > > endpoint
> > > > > > > > > > states that "Name must be the fully qualified class name of the
> > > > > > > plugin".
> > > > > > > > > > Any reason not to also support aliases (e.g.,
> > > > > > > "FileStreamSinkConnector"
> > > > > > > > > or
> > > > > > > > > > "FileStreamSink" instead of
> > > > > > > > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > >
> > > > > > > > > > Chris
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <
> > > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks all for the feedback!
> > > > > > > > > > >
> > > > > > > > > > > Chris,
> > > > > > > > > > > I agree that fixing the current endpoint helps a lot. Thanks for
> > > > > > > > > > > raising these JIRAs and submitting a PR!
> > > > > > > > > > > However thinking about the issue further, I decided to expand the
> > > > > > > > > > > scope of the KIP to cover all user-visible plugins.
> > > > > > > > > > > In practice, users want to know about all available plugins not
> > > > > > > only
> > > > > > > > > > > connectors. This includes transformations, converters,
> > > > > > > > > > > header_converters and predicates. As we also want to retrieve
> > > > > > > > > > > configdef for these too, I think it makes sense to introduce a new
> > > > > > > > > > > endpoint to do so. Alongside we obviously need a new endpoint for
> > > > > > > > > > > listing all plugins.
> > > > > > > > > > >
> > > > > > > > > > > Gunnar,
> > > > > > > > > > > I took a look at exposing valid values via the API. I think the
> > > > > > > issue
> > > > > > > > > > > is that Validators don't expose a way to retrieve valid values.
> > > > > > > > > > > Changing validators will have an impact on all components so I'd
> > > > > > > > > > > prefer to address this requirement in a separate KIP. I agree this
> > > > > > > > > > > would be an interesting improvement and I'd happy to write a KIP
> > > > > > > for
> > > > > > > > > > > it too.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I have updated the KIP accordingly. Let me know if you have further
> > > > > > > > > > > feedback.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Mickael
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > > > > > > > > > > <gu...@googlemail.com.invalid> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > I'm +1 for adding a GET endpoint for obtaining config
> > > > > > > definitions. It
> > > > > > > > > > > > always felt odd to me that one has to issue a PUT for that
> > > > > > > purpose.
> > > > > > > > > If
> > > > > > > > > > > > nothing else, it'd be better in terms of discoverability of the
> > > > > > > KC
> > > > > > > > > REST
> > > > > > > > > > > API.
> > > > > > > > > > > >
> > > > > > > > > > > > One additional feature request I'd have is to expose the valid
> > > > > > > enum
> > > > > > > > > > > > constants for enum-typed options. That'll help to display the
> > > > > > > values
> > > > > > > > > in a
> > > > > > > > > > > > drop-down or via radio buttons in a UI, give us tab completion in
> > > > > > > > > kcctl,
> > > > > > > > > > > > etc.
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > >
> > > > > > > > > > > > --Gunnar
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > > > > > > > > > > > <ch...@confluent.io.invalid>:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Viktor,
> > > > > > > > > > > > >
> > > > > > > > > > > > > It sounds like there are three major points here in favor of a
> > > > > > > new
> > > > > > > > > GET
> > > > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. You cannot issue a blank ("dummy") request for sink
> > > > > > > connectors
> > > > > > > > > > > because a
> > > > > > > > > > > > > topic list/topic regex has to be supplied (otherwise the PUT
> > > > > > > > > endpoint
> > > > > > > > > > > > > returns a 500 response)
> > > > > > > > > > > > > 2. A dummy request still triggers custom validations by the
> > > > > > > > > connector,
> > > > > > > > > > > > > which may be best to avoid if we know for sure that the config
> > > > > > > > > isn't
> > > > > > > > > > > worth
> > > > > > > > > > > > > validating yet
> > > > > > > > > > > > > 3. It's more ergonomic and intuitive to be able to issue a GET
> > > > > > > > > request
> > > > > > > > > > > > > without having to give a dummy connector config
> > > > > > > > > > > > >
> > > > > > > > > > > > > With regards to 1, this is actually a bug in Connect (
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix
> > > > > > > > > already
> > > > > > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > > > > > https://github.com/apache/kafka/pull/11369). I think it'd be
> > > > > > > > > better to
> > > > > > > > > > > > > focus on fixing this bug in general instead of implementing a
> > > > > > > new
> > > > > > > > > REST
> > > > > > > > > > > > > endpoint in order to allow people to work around it.
> > > > > > > > > > > > >
> > > > > > > > > > > > > With regards to 2, this is technically possible but I'm unsure
> > > > > > > > > it'd be
> > > > > > > > > > > too
> > > > > > > > > > > > > common out in the wild given that most validations that could
> > > > > > > be
> > > > > > > > > > > expensive
> > > > > > > > > > > > > would involve things like connecting to a database, checking
> > > > > > > if a
> > > > > > > > > cloud
> > > > > > > > > > > > > storage bucket exists, etc., none of which are possible without
> > > > > > > > > some
> > > > > > > > > > > > > configuration properties from the user (db hostname, bucket
> > > > > > > name,
> > > > > > > > > > > etc.).
> > > > > > > > > > > > >
> > > > > > > > > > > > > With regards to 3, I do agree that it'd be easier for people
> > > > > > > > > designing
> > > > > > > > > > > UIs
> > > > > > > > > > > > > to have a GET API to work against. I'm just not sure it's
> > > > > > > worth the
> > > > > > > > > > > > > additional implementation, testing, and maintenance burden. If
> > > > > > > it
> > > > > > > > > were
> > > > > > > > > > > > > possible to issue a PUT request without unexpected 500s for
> > > > > > > invalid
> > > > > > > > > > > > > configs, would that suffice? AFAICT it'd basically be as
> > > > > > > simple as
> > > > > > > > > > > issuing
> > > > > > > > > > > > > a PUT request with a dummy body consisting of nothing except
> > > > > > > the
> > > > > > > > > > > connector
> > > > > > > > > > > > > class (which at this point we might even make unnecessary and
> > > > > > > just
> > > > > > > > > > > > > automatically replace with the connector class from the URL)
> > > > > > > and
> > > > > > > > > then
> > > > > > > > > > > > > filtering the response to just grab the "definition" field of
> > > > > > > each
> > > > > > > > > > > element
> > > > > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Chris
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Folks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I too think this would be a very useful feature. Some of our
> > > > > > > > > > > management
> > > > > > > > > > > > > > applications would provide a wizard for creating connectors.
> > > > > > > In
> > > > > > > > > this
> > > > > > > > > > > > > > scenario the user basically would fill out a sample
> > > > > > > configuration
> > > > > > > > > > > > > generated
> > > > > > > > > > > > > > by the UI which would send it back to Connect for validation
> > > > > > > and
> > > > > > > > > > > > > eventually
> > > > > > > > > > > > > > create a new connector. The first part of this workflow can
> > > > > > > be
> > > > > > > > > > > enhanced
> > > > > > > > > > > > > if
> > > > > > > > > > > > > > we had an API that can return the configuration definition
> > > > > > > of the
> > > > > > > > > > > given
> > > > > > > > > > > > > > type of connector as the UI application would be able to
> > > > > > > > > generate a
> > > > > > > > > > > > > sample
> > > > > > > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > > > > The connector-plugins/{connectorType}/config/validate API
> > > > > > > > > essentially
> > > > > > > > > > > > > works
> > > > > > > > > > > > > > and returns the data that we need, however it is a HTTP PUT
> > > > > > > API
> > > > > > > > > that
> > > > > > > > > > > is a
> > > > > > > > > > > > > > bit unintuitive for a fetch-like functionality and also
> > > > > > > > > functionally
> > > > > > > > > > > > > > different as it validates the given (dummy) request. In case
> > > > > > > of
> > > > > > > > > sink
> > > > > > > > > > > > > > connectors one would need to also provide a topic name.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > A suggestion for the KIP: I think it can be useful to return
> > > > > > > the
> > > > > > > > > > > config
> > > > > > > > > > > > > > groups and the connector class' name similarly to the
> > > > > > > validate
> > > > > > > > > API
> > > > > > > > > > > just
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > case any frontend needs them (and also the response would be
> > > > > > > more
> > > > > > > > > > > like
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > validate API but simpler).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Viktor
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > > > > > ryannedolan@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think it'd be worth adding a GET version, fwiw. Could be
> > > > > > > the
> > > > > > > > > same
> > > > > > > > > > > > > > handler
> > > > > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > You're right, you can achieve the same functionality
> > > > > > > using
> > > > > > > > > the
> > > > > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > > > > In my mind it was only for validation once you have
> > > > > > > build a
> > > > > > > > > > > > > > > > configuration but when used with an empty configuration,
> > > > > > > it
> > > > > > > > > > > basically
> > > > > > > > > > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think it's a bit easier to use a GET endpoint but I
> > > > > > > don't
> > > > > > > > > > > think it
> > > > > > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I'm wondering about the use case here. The motivation
> > > > > > > > > section
> > > > > > > > > > > > > states
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > "Connect does not provide a way to see what
> > > > > > > configurations
> > > > > > > > > a
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > requires. Instead users have to go look at the
> > > > > > > connector
> > > > > > > > > > > > > > documentation
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > in the worst case, look directly at the connector
> > > > > > > source
> > > > > > > > > > > code.",
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > with this KIP, "users will be able to discover the
> > > > > > > required
> > > > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > > > for connectors installed in a Connect cluster" and
> > > > > > > "tools
> > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > > > able
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > generate wizards for configuring and starting
> > > > > > > connectors".
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > > > > endpoint not address these points? What will the
> > > > > > > > > newly-proposed
> > > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > allow users to do that they will not already be able
> > > > > > > to do
> > > > > > > > > > > with the
> > > > > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I've created KIP-769 to expose connector
> > > > > > > configuration
> > > > > > > > > > > > > definitions
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Please take a look and let me know if you have any
> > > > > > > > > feedback.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Mickael Maison <mi...@gmail.com>]

Hi Randall,

Sure. The types are sink, source, converter, header_converter,
transformation and predicate. I've updated the KIP.

Thanks,
Mickael

On Fri, Dec 3, 2021 at 10:14 PM Randall Hauch <rh...@gmail.com> wrote:
>
> One more thing, inline below:
>
> Randall
>
> On Fri, Dec 3, 2021 at 10:38 AM Mickael Maison <mi...@gmail.com> wrote:
> >
> > Thanks Randall for the feedback!
> >
> > 1. Done
> >
> > 2. Yes that makes sense. I've updated the KIP
> >
> > 3.a I've updated to explicitly mention the plugin can be provided
> > using the fully qualified class or the alias.
> > 3.b Only concrete classes will be returned. I made the mistake when I
> > crafted the examples by hand. Fixed
>
> Can we also enumerate all of the values that can be included in the
> "type" field in the response from the "GET
> /connector-plugins?connectorsOnly=false"?
>
> >
> > 4. If a plugin does not override config(), the response is an empty
> > array. I've updated the KIP
> >
> > 5. I have updated the compatibility section with more details
> >
> > 6. Can you clarify what you mean? KIP-507 was for securing internal
> > REST endpoints.
> >
> > 7. I don't think it is necessary. I think the use case Chris mentioned
> > could be useful for admins. I'm interested in exposing the
> > connector-level plugins and their configs. I thought it was easy to
> > add an API for worker plugins while working in this area but if there
> > isn't a strong consensus, I'm happy to drop it.
> >
> > Mickael
> >
> >
> >
> >
> >
> >
> > On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch <rh...@gmail.com> wrote:
> > >
> > > Thanks for the KIP, Mickael. And thanks to everyone else for the good
> > > discussion so far.
> > >
> > > Overall this is pretty good, but I do have a few questions/comments:
> > > 1. Can we include the HTTP method/verb in each of the endpoints in the
> > > "Public Interfaces" section?
> > > 2. The sample output for `GET /connector-plugins?connectorsOnly=false`
> > > includes a `version` field for the connector plugin, but not for the
> > > transform or converter plugins. Neither the `Transformation` or
> > > `Converter` interfaces have a version nor implement the `Versioned`
> > > interface, so should we add the `Versioned` interface to the
> > > `Transformation` and `Converter` interfaces and add a default method
> > > that returns "unknown"? Yes, we could do that in a separate KIP, but I
> > > think it's appropriate to do it in this KIP for a few reasons. First,
> > > we're already modifying the `Converter` interface in this KIP and it
> > > would be quite straightforward to also extend the `Versioned`
> > > interface and add a default method; then doing that for
> > > `Transformation` would also be relatively straightforward. Second,
> > > this KIP provides the justification for adding a version to the
> > > `Transformation` and `Converter` interfaces.
> > > 3. In the "Public Interfaces" section, there are two inconsistencies
> > > between the `GET /connector-plugins` and `GET
> > > /connector-plugins/<plugin>/config` methods. Both will make it
> > > difficult to use the second method, since the input to the second
> > > method is not directly found in the response from the first method:
> > >  a. The example for `GET /connector-plugins/Cast$Value/config` uses
> > > short name, or what Connect often refers to as the alias, "Cast$Value"
> > > for the plugin, rather than the fully-qualified name
> > > "org.apache.kafka.connect.transforms.Cast$Value". The KIP should
> > > specify whether it will support both the shortened name and the
> > > fully-qualified name, or whether only one of these will be supported.
> > > Not supporting the fully-qualified name will make it impossible to use
> > > the response from the first method as an input to the second.
> > >  a. The example for the list method returns the `Transformation` base
> > > and abstract classes (e.g.,
> > > "org.apache.kafka.connect.transforms.Cast") rather than the concrete
> > > classes (e.g., "org.apache.kafka.connect.transforms.Cast$Value" and
> > > "org.apache.kafka.connect.transforms.Cast$Key"), while the `GET
> > > /connector-plugins/<plugin>/config` will require the concrete class.
> > > 4. The "Public Interfaces" section's description of the `GET
> > > /connector-plugins/<plugin>/config` method should probably specify the
> > > expected behavior for `Converter` implementations that don't override
> > > the new `config()` method.
> > > 5. The "Compatibility, Deprecation, and Migration Plan" section should
> > > describe whether older `Converter` implementations (compiled with
> > > older versions of the Connect API without Converter extending
> > > Configurable) will be able to be installed into newer Connect
> > > runtimes, and what the behavior will be for the `GET
> > > /connector-plugins/<plugin>/config` method. (Although my suggestion
> > > for #5 will partially address the latter, we still need to talk about
> > > it in terms of compatibility.)
> > > 6. Can we add a small section addressing the security implications of
> > > these new APIs? KIP-507 [1] applied to APIs that existed at the time,
> > > and this KIP should at least mention how it aligns with that
> > > mechanism.
> > > 7. Do we really think adding `GET /worker-plugins` is necessary? While
> > > I understand the reasoning, I'm less convinced that it's worth adding
> > > to the public API. First, an administrator still has to modify the
> > > worker config for all workers to take advantage of that. Second,
> > > adding this method increases the surface area and therefore risk of
> > > additional attack vectors, despite not really being able to use the
> > > resulting information in any other part of the API.
> > >
> > > Best regards,
> > >
> > > Randall
> > >
> > > [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
> > >
> > >
> > > On Wed, Dec 1, 2021 at 7:52 AM Mickael Maison <mi...@gmail.com> wrote:
> > > >
> > > > Hi Chris,
> > > >
> > > > Thanks again for the feedback! I've updated the KIP based on our last
> > > > discussions. I've decided to include the new endpoint for worker
> > > > plugins.
> > > >
> > > > 1. Yes I agree, it's best to gate the new behavior.
> > > > 2. Yes, it was a remnant from the original proposal. I've now removed
> > > > the location field.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
> > > > <ch...@confluent.io.invalid> wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > I think that's a great idea! I especially like how we can establish the
> > > > > expectation that any plugin type that appears in the response from the GET
> > > > > /connector-plugins endpoint will have a corresponding GET
> > > > > /connector-plugins/<type>/config endpoint, but (if we decide to add them in
> > > > > the future), worker plugins won't be expected to expose this kind of
> > > > > information and the different root path helps give a decent hint about this.
> > > > >
> > > > > I also like the choice to return an empty ConfigDef from Converter::config
> > > > > instead of null.
> > > > >
> > > > > Two things come to mind:
> > > > >
> > > > > 1. We may want to gate this behind a URL query parameter (maybe something
> > > > > like "connectorsOnly") that defaults to the old behavior in order to avoid
> > > > > breaking existing tools such as programmatic UIs that use the endpoint
> > > > > today to discover the connectors that can be created by the user. We can
> > > > > even plan to change the default for that parameter to the newly-proposed
> > > > > behavior in the next major release, which should give people enough time to
> > > > > either adapt to the expanded response format or add the query parameter to
> > > > > their tooling.
> > > > >
> > > > > 2. The existing GET /connector-plugins endpoint doesn't contain information
> > > > > on the location of the plugin on the worker's file system. Do you think we
> > > > > should still include this info in the new response format? Correct me if
> > > > > I'm wrong but it seems it may have been proposed originally to help prevent
> > > > > already-addressed bugs in Connect classloading from striking; all else
> > > > > equal, I'd personally err on the side of leaving this info out or at least
> > > > > reducing permitted values for it to just "classpath" or "plugin path" in
> > > > > order to avoid leaking worker file paths into the REST API, which might
> > > > > bother super security-conscious users.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <mi...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Chris,
> > > > > >
> > > > > > Yes to keep compatibility we want a default implementation for
> > > > > > Converter.configs(), I've updated the KIP.
> > > > > >
> > > > > > Regarding worker plugins, the use case you described seems valuable.
> > > > > > I'd prefer not mixing worker and connector plugins on the same
> > > > > > endpoint but I agree using /plugins and /worker-plugins could be
> > > > > > confusing.
> > > > > >
> > > > > > One alternative is to expose all connector-level plugins via the
> > > > > > existing /connector-plugins endpoint. In that case, we'd need to keep
> > > > > > the current JSON schema and not group plugins by type. As the current
> > > > > > schema already has a type field for each entry, we'll still be able to
> > > > > > tell them apart. Then we can have /worker-plugins and a relatively
> > > > > > clean API. What do you think?
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > >
> > > > > > > Hi Mickael,
> > > > > > >
> > > > > > > I think one potential use case for exposing worker-level plugins is that
> > > > > > it
> > > > > > > may make it easier to determine whether a worker is set up correctly (the
> > > > > > > current alternative requires looking through log files and can be a
> > > > > > little
> > > > > > > tedious), and might even make it possible to automatically identify
> > > > > > > discrepancies within a cluster by diffing the contents of that endpoint
> > > > > > > across each worker. But I don't think this has to be addressed by the
> > > > > > > current KIP; the only thing that bothers me a little is that "plugins" is
> > > > > > > generic and it may confuse people down the road if we add an endpoint for
> > > > > > > worker-level plugins ("why is one just called 'plugins' and the other one
> > > > > > > is 'worker-plugins'?"). Probably not worth blocking on, though.
> > > > > > >
> > > > > > > Agreed that the suggestion for improved validation should be made on the
> > > > > > > KIP-802 thread.
> > > > > > >
> > > > > > > I also noticed that the newly-proposed config method for the Converter
> > > > > > > interface doesn't have a default implementation, making it
> > > > > > > backwards-incompatible. Should we add a default implementation that
> > > > > > returns
> > > > > > > either null or an empty ConfigDef?
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <mickael.maison@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Chris,
> > > > > > > >
> > > > > > > > 1. If we want to expose worker plugins, I think we should do it via a
> > > > > > > > separate endpoint. But to be honest, I'm not even sure I see strong
> > > > > > > > use cases for exposing them as they are either enabled or not and
> > > > > > > > can't be changed at runtime. So I'd prefer to stick to "connector
> > > > > > > > level" plugins in this KIP. Let me now if you have use cases, I'm open
> > > > > > > > to reconsider this choice.
> > > > > > > > I'll add that in the rejected alternatives section for now
> > > > > > > >
> > > > > > > > 2. I remembered seeing issues in the past with multiple plugin.path
> > > > > > > > entries but I tried today and I was able to mix and match plugins from
> > > > > > > > different paths. So my bad for getting confused.
> > > > > > > > Then I agree, it makes more sense to group them by plugin type.
> > > > > > > >
> > > > > > > > 3. Yes this should be covered in KIP-802:
> > > > > > > >
> > > > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > > > > > >
> > > > > > > > 4. No particular reason. We can support both formats like today. I've
> > > > > > > > updated the KIP
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mickael
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > >
> > > > > > > > > Hi Mickael,
> > > > > > > > >
> > > > > > > > > I think the increase in scope here is great and the added value
> > > > > > certainly
> > > > > > > > > justifies the proposed changes. I have some thoughts but overall I
> > > > > > like
> > > > > > > > the
> > > > > > > > > direction this is going in now.
> > > > > > > > >
> > > > > > > > > 1. The new /plugins endpoint is described as containing "all plugins
> > > > > > that
> > > > > > > > > are Connectors, Transformations, Converters, HeaderConverters and
> > > > > > > > > Predicates". So essentially, it looks like we want to expose all
> > > > > > plugins
> > > > > > > > > that are configured on a per-connector basis, but exclude plugins
> > > > > > that
> > > > > > > > are
> > > > > > > > > configured on a per-worker basis (such as config providers and REST
> > > > > > > > > extensions). Do you think it may be valuable to expose information on
> > > > > > > > > worker-level plugins as well?
> > > > > > > > >
> > > > > > > > > 2. The description for the new /plugins endpoint also states that
> > > > > > > > "Plugins
> > > > > > > > > will be grouped by plugin.path. This will make it clear to users
> > > > > > what's
> > > > > > > > > available to use as it's not possible to use a Connector from one
> > > > > > path
> > > > > > > > with
> > > > > > > > > Transformations from another.". Is this true? I thought that
> > > > > > Connect's
> > > > > > > > > classloading made it possible to package
> > > > > > > > > converters/transformations/predicates completely independently from
> > > > > > each
> > > > > > > > > other, and to reference them from also-independently-packaged
> > > > > > connectors.
> > > > > > > > > If it turns out that this is the case, could we consider
> > > > > > restructuring
> > > > > > > > the
> > > > > > > > > response to be grouped by plugin type instead of by classloader?
> > > > > > There's
> > > > > > > > > also the ungrouped format proposed in KIP-494 (
> > > > > > > > >
> > > > > > > >
> > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > > > > > )
> > > > > > > > > which we might consider as well.
> > > > > > > > >
> > > > > > > > > 3. I think this can be left for a follow-up KIP if necessary, but I'm
> > > > > > > > > curious about your thoughts on adding new validate methods to all
> > > > > > > > > connector-level plugins that can be used similarly to how the
> > > > > > existing
> > > > > > > > > Connector::validate method (
> > > > > > > > >
> > > > > > > >
> > > > > > https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > > > > > )
> > > > > > > > > is used. This would allow for plugins to perform validation that's
> > > > > > more
> > > > > > > > > sophisticated than what the ConfigDef is capable of, such as
> > > > > > validating
> > > > > > > > > combinations of properties like a hostname and credentials for
> > > > > > reaching
> > > > > > > > it.
> > > > > > > > > I know that at least Confluent's Avro, protobuf, and JSON schema
> > > > > > > > converters
> > > > > > > > > would benefit from this kind of feature. It's a little tangential to
> > > > > > this
> > > > > > > > > KIP (which at the moment is about discovering plugins and their
> > > > > > > > > configuration surfaces, as opposed to validating them), but I
> > > > > > figured I'd
> > > > > > > > > ask since we're going to be expanding the Converter interface and it
> > > > > > may
> > > > > > > > be
> > > > > > > > > useful to tackle this while we're in the neighborhood.
> > > > > > > > >
> > > > > > > > > 4. The description for the new /plugins/<type>/<name>/configdef
> > > > > > endpoint
> > > > > > > > > states that "Name must be the fully qualified class name of the
> > > > > > plugin".
> > > > > > > > > Any reason not to also support aliases (e.g.,
> > > > > > "FileStreamSinkConnector"
> > > > > > > > or
> > > > > > > > > "FileStreamSink" instead of
> > > > > > > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <
> > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks all for the feedback!
> > > > > > > > > >
> > > > > > > > > > Chris,
> > > > > > > > > > I agree that fixing the current endpoint helps a lot. Thanks for
> > > > > > > > > > raising these JIRAs and submitting a PR!
> > > > > > > > > > However thinking about the issue further, I decided to expand the
> > > > > > > > > > scope of the KIP to cover all user-visible plugins.
> > > > > > > > > > In practice, users want to know about all available plugins not
> > > > > > only
> > > > > > > > > > connectors. This includes transformations, converters,
> > > > > > > > > > header_converters and predicates. As we also want to retrieve
> > > > > > > > > > configdef for these too, I think it makes sense to introduce a new
> > > > > > > > > > endpoint to do so. Alongside we obviously need a new endpoint for
> > > > > > > > > > listing all plugins.
> > > > > > > > > >
> > > > > > > > > > Gunnar,
> > > > > > > > > > I took a look at exposing valid values via the API. I think the
> > > > > > issue
> > > > > > > > > > is that Validators don't expose a way to retrieve valid values.
> > > > > > > > > > Changing validators will have an impact on all components so I'd
> > > > > > > > > > prefer to address this requirement in a separate KIP. I agree this
> > > > > > > > > > would be an interesting improvement and I'd happy to write a KIP
> > > > > > for
> > > > > > > > > > it too.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I have updated the KIP accordingly. Let me know if you have further
> > > > > > > > > > feedback.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Mickael
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > > > > > > > > > <gu...@googlemail.com.invalid> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > I'm +1 for adding a GET endpoint for obtaining config
> > > > > > definitions. It
> > > > > > > > > > > always felt odd to me that one has to issue a PUT for that
> > > > > > purpose.
> > > > > > > > If
> > > > > > > > > > > nothing else, it'd be better in terms of discoverability of the
> > > > > > KC
> > > > > > > > REST
> > > > > > > > > > API.
> > > > > > > > > > >
> > > > > > > > > > > One additional feature request I'd have is to expose the valid
> > > > > > enum
> > > > > > > > > > > constants for enum-typed options. That'll help to display the
> > > > > > values
> > > > > > > > in a
> > > > > > > > > > > drop-down or via radio buttons in a UI, give us tab completion in
> > > > > > > > kcctl,
> > > > > > > > > > > etc.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > >
> > > > > > > > > > > --Gunnar
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > > > > > > > > > > <ch...@confluent.io.invalid>:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Viktor,
> > > > > > > > > > > >
> > > > > > > > > > > > It sounds like there are three major points here in favor of a
> > > > > > new
> > > > > > > > GET
> > > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > > >
> > > > > > > > > > > > 1. You cannot issue a blank ("dummy") request for sink
> > > > > > connectors
> > > > > > > > > > because a
> > > > > > > > > > > > topic list/topic regex has to be supplied (otherwise the PUT
> > > > > > > > endpoint
> > > > > > > > > > > > returns a 500 response)
> > > > > > > > > > > > 2. A dummy request still triggers custom validations by the
> > > > > > > > connector,
> > > > > > > > > > > > which may be best to avoid if we know for sure that the config
> > > > > > > > isn't
> > > > > > > > > > worth
> > > > > > > > > > > > validating yet
> > > > > > > > > > > > 3. It's more ergonomic and intuitive to be able to issue a GET
> > > > > > > > request
> > > > > > > > > > > > without having to give a dummy connector config
> > > > > > > > > > > >
> > > > > > > > > > > > With regards to 1, this is actually a bug in Connect (
> > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix
> > > > > > > > already
> > > > > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > > > > https://github.com/apache/kafka/pull/11369). I think it'd be
> > > > > > > > better to
> > > > > > > > > > > > focus on fixing this bug in general instead of implementing a
> > > > > > new
> > > > > > > > REST
> > > > > > > > > > > > endpoint in order to allow people to work around it.
> > > > > > > > > > > >
> > > > > > > > > > > > With regards to 2, this is technically possible but I'm unsure
> > > > > > > > it'd be
> > > > > > > > > > too
> > > > > > > > > > > > common out in the wild given that most validations that could
> > > > > > be
> > > > > > > > > > expensive
> > > > > > > > > > > > would involve things like connecting to a database, checking
> > > > > > if a
> > > > > > > > cloud
> > > > > > > > > > > > storage bucket exists, etc., none of which are possible without
> > > > > > > > some
> > > > > > > > > > > > configuration properties from the user (db hostname, bucket
> > > > > > name,
> > > > > > > > > > etc.).
> > > > > > > > > > > >
> > > > > > > > > > > > With regards to 3, I do agree that it'd be easier for people
> > > > > > > > designing
> > > > > > > > > > UIs
> > > > > > > > > > > > to have a GET API to work against. I'm just not sure it's
> > > > > > worth the
> > > > > > > > > > > > additional implementation, testing, and maintenance burden. If
> > > > > > it
> > > > > > > > were
> > > > > > > > > > > > possible to issue a PUT request without unexpected 500s for
> > > > > > invalid
> > > > > > > > > > > > configs, would that suffice? AFAICT it'd basically be as
> > > > > > simple as
> > > > > > > > > > issuing
> > > > > > > > > > > > a PUT request with a dummy body consisting of nothing except
> > > > > > the
> > > > > > > > > > connector
> > > > > > > > > > > > class (which at this point we might even make unnecessary and
> > > > > > just
> > > > > > > > > > > > automatically replace with the connector class from the URL)
> > > > > > and
> > > > > > > > then
> > > > > > > > > > > > filtering the response to just grab the "definition" field of
> > > > > > each
> > > > > > > > > > element
> > > > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > >
> > > > > > > > > > > > Chris
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Folks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I too think this would be a very useful feature. Some of our
> > > > > > > > > > management
> > > > > > > > > > > > > applications would provide a wizard for creating connectors.
> > > > > > In
> > > > > > > > this
> > > > > > > > > > > > > scenario the user basically would fill out a sample
> > > > > > configuration
> > > > > > > > > > > > generated
> > > > > > > > > > > > > by the UI which would send it back to Connect for validation
> > > > > > and
> > > > > > > > > > > > eventually
> > > > > > > > > > > > > create a new connector. The first part of this workflow can
> > > > > > be
> > > > > > > > > > enhanced
> > > > > > > > > > > > if
> > > > > > > > > > > > > we had an API that can return the configuration definition
> > > > > > of the
> > > > > > > > > > given
> > > > > > > > > > > > > type of connector as the UI application would be able to
> > > > > > > > generate a
> > > > > > > > > > > > sample
> > > > > > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > > > The connector-plugins/{connectorType}/config/validate API
> > > > > > > > essentially
> > > > > > > > > > > > works
> > > > > > > > > > > > > and returns the data that we need, however it is a HTTP PUT
> > > > > > API
> > > > > > > > that
> > > > > > > > > > is a
> > > > > > > > > > > > > bit unintuitive for a fetch-like functionality and also
> > > > > > > > functionally
> > > > > > > > > > > > > different as it validates the given (dummy) request. In case
> > > > > > of
> > > > > > > > sink
> > > > > > > > > > > > > connectors one would need to also provide a topic name.
> > > > > > > > > > > > >
> > > > > > > > > > > > > A suggestion for the KIP: I think it can be useful to return
> > > > > > the
> > > > > > > > > > config
> > > > > > > > > > > > > groups and the connector class' name similarly to the
> > > > > > validate
> > > > > > > > API
> > > > > > > > > > just
> > > > > > > > > > > > in
> > > > > > > > > > > > > case any frontend needs them (and also the response would be
> > > > > > more
> > > > > > > > > > like
> > > > > > > > > > > > the
> > > > > > > > > > > > > validate API but simpler).
> > > > > > > > > > > > >
> > > > > > > > > > > > > Viktor
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > > > > ryannedolan@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I think it'd be worth adding a GET version, fwiw. Could be
> > > > > > the
> > > > > > > > same
> > > > > > > > > > > > > handler
> > > > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > You're right, you can achieve the same functionality
> > > > > > using
> > > > > > > > the
> > > > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > > > In my mind it was only for validation once you have
> > > > > > build a
> > > > > > > > > > > > > > > configuration but when used with an empty configuration,
> > > > > > it
> > > > > > > > > > basically
> > > > > > > > > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think it's a bit easier to use a GET endpoint but I
> > > > > > don't
> > > > > > > > > > think it
> > > > > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'm wondering about the use case here. The motivation
> > > > > > > > section
> > > > > > > > > > > > states
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > "Connect does not provide a way to see what
> > > > > > configurations
> > > > > > > > a
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > requires. Instead users have to go look at the
> > > > > > connector
> > > > > > > > > > > > > documentation
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > in the worst case, look directly at the connector
> > > > > > source
> > > > > > > > > > code.",
> > > > > > > > > > > > and
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > with this KIP, "users will be able to discover the
> > > > > > required
> > > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > > for connectors installed in a Connect cluster" and
> > > > > > "tools
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > > able
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > generate wizards for configuring and starting
> > > > > > connectors".
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > > > endpoint not address these points? What will the
> > > > > > > > newly-proposed
> > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > allow users to do that they will not already be able
> > > > > > to do
> > > > > > > > > > with the
> > > > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I've created KIP-769 to expose connector
> > > > > > configuration
> > > > > > > > > > > > definitions
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Please take a look and let me know if you have any
> > > > > > > > feedback.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Randall Hauch <rh...@gmail.com>]

One more thing, inline below:

Randall

On Fri, Dec 3, 2021 at 10:38 AM Mickael Maison <mi...@gmail.com> wrote:
>
> Thanks Randall for the feedback!
>
> 1. Done
>
> 2. Yes that makes sense. I've updated the KIP
>
> 3.a I've updated to explicitly mention the plugin can be provided
> using the fully qualified class or the alias.
> 3.b Only concrete classes will be returned. I made the mistake when I
> crafted the examples by hand. Fixed

Can we also enumerate all of the values that can be included in the
"type" field in the response from the "GET
/connector-plugins?connectorsOnly=false"?

>
> 4. If a plugin does not override config(), the response is an empty
> array. I've updated the KIP
>
> 5. I have updated the compatibility section with more details
>
> 6. Can you clarify what you mean? KIP-507 was for securing internal
> REST endpoints.
>
> 7. I don't think it is necessary. I think the use case Chris mentioned
> could be useful for admins. I'm interested in exposing the
> connector-level plugins and their configs. I thought it was easy to
> add an API for worker plugins while working in this area but if there
> isn't a strong consensus, I'm happy to drop it.
>
> Mickael
>
>
>
>
>
>
> On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch <rh...@gmail.com> wrote:
> >
> > Thanks for the KIP, Mickael. And thanks to everyone else for the good
> > discussion so far.
> >
> > Overall this is pretty good, but I do have a few questions/comments:
> > 1. Can we include the HTTP method/verb in each of the endpoints in the
> > "Public Interfaces" section?
> > 2. The sample output for `GET /connector-plugins?connectorsOnly=false`
> > includes a `version` field for the connector plugin, but not for the
> > transform or converter plugins. Neither the `Transformation` or
> > `Converter` interfaces have a version nor implement the `Versioned`
> > interface, so should we add the `Versioned` interface to the
> > `Transformation` and `Converter` interfaces and add a default method
> > that returns "unknown"? Yes, we could do that in a separate KIP, but I
> > think it's appropriate to do it in this KIP for a few reasons. First,
> > we're already modifying the `Converter` interface in this KIP and it
> > would be quite straightforward to also extend the `Versioned`
> > interface and add a default method; then doing that for
> > `Transformation` would also be relatively straightforward. Second,
> > this KIP provides the justification for adding a version to the
> > `Transformation` and `Converter` interfaces.
> > 3. In the "Public Interfaces" section, there are two inconsistencies
> > between the `GET /connector-plugins` and `GET
> > /connector-plugins/<plugin>/config` methods. Both will make it
> > difficult to use the second method, since the input to the second
> > method is not directly found in the response from the first method:
> >  a. The example for `GET /connector-plugins/Cast$Value/config` uses
> > short name, or what Connect often refers to as the alias, "Cast$Value"
> > for the plugin, rather than the fully-qualified name
> > "org.apache.kafka.connect.transforms.Cast$Value". The KIP should
> > specify whether it will support both the shortened name and the
> > fully-qualified name, or whether only one of these will be supported.
> > Not supporting the fully-qualified name will make it impossible to use
> > the response from the first method as an input to the second.
> >  a. The example for the list method returns the `Transformation` base
> > and abstract classes (e.g.,
> > "org.apache.kafka.connect.transforms.Cast") rather than the concrete
> > classes (e.g., "org.apache.kafka.connect.transforms.Cast$Value" and
> > "org.apache.kafka.connect.transforms.Cast$Key"), while the `GET
> > /connector-plugins/<plugin>/config` will require the concrete class.
> > 4. The "Public Interfaces" section's description of the `GET
> > /connector-plugins/<plugin>/config` method should probably specify the
> > expected behavior for `Converter` implementations that don't override
> > the new `config()` method.
> > 5. The "Compatibility, Deprecation, and Migration Plan" section should
> > describe whether older `Converter` implementations (compiled with
> > older versions of the Connect API without Converter extending
> > Configurable) will be able to be installed into newer Connect
> > runtimes, and what the behavior will be for the `GET
> > /connector-plugins/<plugin>/config` method. (Although my suggestion
> > for #5 will partially address the latter, we still need to talk about
> > it in terms of compatibility.)
> > 6. Can we add a small section addressing the security implications of
> > these new APIs? KIP-507 [1] applied to APIs that existed at the time,
> > and this KIP should at least mention how it aligns with that
> > mechanism.
> > 7. Do we really think adding `GET /worker-plugins` is necessary? While
> > I understand the reasoning, I'm less convinced that it's worth adding
> > to the public API. First, an administrator still has to modify the
> > worker config for all workers to take advantage of that. Second,
> > adding this method increases the surface area and therefore risk of
> > additional attack vectors, despite not really being able to use the
> > resulting information in any other part of the API.
> >
> > Best regards,
> >
> > Randall
> >
> > [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
> >
> >
> > On Wed, Dec 1, 2021 at 7:52 AM Mickael Maison <mi...@gmail.com> wrote:
> > >
> > > Hi Chris,
> > >
> > > Thanks again for the feedback! I've updated the KIP based on our last
> > > discussions. I've decided to include the new endpoint for worker
> > > plugins.
> > >
> > > 1. Yes I agree, it's best to gate the new behavior.
> > > 2. Yes, it was a remnant from the original proposal. I've now removed
> > > the location field.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
> > > <ch...@confluent.io.invalid> wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > I think that's a great idea! I especially like how we can establish the
> > > > expectation that any plugin type that appears in the response from the GET
> > > > /connector-plugins endpoint will have a corresponding GET
> > > > /connector-plugins/<type>/config endpoint, but (if we decide to add them in
> > > > the future), worker plugins won't be expected to expose this kind of
> > > > information and the different root path helps give a decent hint about this.
> > > >
> > > > I also like the choice to return an empty ConfigDef from Converter::config
> > > > instead of null.
> > > >
> > > > Two things come to mind:
> > > >
> > > > 1. We may want to gate this behind a URL query parameter (maybe something
> > > > like "connectorsOnly") that defaults to the old behavior in order to avoid
> > > > breaking existing tools such as programmatic UIs that use the endpoint
> > > > today to discover the connectors that can be created by the user. We can
> > > > even plan to change the default for that parameter to the newly-proposed
> > > > behavior in the next major release, which should give people enough time to
> > > > either adapt to the expanded response format or add the query parameter to
> > > > their tooling.
> > > >
> > > > 2. The existing GET /connector-plugins endpoint doesn't contain information
> > > > on the location of the plugin on the worker's file system. Do you think we
> > > > should still include this info in the new response format? Correct me if
> > > > I'm wrong but it seems it may have been proposed originally to help prevent
> > > > already-addressed bugs in Connect classloading from striking; all else
> > > > equal, I'd personally err on the side of leaving this info out or at least
> > > > reducing permitted values for it to just "classpath" or "plugin path" in
> > > > order to avoid leaking worker file paths into the REST API, which might
> > > > bother super security-conscious users.
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <mi...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Chris,
> > > > >
> > > > > Yes to keep compatibility we want a default implementation for
> > > > > Converter.configs(), I've updated the KIP.
> > > > >
> > > > > Regarding worker plugins, the use case you described seems valuable.
> > > > > I'd prefer not mixing worker and connector plugins on the same
> > > > > endpoint but I agree using /plugins and /worker-plugins could be
> > > > > confusing.
> > > > >
> > > > > One alternative is to expose all connector-level plugins via the
> > > > > existing /connector-plugins endpoint. In that case, we'd need to keep
> > > > > the current JSON schema and not group plugins by type. As the current
> > > > > schema already has a type field for each entry, we'll still be able to
> > > > > tell them apart. Then we can have /worker-plugins and a relatively
> > > > > clean API. What do you think?
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > > > > <ch...@confluent.io.invalid> wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > I think one potential use case for exposing worker-level plugins is that
> > > > > it
> > > > > > may make it easier to determine whether a worker is set up correctly (the
> > > > > > current alternative requires looking through log files and can be a
> > > > > little
> > > > > > tedious), and might even make it possible to automatically identify
> > > > > > discrepancies within a cluster by diffing the contents of that endpoint
> > > > > > across each worker. But I don't think this has to be addressed by the
> > > > > > current KIP; the only thing that bothers me a little is that "plugins" is
> > > > > > generic and it may confuse people down the road if we add an endpoint for
> > > > > > worker-level plugins ("why is one just called 'plugins' and the other one
> > > > > > is 'worker-plugins'?"). Probably not worth blocking on, though.
> > > > > >
> > > > > > Agreed that the suggestion for improved validation should be made on the
> > > > > > KIP-802 thread.
> > > > > >
> > > > > > I also noticed that the newly-proposed config method for the Converter
> > > > > > interface doesn't have a default implementation, making it
> > > > > > backwards-incompatible. Should we add a default implementation that
> > > > > returns
> > > > > > either null or an empty ConfigDef?
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <mickael.maison@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Chris,
> > > > > > >
> > > > > > > 1. If we want to expose worker plugins, I think we should do it via a
> > > > > > > separate endpoint. But to be honest, I'm not even sure I see strong
> > > > > > > use cases for exposing them as they are either enabled or not and
> > > > > > > can't be changed at runtime. So I'd prefer to stick to "connector
> > > > > > > level" plugins in this KIP. Let me now if you have use cases, I'm open
> > > > > > > to reconsider this choice.
> > > > > > > I'll add that in the rejected alternatives section for now
> > > > > > >
> > > > > > > 2. I remembered seeing issues in the past with multiple plugin.path
> > > > > > > entries but I tried today and I was able to mix and match plugins from
> > > > > > > different paths. So my bad for getting confused.
> > > > > > > Then I agree, it makes more sense to group them by plugin type.
> > > > > > >
> > > > > > > 3. Yes this should be covered in KIP-802:
> > > > > > >
> > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > > > > >
> > > > > > > 4. No particular reason. We can support both formats like today. I've
> > > > > > > updated the KIP
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mickael
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > >
> > > > > > > > Hi Mickael,
> > > > > > > >
> > > > > > > > I think the increase in scope here is great and the added value
> > > > > certainly
> > > > > > > > justifies the proposed changes. I have some thoughts but overall I
> > > > > like
> > > > > > > the
> > > > > > > > direction this is going in now.
> > > > > > > >
> > > > > > > > 1. The new /plugins endpoint is described as containing "all plugins
> > > > > that
> > > > > > > > are Connectors, Transformations, Converters, HeaderConverters and
> > > > > > > > Predicates". So essentially, it looks like we want to expose all
> > > > > plugins
> > > > > > > > that are configured on a per-connector basis, but exclude plugins
> > > > > that
> > > > > > > are
> > > > > > > > configured on a per-worker basis (such as config providers and REST
> > > > > > > > extensions). Do you think it may be valuable to expose information on
> > > > > > > > worker-level plugins as well?
> > > > > > > >
> > > > > > > > 2. The description for the new /plugins endpoint also states that
> > > > > > > "Plugins
> > > > > > > > will be grouped by plugin.path. This will make it clear to users
> > > > > what's
> > > > > > > > available to use as it's not possible to use a Connector from one
> > > > > path
> > > > > > > with
> > > > > > > > Transformations from another.". Is this true? I thought that
> > > > > Connect's
> > > > > > > > classloading made it possible to package
> > > > > > > > converters/transformations/predicates completely independently from
> > > > > each
> > > > > > > > other, and to reference them from also-independently-packaged
> > > > > connectors.
> > > > > > > > If it turns out that this is the case, could we consider
> > > > > restructuring
> > > > > > > the
> > > > > > > > response to be grouped by plugin type instead of by classloader?
> > > > > There's
> > > > > > > > also the ungrouped format proposed in KIP-494 (
> > > > > > > >
> > > > > > >
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > > > > )
> > > > > > > > which we might consider as well.
> > > > > > > >
> > > > > > > > 3. I think this can be left for a follow-up KIP if necessary, but I'm
> > > > > > > > curious about your thoughts on adding new validate methods to all
> > > > > > > > connector-level plugins that can be used similarly to how the
> > > > > existing
> > > > > > > > Connector::validate method (
> > > > > > > >
> > > > > > >
> > > > > https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > > > > )
> > > > > > > > is used. This would allow for plugins to perform validation that's
> > > > > more
> > > > > > > > sophisticated than what the ConfigDef is capable of, such as
> > > > > validating
> > > > > > > > combinations of properties like a hostname and credentials for
> > > > > reaching
> > > > > > > it.
> > > > > > > > I know that at least Confluent's Avro, protobuf, and JSON schema
> > > > > > > converters
> > > > > > > > would benefit from this kind of feature. It's a little tangential to
> > > > > this
> > > > > > > > KIP (which at the moment is about discovering plugins and their
> > > > > > > > configuration surfaces, as opposed to validating them), but I
> > > > > figured I'd
> > > > > > > > ask since we're going to be expanding the Converter interface and it
> > > > > may
> > > > > > > be
> > > > > > > > useful to tackle this while we're in the neighborhood.
> > > > > > > >
> > > > > > > > 4. The description for the new /plugins/<type>/<name>/configdef
> > > > > endpoint
> > > > > > > > states that "Name must be the fully qualified class name of the
> > > > > plugin".
> > > > > > > > Any reason not to also support aliases (e.g.,
> > > > > "FileStreamSinkConnector"
> > > > > > > or
> > > > > > > > "FileStreamSink" instead of
> > > > > > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <
> > > > > > > mickael.maison@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks all for the feedback!
> > > > > > > > >
> > > > > > > > > Chris,
> > > > > > > > > I agree that fixing the current endpoint helps a lot. Thanks for
> > > > > > > > > raising these JIRAs and submitting a PR!
> > > > > > > > > However thinking about the issue further, I decided to expand the
> > > > > > > > > scope of the KIP to cover all user-visible plugins.
> > > > > > > > > In practice, users want to know about all available plugins not
> > > > > only
> > > > > > > > > connectors. This includes transformations, converters,
> > > > > > > > > header_converters and predicates. As we also want to retrieve
> > > > > > > > > configdef for these too, I think it makes sense to introduce a new
> > > > > > > > > endpoint to do so. Alongside we obviously need a new endpoint for
> > > > > > > > > listing all plugins.
> > > > > > > > >
> > > > > > > > > Gunnar,
> > > > > > > > > I took a look at exposing valid values via the API. I think the
> > > > > issue
> > > > > > > > > is that Validators don't expose a way to retrieve valid values.
> > > > > > > > > Changing validators will have an impact on all components so I'd
> > > > > > > > > prefer to address this requirement in a separate KIP. I agree this
> > > > > > > > > would be an interesting improvement and I'd happy to write a KIP
> > > > > for
> > > > > > > > > it too.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I have updated the KIP accordingly. Let me know if you have further
> > > > > > > > > feedback.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Mickael
> > > > > > > > >
> > > > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > > > > > > > > <gu...@googlemail.com.invalid> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I'm +1 for adding a GET endpoint for obtaining config
> > > > > definitions. It
> > > > > > > > > > always felt odd to me that one has to issue a PUT for that
> > > > > purpose.
> > > > > > > If
> > > > > > > > > > nothing else, it'd be better in terms of discoverability of the
> > > > > KC
> > > > > > > REST
> > > > > > > > > API.
> > > > > > > > > >
> > > > > > > > > > One additional feature request I'd have is to expose the valid
> > > > > enum
> > > > > > > > > > constants for enum-typed options. That'll help to display the
> > > > > values
> > > > > > > in a
> > > > > > > > > > drop-down or via radio buttons in a UI, give us tab completion in
> > > > > > > kcctl,
> > > > > > > > > > etc.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > >
> > > > > > > > > > --Gunnar
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > > > > > > > > > <ch...@confluent.io.invalid>:
> > > > > > > > > >
> > > > > > > > > > > Hi Viktor,
> > > > > > > > > > >
> > > > > > > > > > > It sounds like there are three major points here in favor of a
> > > > > new
> > > > > > > GET
> > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > >
> > > > > > > > > > > 1. You cannot issue a blank ("dummy") request for sink
> > > > > connectors
> > > > > > > > > because a
> > > > > > > > > > > topic list/topic regex has to be supplied (otherwise the PUT
> > > > > > > endpoint
> > > > > > > > > > > returns a 500 response)
> > > > > > > > > > > 2. A dummy request still triggers custom validations by the
> > > > > > > connector,
> > > > > > > > > > > which may be best to avoid if we know for sure that the config
> > > > > > > isn't
> > > > > > > > > worth
> > > > > > > > > > > validating yet
> > > > > > > > > > > 3. It's more ergonomic and intuitive to be able to issue a GET
> > > > > > > request
> > > > > > > > > > > without having to give a dummy connector config
> > > > > > > > > > >
> > > > > > > > > > > With regards to 1, this is actually a bug in Connect (
> > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix
> > > > > > > already
> > > > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > > > https://github.com/apache/kafka/pull/11369). I think it'd be
> > > > > > > better to
> > > > > > > > > > > focus on fixing this bug in general instead of implementing a
> > > > > new
> > > > > > > REST
> > > > > > > > > > > endpoint in order to allow people to work around it.
> > > > > > > > > > >
> > > > > > > > > > > With regards to 2, this is technically possible but I'm unsure
> > > > > > > it'd be
> > > > > > > > > too
> > > > > > > > > > > common out in the wild given that most validations that could
> > > > > be
> > > > > > > > > expensive
> > > > > > > > > > > would involve things like connecting to a database, checking
> > > > > if a
> > > > > > > cloud
> > > > > > > > > > > storage bucket exists, etc., none of which are possible without
> > > > > > > some
> > > > > > > > > > > configuration properties from the user (db hostname, bucket
> > > > > name,
> > > > > > > > > etc.).
> > > > > > > > > > >
> > > > > > > > > > > With regards to 3, I do agree that it'd be easier for people
> > > > > > > designing
> > > > > > > > > UIs
> > > > > > > > > > > to have a GET API to work against. I'm just not sure it's
> > > > > worth the
> > > > > > > > > > > additional implementation, testing, and maintenance burden. If
> > > > > it
> > > > > > > were
> > > > > > > > > > > possible to issue a PUT request without unexpected 500s for
> > > > > invalid
> > > > > > > > > > > configs, would that suffice? AFAICT it'd basically be as
> > > > > simple as
> > > > > > > > > issuing
> > > > > > > > > > > a PUT request with a dummy body consisting of nothing except
> > > > > the
> > > > > > > > > connector
> > > > > > > > > > > class (which at this point we might even make unnecessary and
> > > > > just
> > > > > > > > > > > automatically replace with the connector class from the URL)
> > > > > and
> > > > > > > then
> > > > > > > > > > > filtering the response to just grab the "definition" field of
> > > > > each
> > > > > > > > > element
> > > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > Chris
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Folks,
> > > > > > > > > > > >
> > > > > > > > > > > > I too think this would be a very useful feature. Some of our
> > > > > > > > > management
> > > > > > > > > > > > applications would provide a wizard for creating connectors.
> > > > > In
> > > > > > > this
> > > > > > > > > > > > scenario the user basically would fill out a sample
> > > > > configuration
> > > > > > > > > > > generated
> > > > > > > > > > > > by the UI which would send it back to Connect for validation
> > > > > and
> > > > > > > > > > > eventually
> > > > > > > > > > > > create a new connector. The first part of this workflow can
> > > > > be
> > > > > > > > > enhanced
> > > > > > > > > > > if
> > > > > > > > > > > > we had an API that can return the configuration definition
> > > > > of the
> > > > > > > > > given
> > > > > > > > > > > > type of connector as the UI application would be able to
> > > > > > > generate a
> > > > > > > > > > > sample
> > > > > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > > The connector-plugins/{connectorType}/config/validate API
> > > > > > > essentially
> > > > > > > > > > > works
> > > > > > > > > > > > and returns the data that we need, however it is a HTTP PUT
> > > > > API
> > > > > > > that
> > > > > > > > > is a
> > > > > > > > > > > > bit unintuitive for a fetch-like functionality and also
> > > > > > > functionally
> > > > > > > > > > > > different as it validates the given (dummy) request. In case
> > > > > of
> > > > > > > sink
> > > > > > > > > > > > connectors one would need to also provide a topic name.
> > > > > > > > > > > >
> > > > > > > > > > > > A suggestion for the KIP: I think it can be useful to return
> > > > > the
> > > > > > > > > config
> > > > > > > > > > > > groups and the connector class' name similarly to the
> > > > > validate
> > > > > > > API
> > > > > > > > > just
> > > > > > > > > > > in
> > > > > > > > > > > > case any frontend needs them (and also the response would be
> > > > > more
> > > > > > > > > like
> > > > > > > > > > > the
> > > > > > > > > > > > validate API but simpler).
> > > > > > > > > > > >
> > > > > > > > > > > > Viktor
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > > > ryannedolan@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I think it'd be worth adding a GET version, fwiw. Could be
> > > > > the
> > > > > > > same
> > > > > > > > > > > > handler
> > > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > You're right, you can achieve the same functionality
> > > > > using
> > > > > > > the
> > > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > > In my mind it was only for validation once you have
> > > > > build a
> > > > > > > > > > > > > > configuration but when used with an empty configuration,
> > > > > it
> > > > > > > > > basically
> > > > > > > > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think it's a bit easier to use a GET endpoint but I
> > > > > don't
> > > > > > > > > think it
> > > > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm wondering about the use case here. The motivation
> > > > > > > section
> > > > > > > > > > > states
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > "Connect does not provide a way to see what
> > > > > configurations
> > > > > > > a
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > > requires. Instead users have to go look at the
> > > > > connector
> > > > > > > > > > > > documentation
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > in the worst case, look directly at the connector
> > > > > source
> > > > > > > > > code.",
> > > > > > > > > > > and
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > with this KIP, "users will be able to discover the
> > > > > required
> > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > for connectors installed in a Connect cluster" and
> > > > > "tools
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > able
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > generate wizards for configuring and starting
> > > > > connectors".
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > > endpoint not address these points? What will the
> > > > > > > newly-proposed
> > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > allow users to do that they will not already be able
> > > > > to do
> > > > > > > > > with the
> > > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I've created KIP-769 to expose connector
> > > > > configuration
> > > > > > > > > > > definitions
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Please take a look and let me know if you have any
> > > > > > > feedback.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Randall Hauch <rh...@gmail.com>]

Thanks, Chris and Mickael.

The use case does make sense, and I can see some value in it. We do
have precedent for some administrative endpoints, including KIP-495
[1] that adds the resource `admin/loggers`. Since the use case persona
is a Connect cluster administrator, perhaps we could move this
`/worker-plugins` endpoint under this `/admin/` path to help
distinguish this and to centralize at least these endpoints in case
they need to be secured either by the admin (e.g., via a proxy or
gateway) or as a future KIP that makes them configurable.

I'm still not sure, though. I accept that the effort to add the
worker-plugins resource is not significant. My main concern is about
the additional surface area and whether the stated use case is
_valuable enough_ to warrant this additional surface area. The
connector, converter, and SMT plugins are all designed to be used via
the API by clients, so exposing these available plugins makes sense.
But the worker plugins are only usable via a worker configuration
change, and exposing the available worker plugins may provide more
information via API to malicious users that wouldn't otherwise without
direct access to the machines.

Best regards,

Randall

[1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-495%3A+Dynamically+Adjust+Log+Levels+in+Connect

On Fri, Dec 3, 2021 at 10:52 AM Chris Egerton
<ch...@confluent.io.invalid> wrote:
>
> Hi Mickael and Randall,
>
> I think there's a decent use case for exposing worker plugins. If a cluster
> administrator wants to enable a worker plugin on their cluster, it may make
> things easier if they can verify ahead of time that that plugin is
> available on every worker in the cluster (and possibly that the same
> version is present on every worker). The current alternative to this would
> require sifting through worker logs.
>
> I don't think the additional workload to expose worker plugins is
> significant (it should largely mirror the same logic we'll use to expose
> connector plugins), but if there's serious objection to including that
> feature in this KIP, we could probably live without it now that we've made
> sure that the current proposal doesn't block us from adding it in a
> reasonable way later on down the road.
>
> Cheers,
>
> Chris
>
> On Fri, Dec 3, 2021 at 11:38 AM Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Thanks Randall for the feedback!
> >
> > 1. Done
> >
> > 2. Yes that makes sense. I've updated the KIP
> >
> > 3.a I've updated to explicitly mention the plugin can be provided
> > using the fully qualified class or the alias.
> > 3.b Only concrete classes will be returned. I made the mistake when I
> > crafted the examples by hand. Fixed
> >
> > 4. If a plugin does not override config(), the response is an empty
> > array. I've updated the KIP
> >
> > 5. I have updated the compatibility section with more details
> >
> > 6. Can you clarify what you mean? KIP-507 was for securing internal
> > REST endpoints.
> >
> > 7. I don't think it is necessary. I think the use case Chris mentioned
> > could be useful for admins. I'm interested in exposing the
> > connector-level plugins and their configs. I thought it was easy to
> > add an API for worker plugins while working in this area but if there
> > isn't a strong consensus, I'm happy to drop it.
> >
> > Mickael
> >
> >
> >
> >
> >
> >
> > On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch <rh...@gmail.com> wrote:
> > >
> > > Thanks for the KIP, Mickael. And thanks to everyone else for the good
> > > discussion so far.
> > >
> > > Overall this is pretty good, but I do have a few questions/comments:
> > > 1. Can we include the HTTP method/verb in each of the endpoints in the
> > > "Public Interfaces" section?
> > > 2. The sample output for `GET /connector-plugins?connectorsOnly=false`
> > > includes a `version` field for the connector plugin, but not for the
> > > transform or converter plugins. Neither the `Transformation` or
> > > `Converter` interfaces have a version nor implement the `Versioned`
> > > interface, so should we add the `Versioned` interface to the
> > > `Transformation` and `Converter` interfaces and add a default method
> > > that returns "unknown"? Yes, we could do that in a separate KIP, but I
> > > think it's appropriate to do it in this KIP for a few reasons. First,
> > > we're already modifying the `Converter` interface in this KIP and it
> > > would be quite straightforward to also extend the `Versioned`
> > > interface and add a default method; then doing that for
> > > `Transformation` would also be relatively straightforward. Second,
> > > this KIP provides the justification for adding a version to the
> > > `Transformation` and `Converter` interfaces.
> > > 3. In the "Public Interfaces" section, there are two inconsistencies
> > > between the `GET /connector-plugins` and `GET
> > > /connector-plugins/<plugin>/config` methods. Both will make it
> > > difficult to use the second method, since the input to the second
> > > method is not directly found in the response from the first method:
> > >  a. The example for `GET /connector-plugins/Cast$Value/config` uses
> > > short name, or what Connect often refers to as the alias, "Cast$Value"
> > > for the plugin, rather than the fully-qualified name
> > > "org.apache.kafka.connect.transforms.Cast$Value". The KIP should
> > > specify whether it will support both the shortened name and the
> > > fully-qualified name, or whether only one of these will be supported.
> > > Not supporting the fully-qualified name will make it impossible to use
> > > the response from the first method as an input to the second.
> > >  a. The example for the list method returns the `Transformation` base
> > > and abstract classes (e.g.,
> > > "org.apache.kafka.connect.transforms.Cast") rather than the concrete
> > > classes (e.g., "org.apache.kafka.connect.transforms.Cast$Value" and
> > > "org.apache.kafka.connect.transforms.Cast$Key"), while the `GET
> > > /connector-plugins/<plugin>/config` will require the concrete class.
> > > 4. The "Public Interfaces" section's description of the `GET
> > > /connector-plugins/<plugin>/config` method should probably specify the
> > > expected behavior for `Converter` implementations that don't override
> > > the new `config()` method.
> > > 5. The "Compatibility, Deprecation, and Migration Plan" section should
> > > describe whether older `Converter` implementations (compiled with
> > > older versions of the Connect API without Converter extending
> > > Configurable) will be able to be installed into newer Connect
> > > runtimes, and what the behavior will be for the `GET
> > > /connector-plugins/<plugin>/config` method. (Although my suggestion
> > > for #5 will partially address the latter, we still need to talk about
> > > it in terms of compatibility.)
> > > 6. Can we add a small section addressing the security implications of
> > > these new APIs? KIP-507 [1] applied to APIs that existed at the time,
> > > and this KIP should at least mention how it aligns with that
> > > mechanism.
> > > 7. Do we really think adding `GET /worker-plugins` is necessary? While
> > > I understand the reasoning, I'm less convinced that it's worth adding
> > > to the public API. First, an administrator still has to modify the
> > > worker config for all workers to take advantage of that. Second,
> > > adding this method increases the surface area and therefore risk of
> > > additional attack vectors, despite not really being able to use the
> > > resulting information in any other part of the API.
> > >
> > > Best regards,
> > >
> > > Randall
> > >
> > > [1]
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
> > >
> > >
> > > On Wed, Dec 1, 2021 at 7:52 AM Mickael Maison <mi...@gmail.com>
> > wrote:
> > > >
> > > > Hi Chris,
> > > >
> > > > Thanks again for the feedback! I've updated the KIP based on our last
> > > > discussions. I've decided to include the new endpoint for worker
> > > > plugins.
> > > >
> > > > 1. Yes I agree, it's best to gate the new behavior.
> > > > 2. Yes, it was a remnant from the original proposal. I've now removed
> > > > the location field.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
> > > > <ch...@confluent.io.invalid> wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > I think that's a great idea! I especially like how we can establish
> > the
> > > > > expectation that any plugin type that appears in the response from
> > the GET
> > > > > /connector-plugins endpoint will have a corresponding GET
> > > > > /connector-plugins/<type>/config endpoint, but (if we decide to add
> > them in
> > > > > the future), worker plugins won't be expected to expose this kind of
> > > > > information and the different root path helps give a decent hint
> > about this.
> > > > >
> > > > > I also like the choice to return an empty ConfigDef from
> > Converter::config
> > > > > instead of null.
> > > > >
> > > > > Two things come to mind:
> > > > >
> > > > > 1. We may want to gate this behind a URL query parameter (maybe
> > something
> > > > > like "connectorsOnly") that defaults to the old behavior in order to
> > avoid
> > > > > breaking existing tools such as programmatic UIs that use the
> > endpoint
> > > > > today to discover the connectors that can be created by the user. We
> > can
> > > > > even plan to change the default for that parameter to the
> > newly-proposed
> > > > > behavior in the next major release, which should give people enough
> > time to
> > > > > either adapt to the expanded response format or add the query
> > parameter to
> > > > > their tooling.
> > > > >
> > > > > 2. The existing GET /connector-plugins endpoint doesn't contain
> > information
> > > > > on the location of the plugin on the worker's file system. Do you
> > think we
> > > > > should still include this info in the new response format? Correct
> > me if
> > > > > I'm wrong but it seems it may have been proposed originally to help
> > prevent
> > > > > already-addressed bugs in Connect classloading from striking; all
> > else
> > > > > equal, I'd personally err on the side of leaving this info out or at
> > least
> > > > > reducing permitted values for it to just "classpath" or "plugin
> > path" in
> > > > > order to avoid leaking worker file paths into the REST API, which
> > might
> > > > > bother super security-conscious users.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <
> > mickael.maison@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Chris,
> > > > > >
> > > > > > Yes to keep compatibility we want a default implementation for
> > > > > > Converter.configs(), I've updated the KIP.
> > > > > >
> > > > > > Regarding worker plugins, the use case you described seems
> > valuable.
> > > > > > I'd prefer not mixing worker and connector plugins on the same
> > > > > > endpoint but I agree using /plugins and /worker-plugins could be
> > > > > > confusing.
> > > > > >
> > > > > > One alternative is to expose all connector-level plugins via the
> > > > > > existing /connector-plugins endpoint. In that case, we'd need to
> > keep
> > > > > > the current JSON schema and not group plugins by type. As the
> > current
> > > > > > schema already has a type field for each entry, we'll still be
> > able to
> > > > > > tell them apart. Then we can have /worker-plugins and a relatively
> > > > > > clean API. What do you think?
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > >
> > > > > > > Hi Mickael,
> > > > > > >
> > > > > > > I think one potential use case for exposing worker-level plugins
> > is that
> > > > > > it
> > > > > > > may make it easier to determine whether a worker is set up
> > correctly (the
> > > > > > > current alternative requires looking through log files and can
> > be a
> > > > > > little
> > > > > > > tedious), and might even make it possible to automatically
> > identify
> > > > > > > discrepancies within a cluster by diffing the contents of that
> > endpoint
> > > > > > > across each worker. But I don't think this has to be addressed
> > by the
> > > > > > > current KIP; the only thing that bothers me a little is that
> > "plugins" is
> > > > > > > generic and it may confuse people down the road if we add an
> > endpoint for
> > > > > > > worker-level plugins ("why is one just called 'plugins' and the
> > other one
> > > > > > > is 'worker-plugins'?"). Probably not worth blocking on, though.
> > > > > > >
> > > > > > > Agreed that the suggestion for improved validation should be
> > made on the
> > > > > > > KIP-802 thread.
> > > > > > >
> > > > > > > I also noticed that the newly-proposed config method for the
> > Converter
> > > > > > > interface doesn't have a default implementation, making it
> > > > > > > backwards-incompatible. Should we add a default implementation
> > that
> > > > > > returns
> > > > > > > either null or an empty ConfigDef?
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <
> > mickael.maison@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Chris,
> > > > > > > >
> > > > > > > > 1. If we want to expose worker plugins, I think we should do
> > it via a
> > > > > > > > separate endpoint. But to be honest, I'm not even sure I see
> > strong
> > > > > > > > use cases for exposing them as they are either enabled or not
> > and
> > > > > > > > can't be changed at runtime. So I'd prefer to stick to
> > "connector
> > > > > > > > level" plugins in this KIP. Let me now if you have use cases,
> > I'm open
> > > > > > > > to reconsider this choice.
> > > > > > > > I'll add that in the rejected alternatives section for now
> > > > > > > >
> > > > > > > > 2. I remembered seeing issues in the past with multiple
> > plugin.path
> > > > > > > > entries but I tried today and I was able to mix and match
> > plugins from
> > > > > > > > different paths. So my bad for getting confused.
> > > > > > > > Then I agree, it makes more sense to group them by plugin type.
> > > > > > > >
> > > > > > > > 3. Yes this should be covered in KIP-802:
> > > > > > > >
> > > > > > > >
> > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > > > > > >
> > > > > > > > 4. No particular reason. We can support both formats like
> > today. I've
> > > > > > > > updated the KIP
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mickael
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > >
> > > > > > > > > Hi Mickael,
> > > > > > > > >
> > > > > > > > > I think the increase in scope here is great and the added
> > value
> > > > > > certainly
> > > > > > > > > justifies the proposed changes. I have some thoughts but
> > overall I
> > > > > > like
> > > > > > > > the
> > > > > > > > > direction this is going in now.
> > > > > > > > >
> > > > > > > > > 1. The new /plugins endpoint is described as containing "all
> > plugins
> > > > > > that
> > > > > > > > > are Connectors, Transformations, Converters,
> > HeaderConverters and
> > > > > > > > > Predicates". So essentially, it looks like we want to expose
> > all
> > > > > > plugins
> > > > > > > > > that are configured on a per-connector basis, but exclude
> > plugins
> > > > > > that
> > > > > > > > are
> > > > > > > > > configured on a per-worker basis (such as config providers
> > and REST
> > > > > > > > > extensions). Do you think it may be valuable to expose
> > information on
> > > > > > > > > worker-level plugins as well?
> > > > > > > > >
> > > > > > > > > 2. The description for the new /plugins endpoint also states
> > that
> > > > > > > > "Plugins
> > > > > > > > > will be grouped by plugin.path. This will make it clear to
> > users
> > > > > > what's
> > > > > > > > > available to use as it's not possible to use a Connector
> > from one
> > > > > > path
> > > > > > > > with
> > > > > > > > > Transformations from another.". Is this true? I thought that
> > > > > > Connect's
> > > > > > > > > classloading made it possible to package
> > > > > > > > > converters/transformations/predicates completely
> > independently from
> > > > > > each
> > > > > > > > > other, and to reference them from also-independently-packaged
> > > > > > connectors.
> > > > > > > > > If it turns out that this is the case, could we consider
> > > > > > restructuring
> > > > > > > > the
> > > > > > > > > response to be grouped by plugin type instead of by
> > classloader?
> > > > > > There's
> > > > > > > > > also the ungrouped format proposed in KIP-494 (
> > > > > > > > >
> > > > > > > >
> > > > > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > > > > > )
> > > > > > > > > which we might consider as well.
> > > > > > > > >
> > > > > > > > > 3. I think this can be left for a follow-up KIP if
> > necessary, but I'm
> > > > > > > > > curious about your thoughts on adding new validate methods
> > to all
> > > > > > > > > connector-level plugins that can be used similarly to how the
> > > > > > existing
> > > > > > > > > Connector::validate method (
> > > > > > > > >
> > > > > > > >
> > > > > >
> > https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > > > > > )
> > > > > > > > > is used. This would allow for plugins to perform validation
> > that's
> > > > > > more
> > > > > > > > > sophisticated than what the ConfigDef is capable of, such as
> > > > > > validating
> > > > > > > > > combinations of properties like a hostname and credentials
> > for
> > > > > > reaching
> > > > > > > > it.
> > > > > > > > > I know that at least Confluent's Avro, protobuf, and JSON
> > schema
> > > > > > > > converters
> > > > > > > > > would benefit from this kind of feature. It's a little
> > tangential to
> > > > > > this
> > > > > > > > > KIP (which at the moment is about discovering plugins and
> > their
> > > > > > > > > configuration surfaces, as opposed to validating them), but I
> > > > > > figured I'd
> > > > > > > > > ask since we're going to be expanding the Converter
> > interface and it
> > > > > > may
> > > > > > > > be
> > > > > > > > > useful to tackle this while we're in the neighborhood.
> > > > > > > > >
> > > > > > > > > 4. The description for the new
> > /plugins/<type>/<name>/configdef
> > > > > > endpoint
> > > > > > > > > states that "Name must be the fully qualified class name of
> > the
> > > > > > plugin".
> > > > > > > > > Any reason not to also support aliases (e.g.,
> > > > > > "FileStreamSinkConnector"
> > > > > > > > or
> > > > > > > > > "FileStreamSink" instead of
> > > > > > > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <
> > > > > > > > mickael.maison@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks all for the feedback!
> > > > > > > > > >
> > > > > > > > > > Chris,
> > > > > > > > > > I agree that fixing the current endpoint helps a lot.
> > Thanks for
> > > > > > > > > > raising these JIRAs and submitting a PR!
> > > > > > > > > > However thinking about the issue further, I decided to
> > expand the
> > > > > > > > > > scope of the KIP to cover all user-visible plugins.
> > > > > > > > > > In practice, users want to know about all available
> > plugins not
> > > > > > only
> > > > > > > > > > connectors. This includes transformations, converters,
> > > > > > > > > > header_converters and predicates. As we also want to
> > retrieve
> > > > > > > > > > configdef for these too, I think it makes sense to
> > introduce a new
> > > > > > > > > > endpoint to do so. Alongside we obviously need a new
> > endpoint for
> > > > > > > > > > listing all plugins.
> > > > > > > > > >
> > > > > > > > > > Gunnar,
> > > > > > > > > > I took a look at exposing valid values via the API. I
> > think the
> > > > > > issue
> > > > > > > > > > is that Validators don't expose a way to retrieve valid
> > values.
> > > > > > > > > > Changing validators will have an impact on all components
> > so I'd
> > > > > > > > > > prefer to address this requirement in a separate KIP. I
> > agree this
> > > > > > > > > > would be an interesting improvement and I'd happy to write
> > a KIP
> > > > > > for
> > > > > > > > > > it too.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I have updated the KIP accordingly. Let me know if you
> > have further
> > > > > > > > > > feedback.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Mickael
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > > > > > > > > > <gu...@googlemail.com.invalid> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > I'm +1 for adding a GET endpoint for obtaining config
> > > > > > definitions. It
> > > > > > > > > > > always felt odd to me that one has to issue a PUT for
> > that
> > > > > > purpose.
> > > > > > > > If
> > > > > > > > > > > nothing else, it'd be better in terms of discoverability
> > of the
> > > > > > KC
> > > > > > > > REST
> > > > > > > > > > API.
> > > > > > > > > > >
> > > > > > > > > > > One additional feature request I'd have is to expose the
> > valid
> > > > > > enum
> > > > > > > > > > > constants for enum-typed options. That'll help to
> > display the
> > > > > > values
> > > > > > > > in a
> > > > > > > > > > > drop-down or via radio buttons in a UI, give us tab
> > completion in
> > > > > > > > kcctl,
> > > > > > > > > > > etc.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > >
> > > > > > > > > > > --Gunnar
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > > > > > > > > > > <ch...@confluent.io.invalid>:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Viktor,
> > > > > > > > > > > >
> > > > > > > > > > > > It sounds like there are three major points here in
> > favor of a
> > > > > > new
> > > > > > > > GET
> > > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > > >
> > > > > > > > > > > > 1. You cannot issue a blank ("dummy") request for sink
> > > > > > connectors
> > > > > > > > > > because a
> > > > > > > > > > > > topic list/topic regex has to be supplied (otherwise
> > the PUT
> > > > > > > > endpoint
> > > > > > > > > > > > returns a 500 response)
> > > > > > > > > > > > 2. A dummy request still triggers custom validations
> > by the
> > > > > > > > connector,
> > > > > > > > > > > > which may be best to avoid if we know for sure that
> > the config
> > > > > > > > isn't
> > > > > > > > > > worth
> > > > > > > > > > > > validating yet
> > > > > > > > > > > > 3. It's more ergonomic and intuitive to be able to
> > issue a GET
> > > > > > > > request
> > > > > > > > > > > > without having to give a dummy connector config
> > > > > > > > > > > >
> > > > > > > > > > > > With regards to 1, this is actually a bug in Connect (
> > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-13327)
> > with a fix
> > > > > > > > already
> > > > > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > > > > https://github.com/apache/kafka/pull/11369). I think
> > it'd be
> > > > > > > > better to
> > > > > > > > > > > > focus on fixing this bug in general instead of
> > implementing a
> > > > > > new
> > > > > > > > REST
> > > > > > > > > > > > endpoint in order to allow people to work around it.
> > > > > > > > > > > >
> > > > > > > > > > > > With regards to 2, this is technically possible but
> > I'm unsure
> > > > > > > > it'd be
> > > > > > > > > > too
> > > > > > > > > > > > common out in the wild given that most validations
> > that could
> > > > > > be
> > > > > > > > > > expensive
> > > > > > > > > > > > would involve things like connecting to a database,
> > checking
> > > > > > if a
> > > > > > > > cloud
> > > > > > > > > > > > storage bucket exists, etc., none of which are
> > possible without
> > > > > > > > some
> > > > > > > > > > > > configuration properties from the user (db hostname,
> > bucket
> > > > > > name,
> > > > > > > > > > etc.).
> > > > > > > > > > > >
> > > > > > > > > > > > With regards to 3, I do agree that it'd be easier for
> > people
> > > > > > > > designing
> > > > > > > > > > UIs
> > > > > > > > > > > > to have a GET API to work against. I'm just not sure
> > it's
> > > > > > worth the
> > > > > > > > > > > > additional implementation, testing, and maintenance
> > burden. If
> > > > > > it
> > > > > > > > were
> > > > > > > > > > > > possible to issue a PUT request without unexpected
> > 500s for
> > > > > > invalid
> > > > > > > > > > > > configs, would that suffice? AFAICT it'd basically be
> > as
> > > > > > simple as
> > > > > > > > > > issuing
> > > > > > > > > > > > a PUT request with a dummy body consisting of nothing
> > except
> > > > > > the
> > > > > > > > > > connector
> > > > > > > > > > > > class (which at this point we might even make
> > unnecessary and
> > > > > > just
> > > > > > > > > > > > automatically replace with the connector class from
> > the URL)
> > > > > > and
> > > > > > > > then
> > > > > > > > > > > > filtering the response to just grab the "definition"
> > field of
> > > > > > each
> > > > > > > > > > element
> > > > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > >
> > > > > > > > > > > > Chris
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Folks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I too think this would be a very useful feature.
> > Some of our
> > > > > > > > > > management
> > > > > > > > > > > > > applications would provide a wizard for creating
> > connectors.
> > > > > > In
> > > > > > > > this
> > > > > > > > > > > > > scenario the user basically would fill out a sample
> > > > > > configuration
> > > > > > > > > > > > generated
> > > > > > > > > > > > > by the UI which would send it back to Connect for
> > validation
> > > > > > and
> > > > > > > > > > > > eventually
> > > > > > > > > > > > > create a new connector. The first part of this
> > workflow can
> > > > > > be
> > > > > > > > > > enhanced
> > > > > > > > > > > > if
> > > > > > > > > > > > > we had an API that can return the configuration
> > definition
> > > > > > of the
> > > > > > > > > > given
> > > > > > > > > > > > > type of connector as the UI application would be
> > able to
> > > > > > > > generate a
> > > > > > > > > > > > sample
> > > > > > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > > > The
> > connector-plugins/{connectorType}/config/validate API
> > > > > > > > essentially
> > > > > > > > > > > > works
> > > > > > > > > > > > > and returns the data that we need, however it is a
> > HTTP PUT
> > > > > > API
> > > > > > > > that
> > > > > > > > > > is a
> > > > > > > > > > > > > bit unintuitive for a fetch-like functionality and
> > also
> > > > > > > > functionally
> > > > > > > > > > > > > different as it validates the given (dummy) request.
> > In case
> > > > > > of
> > > > > > > > sink
> > > > > > > > > > > > > connectors one would need to also provide a topic
> > name.
> > > > > > > > > > > > >
> > > > > > > > > > > > > A suggestion for the KIP: I think it can be useful
> > to return
> > > > > > the
> > > > > > > > > > config
> > > > > > > > > > > > > groups and the connector class' name similarly to the
> > > > > > validate
> > > > > > > > API
> > > > > > > > > > just
> > > > > > > > > > > > in
> > > > > > > > > > > > > case any frontend needs them (and also the response
> > would be
> > > > > > more
> > > > > > > > > > like
> > > > > > > > > > > > the
> > > > > > > > > > > > > validate API but simpler).
> > > > > > > > > > > > >
> > > > > > > > > > > > > Viktor
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > > > > ryannedolan@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I think it'd be worth adding a GET version, fwiw.
> > Could be
> > > > > > the
> > > > > > > > same
> > > > > > > > > > > > > handler
> > > > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > You're right, you can achieve the same
> > functionality
> > > > > > using
> > > > > > > > the
> > > > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > > > In my mind it was only for validation once you
> > have
> > > > > > build a
> > > > > > > > > > > > > > > configuration but when used with an empty
> > configuration,
> > > > > > it
> > > > > > > > > > basically
> > > > > > > > > > > > > > > serves the same purpose as the proposed new
> > endpoint.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think it's a bit easier to use a GET endpoint
> > but I
> > > > > > don't
> > > > > > > > > > think it
> > > > > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'm wondering about the use case here. The
> > motivation
> > > > > > > > section
> > > > > > > > > > > > states
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > "Connect does not provide a way to see what
> > > > > > configurations
> > > > > > > > a
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > requires. Instead users have to go look at the
> > > > > > connector
> > > > > > > > > > > > > documentation
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > in the worst case, look directly at the
> > connector
> > > > > > source
> > > > > > > > > > code.",
> > > > > > > > > > > > and
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > with this KIP, "users will be able to discover
> > the
> > > > > > required
> > > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > > for connectors installed in a Connect cluster"
> > and
> > > > > > "tools
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > > able
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > generate wizards for configuring and starting
> > > > > > connectors".
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > > >
> > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > > > endpoint not address these points? What will
> > the
> > > > > > > > newly-proposed
> > > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > allow users to do that they will not already
> > be able
> > > > > > to do
> > > > > > > > > > with the
> > > > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison
> > <
> > > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I've created KIP-769 to expose connector
> > > > > > configuration
> > > > > > > > > > > > definitions
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Please take a look and let me know if you
> > have any
> > > > > > > > feedback.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> >

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Chris Egerton <ch...@confluent.io.INVALID>]

Hi Mickael and Randall,

I think there's a decent use case for exposing worker plugins. If a cluster
administrator wants to enable a worker plugin on their cluster, it may make
things easier if they can verify ahead of time that that plugin is
available on every worker in the cluster (and possibly that the same
version is present on every worker). The current alternative to this would
require sifting through worker logs.

I don't think the additional workload to expose worker plugins is
significant (it should largely mirror the same logic we'll use to expose
connector plugins), but if there's serious objection to including that
feature in this KIP, we could probably live without it now that we've made
sure that the current proposal doesn't block us from adding it in a
reasonable way later on down the road.

Cheers,

Chris

On Fri, Dec 3, 2021 at 11:38 AM Mickael Maison <mi...@gmail.com>
wrote:

> Thanks Randall for the feedback!
>
> 1. Done
>
> 2. Yes that makes sense. I've updated the KIP
>
> 3.a I've updated to explicitly mention the plugin can be provided
> using the fully qualified class or the alias.
> 3.b Only concrete classes will be returned. I made the mistake when I
> crafted the examples by hand. Fixed
>
> 4. If a plugin does not override config(), the response is an empty
> array. I've updated the KIP
>
> 5. I have updated the compatibility section with more details
>
> 6. Can you clarify what you mean? KIP-507 was for securing internal
> REST endpoints.
>
> 7. I don't think it is necessary. I think the use case Chris mentioned
> could be useful for admins. I'm interested in exposing the
> connector-level plugins and their configs. I thought it was easy to
> add an API for worker plugins while working in this area but if there
> isn't a strong consensus, I'm happy to drop it.
>
> Mickael
>
>
>
>
>
>
> On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch <rh...@gmail.com> wrote:
> >
> > Thanks for the KIP, Mickael. And thanks to everyone else for the good
> > discussion so far.
> >
> > Overall this is pretty good, but I do have a few questions/comments:
> > 1. Can we include the HTTP method/verb in each of the endpoints in the
> > "Public Interfaces" section?
> > 2. The sample output for `GET /connector-plugins?connectorsOnly=false`
> > includes a `version` field for the connector plugin, but not for the
> > transform or converter plugins. Neither the `Transformation` or
> > `Converter` interfaces have a version nor implement the `Versioned`
> > interface, so should we add the `Versioned` interface to the
> > `Transformation` and `Converter` interfaces and add a default method
> > that returns "unknown"? Yes, we could do that in a separate KIP, but I
> > think it's appropriate to do it in this KIP for a few reasons. First,
> > we're already modifying the `Converter` interface in this KIP and it
> > would be quite straightforward to also extend the `Versioned`
> > interface and add a default method; then doing that for
> > `Transformation` would also be relatively straightforward. Second,
> > this KIP provides the justification for adding a version to the
> > `Transformation` and `Converter` interfaces.
> > 3. In the "Public Interfaces" section, there are two inconsistencies
> > between the `GET /connector-plugins` and `GET
> > /connector-plugins/<plugin>/config` methods. Both will make it
> > difficult to use the second method, since the input to the second
> > method is not directly found in the response from the first method:
> >  a. The example for `GET /connector-plugins/Cast$Value/config` uses
> > short name, or what Connect often refers to as the alias, "Cast$Value"
> > for the plugin, rather than the fully-qualified name
> > "org.apache.kafka.connect.transforms.Cast$Value". The KIP should
> > specify whether it will support both the shortened name and the
> > fully-qualified name, or whether only one of these will be supported.
> > Not supporting the fully-qualified name will make it impossible to use
> > the response from the first method as an input to the second.
> >  a. The example for the list method returns the `Transformation` base
> > and abstract classes (e.g.,
> > "org.apache.kafka.connect.transforms.Cast") rather than the concrete
> > classes (e.g., "org.apache.kafka.connect.transforms.Cast$Value" and
> > "org.apache.kafka.connect.transforms.Cast$Key"), while the `GET
> > /connector-plugins/<plugin>/config` will require the concrete class.
> > 4. The "Public Interfaces" section's description of the `GET
> > /connector-plugins/<plugin>/config` method should probably specify the
> > expected behavior for `Converter` implementations that don't override
> > the new `config()` method.
> > 5. The "Compatibility, Deprecation, and Migration Plan" section should
> > describe whether older `Converter` implementations (compiled with
> > older versions of the Connect API without Converter extending
> > Configurable) will be able to be installed into newer Connect
> > runtimes, and what the behavior will be for the `GET
> > /connector-plugins/<plugin>/config` method. (Although my suggestion
> > for #5 will partially address the latter, we still need to talk about
> > it in terms of compatibility.)
> > 6. Can we add a small section addressing the security implications of
> > these new APIs? KIP-507 [1] applied to APIs that existed at the time,
> > and this KIP should at least mention how it aligns with that
> > mechanism.
> > 7. Do we really think adding `GET /worker-plugins` is necessary? While
> > I understand the reasoning, I'm less convinced that it's worth adding
> > to the public API. First, an administrator still has to modify the
> > worker config for all workers to take advantage of that. Second,
> > adding this method increases the surface area and therefore risk of
> > additional attack vectors, despite not really being able to use the
> > resulting information in any other part of the API.
> >
> > Best regards,
> >
> > Randall
> >
> > [1]
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
> >
> >
> > On Wed, Dec 1, 2021 at 7:52 AM Mickael Maison <mi...@gmail.com>
> wrote:
> > >
> > > Hi Chris,
> > >
> > > Thanks again for the feedback! I've updated the KIP based on our last
> > > discussions. I've decided to include the new endpoint for worker
> > > plugins.
> > >
> > > 1. Yes I agree, it's best to gate the new behavior.
> > > 2. Yes, it was a remnant from the original proposal. I've now removed
> > > the location field.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
> > > <ch...@confluent.io.invalid> wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > I think that's a great idea! I especially like how we can establish
> the
> > > > expectation that any plugin type that appears in the response from
> the GET
> > > > /connector-plugins endpoint will have a corresponding GET
> > > > /connector-plugins/<type>/config endpoint, but (if we decide to add
> them in
> > > > the future), worker plugins won't be expected to expose this kind of
> > > > information and the different root path helps give a decent hint
> about this.
> > > >
> > > > I also like the choice to return an empty ConfigDef from
> Converter::config
> > > > instead of null.
> > > >
> > > > Two things come to mind:
> > > >
> > > > 1. We may want to gate this behind a URL query parameter (maybe
> something
> > > > like "connectorsOnly") that defaults to the old behavior in order to
> avoid
> > > > breaking existing tools such as programmatic UIs that use the
> endpoint
> > > > today to discover the connectors that can be created by the user. We
> can
> > > > even plan to change the default for that parameter to the
> newly-proposed
> > > > behavior in the next major release, which should give people enough
> time to
> > > > either adapt to the expanded response format or add the query
> parameter to
> > > > their tooling.
> > > >
> > > > 2. The existing GET /connector-plugins endpoint doesn't contain
> information
> > > > on the location of the plugin on the worker's file system. Do you
> think we
> > > > should still include this info in the new response format? Correct
> me if
> > > > I'm wrong but it seems it may have been proposed originally to help
> prevent
> > > > already-addressed bugs in Connect classloading from striking; all
> else
> > > > equal, I'd personally err on the side of leaving this info out or at
> least
> > > > reducing permitted values for it to just "classpath" or "plugin
> path" in
> > > > order to avoid leaking worker file paths into the REST API, which
> might
> > > > bother super security-conscious users.
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <
> mickael.maison@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Chris,
> > > > >
> > > > > Yes to keep compatibility we want a default implementation for
> > > > > Converter.configs(), I've updated the KIP.
> > > > >
> > > > > Regarding worker plugins, the use case you described seems
> valuable.
> > > > > I'd prefer not mixing worker and connector plugins on the same
> > > > > endpoint but I agree using /plugins and /worker-plugins could be
> > > > > confusing.
> > > > >
> > > > > One alternative is to expose all connector-level plugins via the
> > > > > existing /connector-plugins endpoint. In that case, we'd need to
> keep
> > > > > the current JSON schema and not group plugins by type. As the
> current
> > > > > schema already has a type field for each entry, we'll still be
> able to
> > > > > tell them apart. Then we can have /worker-plugins and a relatively
> > > > > clean API. What do you think?
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > > > > <ch...@confluent.io.invalid> wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > I think one potential use case for exposing worker-level plugins
> is that
> > > > > it
> > > > > > may make it easier to determine whether a worker is set up
> correctly (the
> > > > > > current alternative requires looking through log files and can
> be a
> > > > > little
> > > > > > tedious), and might even make it possible to automatically
> identify
> > > > > > discrepancies within a cluster by diffing the contents of that
> endpoint
> > > > > > across each worker. But I don't think this has to be addressed
> by the
> > > > > > current KIP; the only thing that bothers me a little is that
> "plugins" is
> > > > > > generic and it may confuse people down the road if we add an
> endpoint for
> > > > > > worker-level plugins ("why is one just called 'plugins' and the
> other one
> > > > > > is 'worker-plugins'?"). Probably not worth blocking on, though.
> > > > > >
> > > > > > Agreed that the suggestion for improved validation should be
> made on the
> > > > > > KIP-802 thread.
> > > > > >
> > > > > > I also noticed that the newly-proposed config method for the
> Converter
> > > > > > interface doesn't have a default implementation, making it
> > > > > > backwards-incompatible. Should we add a default implementation
> that
> > > > > returns
> > > > > > either null or an empty ConfigDef?
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <
> mickael.maison@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Chris,
> > > > > > >
> > > > > > > 1. If we want to expose worker plugins, I think we should do
> it via a
> > > > > > > separate endpoint. But to be honest, I'm not even sure I see
> strong
> > > > > > > use cases for exposing them as they are either enabled or not
> and
> > > > > > > can't be changed at runtime. So I'd prefer to stick to
> "connector
> > > > > > > level" plugins in this KIP. Let me now if you have use cases,
> I'm open
> > > > > > > to reconsider this choice.
> > > > > > > I'll add that in the rejected alternatives section for now
> > > > > > >
> > > > > > > 2. I remembered seeing issues in the past with multiple
> plugin.path
> > > > > > > entries but I tried today and I was able to mix and match
> plugins from
> > > > > > > different paths. So my bad for getting confused.
> > > > > > > Then I agree, it makes more sense to group them by plugin type.
> > > > > > >
> > > > > > > 3. Yes this should be covered in KIP-802:
> > > > > > >
> > > > > > >
> > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > > > > >
> > > > > > > 4. No particular reason. We can support both formats like
> today. I've
> > > > > > > updated the KIP
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mickael
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > >
> > > > > > > > Hi Mickael,
> > > > > > > >
> > > > > > > > I think the increase in scope here is great and the added
> value
> > > > > certainly
> > > > > > > > justifies the proposed changes. I have some thoughts but
> overall I
> > > > > like
> > > > > > > the
> > > > > > > > direction this is going in now.
> > > > > > > >
> > > > > > > > 1. The new /plugins endpoint is described as containing "all
> plugins
> > > > > that
> > > > > > > > are Connectors, Transformations, Converters,
> HeaderConverters and
> > > > > > > > Predicates". So essentially, it looks like we want to expose
> all
> > > > > plugins
> > > > > > > > that are configured on a per-connector basis, but exclude
> plugins
> > > > > that
> > > > > > > are
> > > > > > > > configured on a per-worker basis (such as config providers
> and REST
> > > > > > > > extensions). Do you think it may be valuable to expose
> information on
> > > > > > > > worker-level plugins as well?
> > > > > > > >
> > > > > > > > 2. The description for the new /plugins endpoint also states
> that
> > > > > > > "Plugins
> > > > > > > > will be grouped by plugin.path. This will make it clear to
> users
> > > > > what's
> > > > > > > > available to use as it's not possible to use a Connector
> from one
> > > > > path
> > > > > > > with
> > > > > > > > Transformations from another.". Is this true? I thought that
> > > > > Connect's
> > > > > > > > classloading made it possible to package
> > > > > > > > converters/transformations/predicates completely
> independently from
> > > > > each
> > > > > > > > other, and to reference them from also-independently-packaged
> > > > > connectors.
> > > > > > > > If it turns out that this is the case, could we consider
> > > > > restructuring
> > > > > > > the
> > > > > > > > response to be grouped by plugin type instead of by
> classloader?
> > > > > There's
> > > > > > > > also the ungrouped format proposed in KIP-494 (
> > > > > > > >
> > > > > > >
> > > > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > > > > )
> > > > > > > > which we might consider as well.
> > > > > > > >
> > > > > > > > 3. I think this can be left for a follow-up KIP if
> necessary, but I'm
> > > > > > > > curious about your thoughts on adding new validate methods
> to all
> > > > > > > > connector-level plugins that can be used similarly to how the
> > > > > existing
> > > > > > > > Connector::validate method (
> > > > > > > >
> > > > > > >
> > > > >
> https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > > > > )
> > > > > > > > is used. This would allow for plugins to perform validation
> that's
> > > > > more
> > > > > > > > sophisticated than what the ConfigDef is capable of, such as
> > > > > validating
> > > > > > > > combinations of properties like a hostname and credentials
> for
> > > > > reaching
> > > > > > > it.
> > > > > > > > I know that at least Confluent's Avro, protobuf, and JSON
> schema
> > > > > > > converters
> > > > > > > > would benefit from this kind of feature. It's a little
> tangential to
> > > > > this
> > > > > > > > KIP (which at the moment is about discovering plugins and
> their
> > > > > > > > configuration surfaces, as opposed to validating them), but I
> > > > > figured I'd
> > > > > > > > ask since we're going to be expanding the Converter
> interface and it
> > > > > may
> > > > > > > be
> > > > > > > > useful to tackle this while we're in the neighborhood.
> > > > > > > >
> > > > > > > > 4. The description for the new
> /plugins/<type>/<name>/configdef
> > > > > endpoint
> > > > > > > > states that "Name must be the fully qualified class name of
> the
> > > > > plugin".
> > > > > > > > Any reason not to also support aliases (e.g.,
> > > > > "FileStreamSinkConnector"
> > > > > > > or
> > > > > > > > "FileStreamSink" instead of
> > > > > > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <
> > > > > > > mickael.maison@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks all for the feedback!
> > > > > > > > >
> > > > > > > > > Chris,
> > > > > > > > > I agree that fixing the current endpoint helps a lot.
> Thanks for
> > > > > > > > > raising these JIRAs and submitting a PR!
> > > > > > > > > However thinking about the issue further, I decided to
> expand the
> > > > > > > > > scope of the KIP to cover all user-visible plugins.
> > > > > > > > > In practice, users want to know about all available
> plugins not
> > > > > only
> > > > > > > > > connectors. This includes transformations, converters,
> > > > > > > > > header_converters and predicates. As we also want to
> retrieve
> > > > > > > > > configdef for these too, I think it makes sense to
> introduce a new
> > > > > > > > > endpoint to do so. Alongside we obviously need a new
> endpoint for
> > > > > > > > > listing all plugins.
> > > > > > > > >
> > > > > > > > > Gunnar,
> > > > > > > > > I took a look at exposing valid values via the API. I
> think the
> > > > > issue
> > > > > > > > > is that Validators don't expose a way to retrieve valid
> values.
> > > > > > > > > Changing validators will have an impact on all components
> so I'd
> > > > > > > > > prefer to address this requirement in a separate KIP. I
> agree this
> > > > > > > > > would be an interesting improvement and I'd happy to write
> a KIP
> > > > > for
> > > > > > > > > it too.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I have updated the KIP accordingly. Let me know if you
> have further
> > > > > > > > > feedback.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Mickael
> > > > > > > > >
> > > > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > > > > > > > > <gu...@googlemail.com.invalid> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I'm +1 for adding a GET endpoint for obtaining config
> > > > > definitions. It
> > > > > > > > > > always felt odd to me that one has to issue a PUT for
> that
> > > > > purpose.
> > > > > > > If
> > > > > > > > > > nothing else, it'd be better in terms of discoverability
> of the
> > > > > KC
> > > > > > > REST
> > > > > > > > > API.
> > > > > > > > > >
> > > > > > > > > > One additional feature request I'd have is to expose the
> valid
> > > > > enum
> > > > > > > > > > constants for enum-typed options. That'll help to
> display the
> > > > > values
> > > > > > > in a
> > > > > > > > > > drop-down or via radio buttons in a UI, give us tab
> completion in
> > > > > > > kcctl,
> > > > > > > > > > etc.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > >
> > > > > > > > > > --Gunnar
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > > > > > > > > > <ch...@confluent.io.invalid>:
> > > > > > > > > >
> > > > > > > > > > > Hi Viktor,
> > > > > > > > > > >
> > > > > > > > > > > It sounds like there are three major points here in
> favor of a
> > > > > new
> > > > > > > GET
> > > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > > >
> > > > > > > > > > > 1. You cannot issue a blank ("dummy") request for sink
> > > > > connectors
> > > > > > > > > because a
> > > > > > > > > > > topic list/topic regex has to be supplied (otherwise
> the PUT
> > > > > > > endpoint
> > > > > > > > > > > returns a 500 response)
> > > > > > > > > > > 2. A dummy request still triggers custom validations
> by the
> > > > > > > connector,
> > > > > > > > > > > which may be best to avoid if we know for sure that
> the config
> > > > > > > isn't
> > > > > > > > > worth
> > > > > > > > > > > validating yet
> > > > > > > > > > > 3. It's more ergonomic and intuitive to be able to
> issue a GET
> > > > > > > request
> > > > > > > > > > > without having to give a dummy connector config
> > > > > > > > > > >
> > > > > > > > > > > With regards to 1, this is actually a bug in Connect (
> > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-13327)
> with a fix
> > > > > > > already
> > > > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > > > https://github.com/apache/kafka/pull/11369). I think
> it'd be
> > > > > > > better to
> > > > > > > > > > > focus on fixing this bug in general instead of
> implementing a
> > > > > new
> > > > > > > REST
> > > > > > > > > > > endpoint in order to allow people to work around it.
> > > > > > > > > > >
> > > > > > > > > > > With regards to 2, this is technically possible but
> I'm unsure
> > > > > > > it'd be
> > > > > > > > > too
> > > > > > > > > > > common out in the wild given that most validations
> that could
> > > > > be
> > > > > > > > > expensive
> > > > > > > > > > > would involve things like connecting to a database,
> checking
> > > > > if a
> > > > > > > cloud
> > > > > > > > > > > storage bucket exists, etc., none of which are
> possible without
> > > > > > > some
> > > > > > > > > > > configuration properties from the user (db hostname,
> bucket
> > > > > name,
> > > > > > > > > etc.).
> > > > > > > > > > >
> > > > > > > > > > > With regards to 3, I do agree that it'd be easier for
> people
> > > > > > > designing
> > > > > > > > > UIs
> > > > > > > > > > > to have a GET API to work against. I'm just not sure
> it's
> > > > > worth the
> > > > > > > > > > > additional implementation, testing, and maintenance
> burden. If
> > > > > it
> > > > > > > were
> > > > > > > > > > > possible to issue a PUT request without unexpected
> 500s for
> > > > > invalid
> > > > > > > > > > > configs, would that suffice? AFAICT it'd basically be
> as
> > > > > simple as
> > > > > > > > > issuing
> > > > > > > > > > > a PUT request with a dummy body consisting of nothing
> except
> > > > > the
> > > > > > > > > connector
> > > > > > > > > > > class (which at this point we might even make
> unnecessary and
> > > > > just
> > > > > > > > > > > automatically replace with the connector class from
> the URL)
> > > > > and
> > > > > > > then
> > > > > > > > > > > filtering the response to just grab the "definition"
> field of
> > > > > each
> > > > > > > > > element
> > > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > Chris
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Folks,
> > > > > > > > > > > >
> > > > > > > > > > > > I too think this would be a very useful feature.
> Some of our
> > > > > > > > > management
> > > > > > > > > > > > applications would provide a wizard for creating
> connectors.
> > > > > In
> > > > > > > this
> > > > > > > > > > > > scenario the user basically would fill out a sample
> > > > > configuration
> > > > > > > > > > > generated
> > > > > > > > > > > > by the UI which would send it back to Connect for
> validation
> > > > > and
> > > > > > > > > > > eventually
> > > > > > > > > > > > create a new connector. The first part of this
> workflow can
> > > > > be
> > > > > > > > > enhanced
> > > > > > > > > > > if
> > > > > > > > > > > > we had an API that can return the configuration
> definition
> > > > > of the
> > > > > > > > > given
> > > > > > > > > > > > type of connector as the UI application would be
> able to
> > > > > > > generate a
> > > > > > > > > > > sample
> > > > > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > > The
> connector-plugins/{connectorType}/config/validate API
> > > > > > > essentially
> > > > > > > > > > > works
> > > > > > > > > > > > and returns the data that we need, however it is a
> HTTP PUT
> > > > > API
> > > > > > > that
> > > > > > > > > is a
> > > > > > > > > > > > bit unintuitive for a fetch-like functionality and
> also
> > > > > > > functionally
> > > > > > > > > > > > different as it validates the given (dummy) request.
> In case
> > > > > of
> > > > > > > sink
> > > > > > > > > > > > connectors one would need to also provide a topic
> name.
> > > > > > > > > > > >
> > > > > > > > > > > > A suggestion for the KIP: I think it can be useful
> to return
> > > > > the
> > > > > > > > > config
> > > > > > > > > > > > groups and the connector class' name similarly to the
> > > > > validate
> > > > > > > API
> > > > > > > > > just
> > > > > > > > > > > in
> > > > > > > > > > > > case any frontend needs them (and also the response
> would be
> > > > > more
> > > > > > > > > like
> > > > > > > > > > > the
> > > > > > > > > > > > validate API but simpler).
> > > > > > > > > > > >
> > > > > > > > > > > > Viktor
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > > > ryannedolan@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I think it'd be worth adding a GET version, fwiw.
> Could be
> > > > > the
> > > > > > > same
> > > > > > > > > > > > handler
> > > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > You're right, you can achieve the same
> functionality
> > > > > using
> > > > > > > the
> > > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > > In my mind it was only for validation once you
> have
> > > > > build a
> > > > > > > > > > > > > > configuration but when used with an empty
> configuration,
> > > > > it
> > > > > > > > > basically
> > > > > > > > > > > > > > serves the same purpose as the proposed new
> endpoint.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think it's a bit easier to use a GET endpoint
> but I
> > > > > don't
> > > > > > > > > think it
> > > > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm wondering about the use case here. The
> motivation
> > > > > > > section
> > > > > > > > > > > states
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > "Connect does not provide a way to see what
> > > > > configurations
> > > > > > > a
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > > requires. Instead users have to go look at the
> > > > > connector
> > > > > > > > > > > > documentation
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > in the worst case, look directly at the
> connector
> > > > > source
> > > > > > > > > code.",
> > > > > > > > > > > and
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > with this KIP, "users will be able to discover
> the
> > > > > required
> > > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > > for connectors installed in a Connect cluster"
> and
> > > > > "tools
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > able
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > generate wizards for configuring and starting
> > > > > connectors".
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > >
> /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > > endpoint not address these points? What will
> the
> > > > > > > newly-proposed
> > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > allow users to do that they will not already
> be able
> > > > > to do
> > > > > > > > > with the
> > > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison
> <
> > > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I've created KIP-769 to expose connector
> > > > > configuration
> > > > > > > > > > > definitions
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Please take a look and let me know if you
> have any
> > > > > > > feedback.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
>

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Mickael Maison <mi...@gmail.com>]

Thanks Randall for the feedback!

1. Done

2. Yes that makes sense. I've updated the KIP

3.a I've updated to explicitly mention the plugin can be provided
using the fully qualified class or the alias.
3.b Only concrete classes will be returned. I made the mistake when I
crafted the examples by hand. Fixed

4. If a plugin does not override config(), the response is an empty
array. I've updated the KIP

5. I have updated the compatibility section with more details

6. Can you clarify what you mean? KIP-507 was for securing internal
REST endpoints.

7. I don't think it is necessary. I think the use case Chris mentioned
could be useful for admins. I'm interested in exposing the
connector-level plugins and their configs. I thought it was easy to
add an API for worker plugins while working in this area but if there
isn't a strong consensus, I'm happy to drop it.

Mickael






On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch <rh...@gmail.com> wrote:
>
> Thanks for the KIP, Mickael. And thanks to everyone else for the good
> discussion so far.
>
> Overall this is pretty good, but I do have a few questions/comments:
> 1. Can we include the HTTP method/verb in each of the endpoints in the
> "Public Interfaces" section?
> 2. The sample output for `GET /connector-plugins?connectorsOnly=false`
> includes a `version` field for the connector plugin, but not for the
> transform or converter plugins. Neither the `Transformation` or
> `Converter` interfaces have a version nor implement the `Versioned`
> interface, so should we add the `Versioned` interface to the
> `Transformation` and `Converter` interfaces and add a default method
> that returns "unknown"? Yes, we could do that in a separate KIP, but I
> think it's appropriate to do it in this KIP for a few reasons. First,
> we're already modifying the `Converter` interface in this KIP and it
> would be quite straightforward to also extend the `Versioned`
> interface and add a default method; then doing that for
> `Transformation` would also be relatively straightforward. Second,
> this KIP provides the justification for adding a version to the
> `Transformation` and `Converter` interfaces.
> 3. In the "Public Interfaces" section, there are two inconsistencies
> between the `GET /connector-plugins` and `GET
> /connector-plugins/<plugin>/config` methods. Both will make it
> difficult to use the second method, since the input to the second
> method is not directly found in the response from the first method:
>  a. The example for `GET /connector-plugins/Cast$Value/config` uses
> short name, or what Connect often refers to as the alias, "Cast$Value"
> for the plugin, rather than the fully-qualified name
> "org.apache.kafka.connect.transforms.Cast$Value". The KIP should
> specify whether it will support both the shortened name and the
> fully-qualified name, or whether only one of these will be supported.
> Not supporting the fully-qualified name will make it impossible to use
> the response from the first method as an input to the second.
>  a. The example for the list method returns the `Transformation` base
> and abstract classes (e.g.,
> "org.apache.kafka.connect.transforms.Cast") rather than the concrete
> classes (e.g., "org.apache.kafka.connect.transforms.Cast$Value" and
> "org.apache.kafka.connect.transforms.Cast$Key"), while the `GET
> /connector-plugins/<plugin>/config` will require the concrete class.
> 4. The "Public Interfaces" section's description of the `GET
> /connector-plugins/<plugin>/config` method should probably specify the
> expected behavior for `Converter` implementations that don't override
> the new `config()` method.
> 5. The "Compatibility, Deprecation, and Migration Plan" section should
> describe whether older `Converter` implementations (compiled with
> older versions of the Connect API without Converter extending
> Configurable) will be able to be installed into newer Connect
> runtimes, and what the behavior will be for the `GET
> /connector-plugins/<plugin>/config` method. (Although my suggestion
> for #5 will partially address the latter, we still need to talk about
> it in terms of compatibility.)
> 6. Can we add a small section addressing the security implications of
> these new APIs? KIP-507 [1] applied to APIs that existed at the time,
> and this KIP should at least mention how it aligns with that
> mechanism.
> 7. Do we really think adding `GET /worker-plugins` is necessary? While
> I understand the reasoning, I'm less convinced that it's worth adding
> to the public API. First, an administrator still has to modify the
> worker config for all workers to take advantage of that. Second,
> adding this method increases the surface area and therefore risk of
> additional attack vectors, despite not really being able to use the
> resulting information in any other part of the API.
>
> Best regards,
>
> Randall
>
> [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
>
>
> On Wed, Dec 1, 2021 at 7:52 AM Mickael Maison <mi...@gmail.com> wrote:
> >
> > Hi Chris,
> >
> > Thanks again for the feedback! I've updated the KIP based on our last
> > discussions. I've decided to include the new endpoint for worker
> > plugins.
> >
> > 1. Yes I agree, it's best to gate the new behavior.
> > 2. Yes, it was a remnant from the original proposal. I've now removed
> > the location field.
> >
> > Thanks,
> > Mickael
> >
> > On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
> > <ch...@confluent.io.invalid> wrote:
> > >
> > > Hi Mickael,
> > >
> > > I think that's a great idea! I especially like how we can establish the
> > > expectation that any plugin type that appears in the response from the GET
> > > /connector-plugins endpoint will have a corresponding GET
> > > /connector-plugins/<type>/config endpoint, but (if we decide to add them in
> > > the future), worker plugins won't be expected to expose this kind of
> > > information and the different root path helps give a decent hint about this.
> > >
> > > I also like the choice to return an empty ConfigDef from Converter::config
> > > instead of null.
> > >
> > > Two things come to mind:
> > >
> > > 1. We may want to gate this behind a URL query parameter (maybe something
> > > like "connectorsOnly") that defaults to the old behavior in order to avoid
> > > breaking existing tools such as programmatic UIs that use the endpoint
> > > today to discover the connectors that can be created by the user. We can
> > > even plan to change the default for that parameter to the newly-proposed
> > > behavior in the next major release, which should give people enough time to
> > > either adapt to the expanded response format or add the query parameter to
> > > their tooling.
> > >
> > > 2. The existing GET /connector-plugins endpoint doesn't contain information
> > > on the location of the plugin on the worker's file system. Do you think we
> > > should still include this info in the new response format? Correct me if
> > > I'm wrong but it seems it may have been proposed originally to help prevent
> > > already-addressed bugs in Connect classloading from striking; all else
> > > equal, I'd personally err on the side of leaving this info out or at least
> > > reducing permitted values for it to just "classpath" or "plugin path" in
> > > order to avoid leaking worker file paths into the REST API, which might
> > > bother super security-conscious users.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <mi...@gmail.com>
> > > wrote:
> > >
> > > > Hi Chris,
> > > >
> > > > Yes to keep compatibility we want a default implementation for
> > > > Converter.configs(), I've updated the KIP.
> > > >
> > > > Regarding worker plugins, the use case you described seems valuable.
> > > > I'd prefer not mixing worker and connector plugins on the same
> > > > endpoint but I agree using /plugins and /worker-plugins could be
> > > > confusing.
> > > >
> > > > One alternative is to expose all connector-level plugins via the
> > > > existing /connector-plugins endpoint. In that case, we'd need to keep
> > > > the current JSON schema and not group plugins by type. As the current
> > > > schema already has a type field for each entry, we'll still be able to
> > > > tell them apart. Then we can have /worker-plugins and a relatively
> > > > clean API. What do you think?
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > > > <ch...@confluent.io.invalid> wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > I think one potential use case for exposing worker-level plugins is that
> > > > it
> > > > > may make it easier to determine whether a worker is set up correctly (the
> > > > > current alternative requires looking through log files and can be a
> > > > little
> > > > > tedious), and might even make it possible to automatically identify
> > > > > discrepancies within a cluster by diffing the contents of that endpoint
> > > > > across each worker. But I don't think this has to be addressed by the
> > > > > current KIP; the only thing that bothers me a little is that "plugins" is
> > > > > generic and it may confuse people down the road if we add an endpoint for
> > > > > worker-level plugins ("why is one just called 'plugins' and the other one
> > > > > is 'worker-plugins'?"). Probably not worth blocking on, though.
> > > > >
> > > > > Agreed that the suggestion for improved validation should be made on the
> > > > > KIP-802 thread.
> > > > >
> > > > > I also noticed that the newly-proposed config method for the Converter
> > > > > interface doesn't have a default implementation, making it
> > > > > backwards-incompatible. Should we add a default implementation that
> > > > returns
> > > > > either null or an empty ConfigDef?
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <mickael.maison@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Chris,
> > > > > >
> > > > > > 1. If we want to expose worker plugins, I think we should do it via a
> > > > > > separate endpoint. But to be honest, I'm not even sure I see strong
> > > > > > use cases for exposing them as they are either enabled or not and
> > > > > > can't be changed at runtime. So I'd prefer to stick to "connector
> > > > > > level" plugins in this KIP. Let me now if you have use cases, I'm open
> > > > > > to reconsider this choice.
> > > > > > I'll add that in the rejected alternatives section for now
> > > > > >
> > > > > > 2. I remembered seeing issues in the past with multiple plugin.path
> > > > > > entries but I tried today and I was able to mix and match plugins from
> > > > > > different paths. So my bad for getting confused.
> > > > > > Then I agree, it makes more sense to group them by plugin type.
> > > > > >
> > > > > > 3. Yes this should be covered in KIP-802:
> > > > > >
> > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > > > >
> > > > > > 4. No particular reason. We can support both formats like today. I've
> > > > > > updated the KIP
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > >
> > > > > > > Hi Mickael,
> > > > > > >
> > > > > > > I think the increase in scope here is great and the added value
> > > > certainly
> > > > > > > justifies the proposed changes. I have some thoughts but overall I
> > > > like
> > > > > > the
> > > > > > > direction this is going in now.
> > > > > > >
> > > > > > > 1. The new /plugins endpoint is described as containing "all plugins
> > > > that
> > > > > > > are Connectors, Transformations, Converters, HeaderConverters and
> > > > > > > Predicates". So essentially, it looks like we want to expose all
> > > > plugins
> > > > > > > that are configured on a per-connector basis, but exclude plugins
> > > > that
> > > > > > are
> > > > > > > configured on a per-worker basis (such as config providers and REST
> > > > > > > extensions). Do you think it may be valuable to expose information on
> > > > > > > worker-level plugins as well?
> > > > > > >
> > > > > > > 2. The description for the new /plugins endpoint also states that
> > > > > > "Plugins
> > > > > > > will be grouped by plugin.path. This will make it clear to users
> > > > what's
> > > > > > > available to use as it's not possible to use a Connector from one
> > > > path
> > > > > > with
> > > > > > > Transformations from another.". Is this true? I thought that
> > > > Connect's
> > > > > > > classloading made it possible to package
> > > > > > > converters/transformations/predicates completely independently from
> > > > each
> > > > > > > other, and to reference them from also-independently-packaged
> > > > connectors.
> > > > > > > If it turns out that this is the case, could we consider
> > > > restructuring
> > > > > > the
> > > > > > > response to be grouped by plugin type instead of by classloader?
> > > > There's
> > > > > > > also the ungrouped format proposed in KIP-494 (
> > > > > > >
> > > > > >
> > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > > > )
> > > > > > > which we might consider as well.
> > > > > > >
> > > > > > > 3. I think this can be left for a follow-up KIP if necessary, but I'm
> > > > > > > curious about your thoughts on adding new validate methods to all
> > > > > > > connector-level plugins that can be used similarly to how the
> > > > existing
> > > > > > > Connector::validate method (
> > > > > > >
> > > > > >
> > > > https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > > > )
> > > > > > > is used. This would allow for plugins to perform validation that's
> > > > more
> > > > > > > sophisticated than what the ConfigDef is capable of, such as
> > > > validating
> > > > > > > combinations of properties like a hostname and credentials for
> > > > reaching
> > > > > > it.
> > > > > > > I know that at least Confluent's Avro, protobuf, and JSON schema
> > > > > > converters
> > > > > > > would benefit from this kind of feature. It's a little tangential to
> > > > this
> > > > > > > KIP (which at the moment is about discovering plugins and their
> > > > > > > configuration surfaces, as opposed to validating them), but I
> > > > figured I'd
> > > > > > > ask since we're going to be expanding the Converter interface and it
> > > > may
> > > > > > be
> > > > > > > useful to tackle this while we're in the neighborhood.
> > > > > > >
> > > > > > > 4. The description for the new /plugins/<type>/<name>/configdef
> > > > endpoint
> > > > > > > states that "Name must be the fully qualified class name of the
> > > > plugin".
> > > > > > > Any reason not to also support aliases (e.g.,
> > > > "FileStreamSinkConnector"
> > > > > > or
> > > > > > > "FileStreamSink" instead of
> > > > > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <
> > > > > > mickael.maison@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks all for the feedback!
> > > > > > > >
> > > > > > > > Chris,
> > > > > > > > I agree that fixing the current endpoint helps a lot. Thanks for
> > > > > > > > raising these JIRAs and submitting a PR!
> > > > > > > > However thinking about the issue further, I decided to expand the
> > > > > > > > scope of the KIP to cover all user-visible plugins.
> > > > > > > > In practice, users want to know about all available plugins not
> > > > only
> > > > > > > > connectors. This includes transformations, converters,
> > > > > > > > header_converters and predicates. As we also want to retrieve
> > > > > > > > configdef for these too, I think it makes sense to introduce a new
> > > > > > > > endpoint to do so. Alongside we obviously need a new endpoint for
> > > > > > > > listing all plugins.
> > > > > > > >
> > > > > > > > Gunnar,
> > > > > > > > I took a look at exposing valid values via the API. I think the
> > > > issue
> > > > > > > > is that Validators don't expose a way to retrieve valid values.
> > > > > > > > Changing validators will have an impact on all components so I'd
> > > > > > > > prefer to address this requirement in a separate KIP. I agree this
> > > > > > > > would be an interesting improvement and I'd happy to write a KIP
> > > > for
> > > > > > > > it too.
> > > > > > > >
> > > > > > > >
> > > > > > > > I have updated the KIP accordingly. Let me know if you have further
> > > > > > > > feedback.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mickael
> > > > > > > >
> > > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > > > > > > > <gu...@googlemail.com.invalid> wrote:
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I'm +1 for adding a GET endpoint for obtaining config
> > > > definitions. It
> > > > > > > > > always felt odd to me that one has to issue a PUT for that
> > > > purpose.
> > > > > > If
> > > > > > > > > nothing else, it'd be better in terms of discoverability of the
> > > > KC
> > > > > > REST
> > > > > > > > API.
> > > > > > > > >
> > > > > > > > > One additional feature request I'd have is to expose the valid
> > > > enum
> > > > > > > > > constants for enum-typed options. That'll help to display the
> > > > values
> > > > > > in a
> > > > > > > > > drop-down or via radio buttons in a UI, give us tab completion in
> > > > > > kcctl,
> > > > > > > > > etc.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > >
> > > > > > > > > --Gunnar
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > > > > > > > > <ch...@confluent.io.invalid>:
> > > > > > > > >
> > > > > > > > > > Hi Viktor,
> > > > > > > > > >
> > > > > > > > > > It sounds like there are three major points here in favor of a
> > > > new
> > > > > > GET
> > > > > > > > > > endpoint for connector config defs.
> > > > > > > > > >
> > > > > > > > > > 1. You cannot issue a blank ("dummy") request for sink
> > > > connectors
> > > > > > > > because a
> > > > > > > > > > topic list/topic regex has to be supplied (otherwise the PUT
> > > > > > endpoint
> > > > > > > > > > returns a 500 response)
> > > > > > > > > > 2. A dummy request still triggers custom validations by the
> > > > > > connector,
> > > > > > > > > > which may be best to avoid if we know for sure that the config
> > > > > > isn't
> > > > > > > > worth
> > > > > > > > > > validating yet
> > > > > > > > > > 3. It's more ergonomic and intuitive to be able to issue a GET
> > > > > > request
> > > > > > > > > > without having to give a dummy connector config
> > > > > > > > > >
> > > > > > > > > > With regards to 1, this is actually a bug in Connect (
> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix
> > > > > > already
> > > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > > https://github.com/apache/kafka/pull/11369). I think it'd be
> > > > > > better to
> > > > > > > > > > focus on fixing this bug in general instead of implementing a
> > > > new
> > > > > > REST
> > > > > > > > > > endpoint in order to allow people to work around it.
> > > > > > > > > >
> > > > > > > > > > With regards to 2, this is technically possible but I'm unsure
> > > > > > it'd be
> > > > > > > > too
> > > > > > > > > > common out in the wild given that most validations that could
> > > > be
> > > > > > > > expensive
> > > > > > > > > > would involve things like connecting to a database, checking
> > > > if a
> > > > > > cloud
> > > > > > > > > > storage bucket exists, etc., none of which are possible without
> > > > > > some
> > > > > > > > > > configuration properties from the user (db hostname, bucket
> > > > name,
> > > > > > > > etc.).
> > > > > > > > > >
> > > > > > > > > > With regards to 3, I do agree that it'd be easier for people
> > > > > > designing
> > > > > > > > UIs
> > > > > > > > > > to have a GET API to work against. I'm just not sure it's
> > > > worth the
> > > > > > > > > > additional implementation, testing, and maintenance burden. If
> > > > it
> > > > > > were
> > > > > > > > > > possible to issue a PUT request without unexpected 500s for
> > > > invalid
> > > > > > > > > > configs, would that suffice? AFAICT it'd basically be as
> > > > simple as
> > > > > > > > issuing
> > > > > > > > > > a PUT request with a dummy body consisting of nothing except
> > > > the
> > > > > > > > connector
> > > > > > > > > > class (which at this point we might even make unnecessary and
> > > > just
> > > > > > > > > > automatically replace with the connector class from the URL)
> > > > and
> > > > > > then
> > > > > > > > > > filtering the response to just grab the "definition" field of
> > > > each
> > > > > > > > element
> > > > > > > > > > in the "configs" array in the response.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > >
> > > > > > > > > > Chris
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Folks,
> > > > > > > > > > >
> > > > > > > > > > > I too think this would be a very useful feature. Some of our
> > > > > > > > management
> > > > > > > > > > > applications would provide a wizard for creating connectors.
> > > > In
> > > > > > this
> > > > > > > > > > > scenario the user basically would fill out a sample
> > > > configuration
> > > > > > > > > > generated
> > > > > > > > > > > by the UI which would send it back to Connect for validation
> > > > and
> > > > > > > > > > eventually
> > > > > > > > > > > create a new connector. The first part of this workflow can
> > > > be
> > > > > > > > enhanced
> > > > > > > > > > if
> > > > > > > > > > > we had an API that can return the configuration definition
> > > > of the
> > > > > > > > given
> > > > > > > > > > > type of connector as the UI application would be able to
> > > > > > generate a
> > > > > > > > > > sample
> > > > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > > The connector-plugins/{connectorType}/config/validate API
> > > > > > essentially
> > > > > > > > > > works
> > > > > > > > > > > and returns the data that we need, however it is a HTTP PUT
> > > > API
> > > > > > that
> > > > > > > > is a
> > > > > > > > > > > bit unintuitive for a fetch-like functionality and also
> > > > > > functionally
> > > > > > > > > > > different as it validates the given (dummy) request. In case
> > > > of
> > > > > > sink
> > > > > > > > > > > connectors one would need to also provide a topic name.
> > > > > > > > > > >
> > > > > > > > > > > A suggestion for the KIP: I think it can be useful to return
> > > > the
> > > > > > > > config
> > > > > > > > > > > groups and the connector class' name similarly to the
> > > > validate
> > > > > > API
> > > > > > > > just
> > > > > > > > > > in
> > > > > > > > > > > case any frontend needs them (and also the response would be
> > > > more
> > > > > > > > like
> > > > > > > > > > the
> > > > > > > > > > > validate API but simpler).
> > > > > > > > > > >
> > > > > > > > > > > Viktor
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > > ryannedolan@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I think it'd be worth adding a GET version, fwiw. Could be
> > > > the
> > > > > > same
> > > > > > > > > > > handler
> > > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > > >
> > > > > > > > > > > > > You're right, you can achieve the same functionality
> > > > using
> > > > > > the
> > > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > > In my mind it was only for validation once you have
> > > > build a
> > > > > > > > > > > > > configuration but when used with an empty configuration,
> > > > it
> > > > > > > > basically
> > > > > > > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think it's a bit easier to use a GET endpoint but I
> > > > don't
> > > > > > > > think it
> > > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'm wondering about the use case here. The motivation
> > > > > > section
> > > > > > > > > > states
> > > > > > > > > > > > that
> > > > > > > > > > > > > > "Connect does not provide a way to see what
> > > > configurations
> > > > > > a
> > > > > > > > > > > connector
> > > > > > > > > > > > > > requires. Instead users have to go look at the
> > > > connector
> > > > > > > > > > > documentation
> > > > > > > > > > > > or
> > > > > > > > > > > > > > in the worst case, look directly at the connector
> > > > source
> > > > > > > > code.",
> > > > > > > > > > and
> > > > > > > > > > > > that
> > > > > > > > > > > > > > with this KIP, "users will be able to discover the
> > > > required
> > > > > > > > > > > > > configurations
> > > > > > > > > > > > > > for connectors installed in a Connect cluster" and
> > > > "tools
> > > > > > will
> > > > > > > > be
> > > > > > > > > > > able
> > > > > > > > > > > > to
> > > > > > > > > > > > > > generate wizards for configuring and starting
> > > > connectors".
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > > endpoint not address these points? What will the
> > > > > > newly-proposed
> > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > allow users to do that they will not already be able
> > > > to do
> > > > > > > > with the
> > > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I've created KIP-769 to expose connector
> > > > configuration
> > > > > > > > > > definitions
> > > > > > > > > > > in
> > > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Please take a look and let me know if you have any
> > > > > > feedback.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Randall Hauch <rh...@gmail.com>]

Thanks for the KIP, Mickael. And thanks to everyone else for the good
discussion so far.

Overall this is pretty good, but I do have a few questions/comments:
1. Can we include the HTTP method/verb in each of the endpoints in the
"Public Interfaces" section?
2. The sample output for `GET /connector-plugins?connectorsOnly=false`
includes a `version` field for the connector plugin, but not for the
transform or converter plugins. Neither the `Transformation` or
`Converter` interfaces have a version nor implement the `Versioned`
interface, so should we add the `Versioned` interface to the
`Transformation` and `Converter` interfaces and add a default method
that returns "unknown"? Yes, we could do that in a separate KIP, but I
think it's appropriate to do it in this KIP for a few reasons. First,
we're already modifying the `Converter` interface in this KIP and it
would be quite straightforward to also extend the `Versioned`
interface and add a default method; then doing that for
`Transformation` would also be relatively straightforward. Second,
this KIP provides the justification for adding a version to the
`Transformation` and `Converter` interfaces.
3. In the "Public Interfaces" section, there are two inconsistencies
between the `GET /connector-plugins` and `GET
/connector-plugins/<plugin>/config` methods. Both will make it
difficult to use the second method, since the input to the second
method is not directly found in the response from the first method:
 a. The example for `GET /connector-plugins/Cast$Value/config` uses
short name, or what Connect often refers to as the alias, "Cast$Value"
for the plugin, rather than the fully-qualified name
"org.apache.kafka.connect.transforms.Cast$Value". The KIP should
specify whether it will support both the shortened name and the
fully-qualified name, or whether only one of these will be supported.
Not supporting the fully-qualified name will make it impossible to use
the response from the first method as an input to the second.
 a. The example for the list method returns the `Transformation` base
and abstract classes (e.g.,
"org.apache.kafka.connect.transforms.Cast") rather than the concrete
classes (e.g., "org.apache.kafka.connect.transforms.Cast$Value" and
"org.apache.kafka.connect.transforms.Cast$Key"), while the `GET
/connector-plugins/<plugin>/config` will require the concrete class.
4. The "Public Interfaces" section's description of the `GET
/connector-plugins/<plugin>/config` method should probably specify the
expected behavior for `Converter` implementations that don't override
the new `config()` method.
5. The "Compatibility, Deprecation, and Migration Plan" section should
describe whether older `Converter` implementations (compiled with
older versions of the Connect API without Converter extending
Configurable) will be able to be installed into newer Connect
runtimes, and what the behavior will be for the `GET
/connector-plugins/<plugin>/config` method. (Although my suggestion
for #5 will partially address the latter, we still need to talk about
it in terms of compatibility.)
6. Can we add a small section addressing the security implications of
these new APIs? KIP-507 [1] applied to APIs that existed at the time,
and this KIP should at least mention how it aligns with that
mechanism.
7. Do we really think adding `GET /worker-plugins` is necessary? While
I understand the reasoning, I'm less convinced that it's worth adding
to the public API. First, an administrator still has to modify the
worker config for all workers to take advantage of that. Second,
adding this method increases the surface area and therefore risk of
additional attack vectors, despite not really being able to use the
resulting information in any other part of the API.

Best regards,

Randall

[1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints


On Wed, Dec 1, 2021 at 7:52 AM Mickael Maison <mi...@gmail.com> wrote:
>
> Hi Chris,
>
> Thanks again for the feedback! I've updated the KIP based on our last
> discussions. I've decided to include the new endpoint for worker
> plugins.
>
> 1. Yes I agree, it's best to gate the new behavior.
> 2. Yes, it was a remnant from the original proposal. I've now removed
> the location field.
>
> Thanks,
> Mickael
>
> On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
> <ch...@confluent.io.invalid> wrote:
> >
> > Hi Mickael,
> >
> > I think that's a great idea! I especially like how we can establish the
> > expectation that any plugin type that appears in the response from the GET
> > /connector-plugins endpoint will have a corresponding GET
> > /connector-plugins/<type>/config endpoint, but (if we decide to add them in
> > the future), worker plugins won't be expected to expose this kind of
> > information and the different root path helps give a decent hint about this.
> >
> > I also like the choice to return an empty ConfigDef from Converter::config
> > instead of null.
> >
> > Two things come to mind:
> >
> > 1. We may want to gate this behind a URL query parameter (maybe something
> > like "connectorsOnly") that defaults to the old behavior in order to avoid
> > breaking existing tools such as programmatic UIs that use the endpoint
> > today to discover the connectors that can be created by the user. We can
> > even plan to change the default for that parameter to the newly-proposed
> > behavior in the next major release, which should give people enough time to
> > either adapt to the expanded response format or add the query parameter to
> > their tooling.
> >
> > 2. The existing GET /connector-plugins endpoint doesn't contain information
> > on the location of the plugin on the worker's file system. Do you think we
> > should still include this info in the new response format? Correct me if
> > I'm wrong but it seems it may have been proposed originally to help prevent
> > already-addressed bugs in Connect classloading from striking; all else
> > equal, I'd personally err on the side of leaving this info out or at least
> > reducing permitted values for it to just "classpath" or "plugin path" in
> > order to avoid leaking worker file paths into the REST API, which might
> > bother super security-conscious users.
> >
> > Cheers,
> >
> > Chris
> >
> > On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <mi...@gmail.com>
> > wrote:
> >
> > > Hi Chris,
> > >
> > > Yes to keep compatibility we want a default implementation for
> > > Converter.configs(), I've updated the KIP.
> > >
> > > Regarding worker plugins, the use case you described seems valuable.
> > > I'd prefer not mixing worker and connector plugins on the same
> > > endpoint but I agree using /plugins and /worker-plugins could be
> > > confusing.
> > >
> > > One alternative is to expose all connector-level plugins via the
> > > existing /connector-plugins endpoint. In that case, we'd need to keep
> > > the current JSON schema and not group plugins by type. As the current
> > > schema already has a type field for each entry, we'll still be able to
> > > tell them apart. Then we can have /worker-plugins and a relatively
> > > clean API. What do you think?
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > > <ch...@confluent.io.invalid> wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > I think one potential use case for exposing worker-level plugins is that
> > > it
> > > > may make it easier to determine whether a worker is set up correctly (the
> > > > current alternative requires looking through log files and can be a
> > > little
> > > > tedious), and might even make it possible to automatically identify
> > > > discrepancies within a cluster by diffing the contents of that endpoint
> > > > across each worker. But I don't think this has to be addressed by the
> > > > current KIP; the only thing that bothers me a little is that "plugins" is
> > > > generic and it may confuse people down the road if we add an endpoint for
> > > > worker-level plugins ("why is one just called 'plugins' and the other one
> > > > is 'worker-plugins'?"). Probably not worth blocking on, though.
> > > >
> > > > Agreed that the suggestion for improved validation should be made on the
> > > > KIP-802 thread.
> > > >
> > > > I also noticed that the newly-proposed config method for the Converter
> > > > interface doesn't have a default implementation, making it
> > > > backwards-incompatible. Should we add a default implementation that
> > > returns
> > > > either null or an empty ConfigDef?
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <mickael.maison@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Chris,
> > > > >
> > > > > 1. If we want to expose worker plugins, I think we should do it via a
> > > > > separate endpoint. But to be honest, I'm not even sure I see strong
> > > > > use cases for exposing them as they are either enabled or not and
> > > > > can't be changed at runtime. So I'd prefer to stick to "connector
> > > > > level" plugins in this KIP. Let me now if you have use cases, I'm open
> > > > > to reconsider this choice.
> > > > > I'll add that in the rejected alternatives section for now
> > > > >
> > > > > 2. I remembered seeing issues in the past with multiple plugin.path
> > > > > entries but I tried today and I was able to mix and match plugins from
> > > > > different paths. So my bad for getting confused.
> > > > > Then I agree, it makes more sense to group them by plugin type.
> > > > >
> > > > > 3. Yes this should be covered in KIP-802:
> > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > > >
> > > > > 4. No particular reason. We can support both formats like today. I've
> > > > > updated the KIP
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > > > <ch...@confluent.io.invalid> wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > I think the increase in scope here is great and the added value
> > > certainly
> > > > > > justifies the proposed changes. I have some thoughts but overall I
> > > like
> > > > > the
> > > > > > direction this is going in now.
> > > > > >
> > > > > > 1. The new /plugins endpoint is described as containing "all plugins
> > > that
> > > > > > are Connectors, Transformations, Converters, HeaderConverters and
> > > > > > Predicates". So essentially, it looks like we want to expose all
> > > plugins
> > > > > > that are configured on a per-connector basis, but exclude plugins
> > > that
> > > > > are
> > > > > > configured on a per-worker basis (such as config providers and REST
> > > > > > extensions). Do you think it may be valuable to expose information on
> > > > > > worker-level plugins as well?
> > > > > >
> > > > > > 2. The description for the new /plugins endpoint also states that
> > > > > "Plugins
> > > > > > will be grouped by plugin.path. This will make it clear to users
> > > what's
> > > > > > available to use as it's not possible to use a Connector from one
> > > path
> > > > > with
> > > > > > Transformations from another.". Is this true? I thought that
> > > Connect's
> > > > > > classloading made it possible to package
> > > > > > converters/transformations/predicates completely independently from
> > > each
> > > > > > other, and to reference them from also-independently-packaged
> > > connectors.
> > > > > > If it turns out that this is the case, could we consider
> > > restructuring
> > > > > the
> > > > > > response to be grouped by plugin type instead of by classloader?
> > > There's
> > > > > > also the ungrouped format proposed in KIP-494 (
> > > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > > )
> > > > > > which we might consider as well.
> > > > > >
> > > > > > 3. I think this can be left for a follow-up KIP if necessary, but I'm
> > > > > > curious about your thoughts on adding new validate methods to all
> > > > > > connector-level plugins that can be used similarly to how the
> > > existing
> > > > > > Connector::validate method (
> > > > > >
> > > > >
> > > https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > > )
> > > > > > is used. This would allow for plugins to perform validation that's
> > > more
> > > > > > sophisticated than what the ConfigDef is capable of, such as
> > > validating
> > > > > > combinations of properties like a hostname and credentials for
> > > reaching
> > > > > it.
> > > > > > I know that at least Confluent's Avro, protobuf, and JSON schema
> > > > > converters
> > > > > > would benefit from this kind of feature. It's a little tangential to
> > > this
> > > > > > KIP (which at the moment is about discovering plugins and their
> > > > > > configuration surfaces, as opposed to validating them), but I
> > > figured I'd
> > > > > > ask since we're going to be expanding the Converter interface and it
> > > may
> > > > > be
> > > > > > useful to tackle this while we're in the neighborhood.
> > > > > >
> > > > > > 4. The description for the new /plugins/<type>/<name>/configdef
> > > endpoint
> > > > > > states that "Name must be the fully qualified class name of the
> > > plugin".
> > > > > > Any reason not to also support aliases (e.g.,
> > > "FileStreamSinkConnector"
> > > > > or
> > > > > > "FileStreamSink" instead of
> > > > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <
> > > > > mickael.maison@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks all for the feedback!
> > > > > > >
> > > > > > > Chris,
> > > > > > > I agree that fixing the current endpoint helps a lot. Thanks for
> > > > > > > raising these JIRAs and submitting a PR!
> > > > > > > However thinking about the issue further, I decided to expand the
> > > > > > > scope of the KIP to cover all user-visible plugins.
> > > > > > > In practice, users want to know about all available plugins not
> > > only
> > > > > > > connectors. This includes transformations, converters,
> > > > > > > header_converters and predicates. As we also want to retrieve
> > > > > > > configdef for these too, I think it makes sense to introduce a new
> > > > > > > endpoint to do so. Alongside we obviously need a new endpoint for
> > > > > > > listing all plugins.
> > > > > > >
> > > > > > > Gunnar,
> > > > > > > I took a look at exposing valid values via the API. I think the
> > > issue
> > > > > > > is that Validators don't expose a way to retrieve valid values.
> > > > > > > Changing validators will have an impact on all components so I'd
> > > > > > > prefer to address this requirement in a separate KIP. I agree this
> > > > > > > would be an interesting improvement and I'd happy to write a KIP
> > > for
> > > > > > > it too.
> > > > > > >
> > > > > > >
> > > > > > > I have updated the KIP accordingly. Let me know if you have further
> > > > > > > feedback.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mickael
> > > > > > >
> > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > > > > > > <gu...@googlemail.com.invalid> wrote:
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I'm +1 for adding a GET endpoint for obtaining config
> > > definitions. It
> > > > > > > > always felt odd to me that one has to issue a PUT for that
> > > purpose.
> > > > > If
> > > > > > > > nothing else, it'd be better in terms of discoverability of the
> > > KC
> > > > > REST
> > > > > > > API.
> > > > > > > >
> > > > > > > > One additional feature request I'd have is to expose the valid
> > > enum
> > > > > > > > constants for enum-typed options. That'll help to display the
> > > values
> > > > > in a
> > > > > > > > drop-down or via radio buttons in a UI, give us tab completion in
> > > > > kcctl,
> > > > > > > > etc.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > --Gunnar
> > > > > > > >
> > > > > > > >
> > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > > > > > > > <ch...@confluent.io.invalid>:
> > > > > > > >
> > > > > > > > > Hi Viktor,
> > > > > > > > >
> > > > > > > > > It sounds like there are three major points here in favor of a
> > > new
> > > > > GET
> > > > > > > > > endpoint for connector config defs.
> > > > > > > > >
> > > > > > > > > 1. You cannot issue a blank ("dummy") request for sink
> > > connectors
> > > > > > > because a
> > > > > > > > > topic list/topic regex has to be supplied (otherwise the PUT
> > > > > endpoint
> > > > > > > > > returns a 500 response)
> > > > > > > > > 2. A dummy request still triggers custom validations by the
> > > > > connector,
> > > > > > > > > which may be best to avoid if we know for sure that the config
> > > > > isn't
> > > > > > > worth
> > > > > > > > > validating yet
> > > > > > > > > 3. It's more ergonomic and intuitive to be able to issue a GET
> > > > > request
> > > > > > > > > without having to give a dummy connector config
> > > > > > > > >
> > > > > > > > > With regards to 1, this is actually a bug in Connect (
> > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix
> > > > > already
> > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > https://github.com/apache/kafka/pull/11369). I think it'd be
> > > > > better to
> > > > > > > > > focus on fixing this bug in general instead of implementing a
> > > new
> > > > > REST
> > > > > > > > > endpoint in order to allow people to work around it.
> > > > > > > > >
> > > > > > > > > With regards to 2, this is technically possible but I'm unsure
> > > > > it'd be
> > > > > > > too
> > > > > > > > > common out in the wild given that most validations that could
> > > be
> > > > > > > expensive
> > > > > > > > > would involve things like connecting to a database, checking
> > > if a
> > > > > cloud
> > > > > > > > > storage bucket exists, etc., none of which are possible without
> > > > > some
> > > > > > > > > configuration properties from the user (db hostname, bucket
> > > name,
> > > > > > > etc.).
> > > > > > > > >
> > > > > > > > > With regards to 3, I do agree that it'd be easier for people
> > > > > designing
> > > > > > > UIs
> > > > > > > > > to have a GET API to work against. I'm just not sure it's
> > > worth the
> > > > > > > > > additional implementation, testing, and maintenance burden. If
> > > it
> > > > > were
> > > > > > > > > possible to issue a PUT request without unexpected 500s for
> > > invalid
> > > > > > > > > configs, would that suffice? AFAICT it'd basically be as
> > > simple as
> > > > > > > issuing
> > > > > > > > > a PUT request with a dummy body consisting of nothing except
> > > the
> > > > > > > connector
> > > > > > > > > class (which at this point we might even make unnecessary and
> > > just
> > > > > > > > > automatically replace with the connector class from the URL)
> > > and
> > > > > then
> > > > > > > > > filtering the response to just grab the "definition" field of
> > > each
> > > > > > > element
> > > > > > > > > in the "configs" array in the response.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Folks,
> > > > > > > > > >
> > > > > > > > > > I too think this would be a very useful feature. Some of our
> > > > > > > management
> > > > > > > > > > applications would provide a wizard for creating connectors.
> > > In
> > > > > this
> > > > > > > > > > scenario the user basically would fill out a sample
> > > configuration
> > > > > > > > > generated
> > > > > > > > > > by the UI which would send it back to Connect for validation
> > > and
> > > > > > > > > eventually
> > > > > > > > > > create a new connector. The first part of this workflow can
> > > be
> > > > > > > enhanced
> > > > > > > > > if
> > > > > > > > > > we had an API that can return the configuration definition
> > > of the
> > > > > > > given
> > > > > > > > > > type of connector as the UI application would be able to
> > > > > generate a
> > > > > > > > > sample
> > > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > The connector-plugins/{connectorType}/config/validate API
> > > > > essentially
> > > > > > > > > works
> > > > > > > > > > and returns the data that we need, however it is a HTTP PUT
> > > API
> > > > > that
> > > > > > > is a
> > > > > > > > > > bit unintuitive for a fetch-like functionality and also
> > > > > functionally
> > > > > > > > > > different as it validates the given (dummy) request. In case
> > > of
> > > > > sink
> > > > > > > > > > connectors one would need to also provide a topic name.
> > > > > > > > > >
> > > > > > > > > > A suggestion for the KIP: I think it can be useful to return
> > > the
> > > > > > > config
> > > > > > > > > > groups and the connector class' name similarly to the
> > > validate
> > > > > API
> > > > > > > just
> > > > > > > > > in
> > > > > > > > > > case any frontend needs them (and also the response would be
> > > more
> > > > > > > like
> > > > > > > > > the
> > > > > > > > > > validate API but simpler).
> > > > > > > > > >
> > > > > > > > > > Viktor
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > ryannedolan@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I think it'd be worth adding a GET version, fwiw. Could be
> > > the
> > > > > same
> > > > > > > > > > handler
> > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > > mickael.maison@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > >
> > > > > > > > > > > > You're right, you can achieve the same functionality
> > > using
> > > > > the
> > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > In my mind it was only for validation once you have
> > > build a
> > > > > > > > > > > > configuration but when used with an empty configuration,
> > > it
> > > > > > > basically
> > > > > > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > > > > > >
> > > > > > > > > > > > I think it's a bit easier to use a GET endpoint but I
> > > don't
> > > > > > > think it
> > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm wondering about the use case here. The motivation
> > > > > section
> > > > > > > > > states
> > > > > > > > > > > that
> > > > > > > > > > > > > "Connect does not provide a way to see what
> > > configurations
> > > > > a
> > > > > > > > > > connector
> > > > > > > > > > > > > requires. Instead users have to go look at the
> > > connector
> > > > > > > > > > documentation
> > > > > > > > > > > or
> > > > > > > > > > > > > in the worst case, look directly at the connector
> > > source
> > > > > > > code.",
> > > > > > > > > and
> > > > > > > > > > > that
> > > > > > > > > > > > > with this KIP, "users will be able to discover the
> > > required
> > > > > > > > > > > > configurations
> > > > > > > > > > > > > for connectors installed in a Connect cluster" and
> > > "tools
> > > > > will
> > > > > > > be
> > > > > > > > > > able
> > > > > > > > > > > to
> > > > > > > > > > > > > generate wizards for configuring and starting
> > > connectors".
> > > > > > > > > > > > >
> > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > endpoint not address these points? What will the
> > > > > newly-proposed
> > > > > > > > > > > endpoint
> > > > > > > > > > > > > allow users to do that they will not already be able
> > > to do
> > > > > > > with the
> > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Chris
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I've created KIP-769 to expose connector
> > > configuration
> > > > > > > > > definitions
> > > > > > > > > > in
> > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please take a look and let me know if you have any
> > > > > feedback.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Chris Egerton <ch...@confluent.io.INVALID>]

Hi Mickael,

Looks good to me! +1 non-binding when the vote thread opens :)

Cheers,

Chris

On Wed, Dec 1, 2021 at 8:53 AM Mickael Maison <mi...@gmail.com>
wrote:

> Hi Chris,
>
> Thanks again for the feedback! I've updated the KIP based on our last
> discussions. I've decided to include the new endpoint for worker
> plugins.
>
> 1. Yes I agree, it's best to gate the new behavior.
> 2. Yes, it was a remnant from the original proposal. I've now removed
> the location field.
>
> Thanks,
> Mickael
>
> On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
> <ch...@confluent.io.invalid> wrote:
> >
> > Hi Mickael,
> >
> > I think that's a great idea! I especially like how we can establish the
> > expectation that any plugin type that appears in the response from the
> GET
> > /connector-plugins endpoint will have a corresponding GET
> > /connector-plugins/<type>/config endpoint, but (if we decide to add them
> in
> > the future), worker plugins won't be expected to expose this kind of
> > information and the different root path helps give a decent hint about
> this.
> >
> > I also like the choice to return an empty ConfigDef from
> Converter::config
> > instead of null.
> >
> > Two things come to mind:
> >
> > 1. We may want to gate this behind a URL query parameter (maybe something
> > like "connectorsOnly") that defaults to the old behavior in order to
> avoid
> > breaking existing tools such as programmatic UIs that use the endpoint
> > today to discover the connectors that can be created by the user. We can
> > even plan to change the default for that parameter to the newly-proposed
> > behavior in the next major release, which should give people enough time
> to
> > either adapt to the expanded response format or add the query parameter
> to
> > their tooling.
> >
> > 2. The existing GET /connector-plugins endpoint doesn't contain
> information
> > on the location of the plugin on the worker's file system. Do you think
> we
> > should still include this info in the new response format? Correct me if
> > I'm wrong but it seems it may have been proposed originally to help
> prevent
> > already-addressed bugs in Connect classloading from striking; all else
> > equal, I'd personally err on the side of leaving this info out or at
> least
> > reducing permitted values for it to just "classpath" or "plugin path" in
> > order to avoid leaking worker file paths into the REST API, which might
> > bother super security-conscious users.
> >
> > Cheers,
> >
> > Chris
> >
> > On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <mickael.maison@gmail.com
> >
> > wrote:
> >
> > > Hi Chris,
> > >
> > > Yes to keep compatibility we want a default implementation for
> > > Converter.configs(), I've updated the KIP.
> > >
> > > Regarding worker plugins, the use case you described seems valuable.
> > > I'd prefer not mixing worker and connector plugins on the same
> > > endpoint but I agree using /plugins and /worker-plugins could be
> > > confusing.
> > >
> > > One alternative is to expose all connector-level plugins via the
> > > existing /connector-plugins endpoint. In that case, we'd need to keep
> > > the current JSON schema and not group plugins by type. As the current
> > > schema already has a type field for each entry, we'll still be able to
> > > tell them apart. Then we can have /worker-plugins and a relatively
> > > clean API. What do you think?
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > > <ch...@confluent.io.invalid> wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > I think one potential use case for exposing worker-level plugins is
> that
> > > it
> > > > may make it easier to determine whether a worker is set up correctly
> (the
> > > > current alternative requires looking through log files and can be a
> > > little
> > > > tedious), and might even make it possible to automatically identify
> > > > discrepancies within a cluster by diffing the contents of that
> endpoint
> > > > across each worker. But I don't think this has to be addressed by the
> > > > current KIP; the only thing that bothers me a little is that
> "plugins" is
> > > > generic and it may confuse people down the road if we add an
> endpoint for
> > > > worker-level plugins ("why is one just called 'plugins' and the
> other one
> > > > is 'worker-plugins'?"). Probably not worth blocking on, though.
> > > >
> > > > Agreed that the suggestion for improved validation should be made on
> the
> > > > KIP-802 thread.
> > > >
> > > > I also noticed that the newly-proposed config method for the
> Converter
> > > > interface doesn't have a default implementation, making it
> > > > backwards-incompatible. Should we add a default implementation that
> > > returns
> > > > either null or an empty ConfigDef?
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <
> mickael.maison@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Chris,
> > > > >
> > > > > 1. If we want to expose worker plugins, I think we should do it
> via a
> > > > > separate endpoint. But to be honest, I'm not even sure I see strong
> > > > > use cases for exposing them as they are either enabled or not and
> > > > > can't be changed at runtime. So I'd prefer to stick to "connector
> > > > > level" plugins in this KIP. Let me now if you have use cases, I'm
> open
> > > > > to reconsider this choice.
> > > > > I'll add that in the rejected alternatives section for now
> > > > >
> > > > > 2. I remembered seeing issues in the past with multiple plugin.path
> > > > > entries but I tried today and I was able to mix and match plugins
> from
> > > > > different paths. So my bad for getting confused.
> > > > > Then I agree, it makes more sense to group them by plugin type.
> > > > >
> > > > > 3. Yes this should be covered in KIP-802:
> > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > > >
> > > > > 4. No particular reason. We can support both formats like today.
> I've
> > > > > updated the KIP
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > > > <ch...@confluent.io.invalid> wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > I think the increase in scope here is great and the added value
> > > certainly
> > > > > > justifies the proposed changes. I have some thoughts but overall
> I
> > > like
> > > > > the
> > > > > > direction this is going in now.
> > > > > >
> > > > > > 1. The new /plugins endpoint is described as containing "all
> plugins
> > > that
> > > > > > are Connectors, Transformations, Converters, HeaderConverters and
> > > > > > Predicates". So essentially, it looks like we want to expose all
> > > plugins
> > > > > > that are configured on a per-connector basis, but exclude plugins
> > > that
> > > > > are
> > > > > > configured on a per-worker basis (such as config providers and
> REST
> > > > > > extensions). Do you think it may be valuable to expose
> information on
> > > > > > worker-level plugins as well?
> > > > > >
> > > > > > 2. The description for the new /plugins endpoint also states that
> > > > > "Plugins
> > > > > > will be grouped by plugin.path. This will make it clear to users
> > > what's
> > > > > > available to use as it's not possible to use a Connector from one
> > > path
> > > > > with
> > > > > > Transformations from another.". Is this true? I thought that
> > > Connect's
> > > > > > classloading made it possible to package
> > > > > > converters/transformations/predicates completely independently
> from
> > > each
> > > > > > other, and to reference them from also-independently-packaged
> > > connectors.
> > > > > > If it turns out that this is the case, could we consider
> > > restructuring
> > > > > the
> > > > > > response to be grouped by plugin type instead of by classloader?
> > > There's
> > > > > > also the ungrouped format proposed in KIP-494 (
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > > )
> > > > > > which we might consider as well.
> > > > > >
> > > > > > 3. I think this can be left for a follow-up KIP if necessary,
> but I'm
> > > > > > curious about your thoughts on adding new validate methods to all
> > > > > > connector-level plugins that can be used similarly to how the
> > > existing
> > > > > > Connector::validate method (
> > > > > >
> > > > >
> > >
> https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > > )
> > > > > > is used. This would allow for plugins to perform validation
> that's
> > > more
> > > > > > sophisticated than what the ConfigDef is capable of, such as
> > > validating
> > > > > > combinations of properties like a hostname and credentials for
> > > reaching
> > > > > it.
> > > > > > I know that at least Confluent's Avro, protobuf, and JSON schema
> > > > > converters
> > > > > > would benefit from this kind of feature. It's a little
> tangential to
> > > this
> > > > > > KIP (which at the moment is about discovering plugins and their
> > > > > > configuration surfaces, as opposed to validating them), but I
> > > figured I'd
> > > > > > ask since we're going to be expanding the Converter interface
> and it
> > > may
> > > > > be
> > > > > > useful to tackle this while we're in the neighborhood.
> > > > > >
> > > > > > 4. The description for the new /plugins/<type>/<name>/configdef
> > > endpoint
> > > > > > states that "Name must be the fully qualified class name of the
> > > plugin".
> > > > > > Any reason not to also support aliases (e.g.,
> > > "FileStreamSinkConnector"
> > > > > or
> > > > > > "FileStreamSink" instead of
> > > > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <
> > > > > mickael.maison@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks all for the feedback!
> > > > > > >
> > > > > > > Chris,
> > > > > > > I agree that fixing the current endpoint helps a lot. Thanks
> for
> > > > > > > raising these JIRAs and submitting a PR!
> > > > > > > However thinking about the issue further, I decided to expand
> the
> > > > > > > scope of the KIP to cover all user-visible plugins.
> > > > > > > In practice, users want to know about all available plugins not
> > > only
> > > > > > > connectors. This includes transformations, converters,
> > > > > > > header_converters and predicates. As we also want to retrieve
> > > > > > > configdef for these too, I think it makes sense to introduce a
> new
> > > > > > > endpoint to do so. Alongside we obviously need a new endpoint
> for
> > > > > > > listing all plugins.
> > > > > > >
> > > > > > > Gunnar,
> > > > > > > I took a look at exposing valid values via the API. I think the
> > > issue
> > > > > > > is that Validators don't expose a way to retrieve valid values.
> > > > > > > Changing validators will have an impact on all components so
> I'd
> > > > > > > prefer to address this requirement in a separate KIP. I agree
> this
> > > > > > > would be an interesting improvement and I'd happy to write a
> KIP
> > > for
> > > > > > > it too.
> > > > > > >
> > > > > > >
> > > > > > > I have updated the KIP accordingly. Let me know if you have
> further
> > > > > > > feedback.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mickael
> > > > > > >
> > > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > > > > > > <gu...@googlemail.com.invalid> wrote:
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I'm +1 for adding a GET endpoint for obtaining config
> > > definitions. It
> > > > > > > > always felt odd to me that one has to issue a PUT for that
> > > purpose.
> > > > > If
> > > > > > > > nothing else, it'd be better in terms of discoverability of
> the
> > > KC
> > > > > REST
> > > > > > > API.
> > > > > > > >
> > > > > > > > One additional feature request I'd have is to expose the
> valid
> > > enum
> > > > > > > > constants for enum-typed options. That'll help to display the
> > > values
> > > > > in a
> > > > > > > > drop-down or via radio buttons in a UI, give us tab
> completion in
> > > > > kcctl,
> > > > > > > > etc.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > --Gunnar
> > > > > > > >
> > > > > > > >
> > > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > > > > > > > <ch...@confluent.io.invalid>:
> > > > > > > >
> > > > > > > > > Hi Viktor,
> > > > > > > > >
> > > > > > > > > It sounds like there are three major points here in favor
> of a
> > > new
> > > > > GET
> > > > > > > > > endpoint for connector config defs.
> > > > > > > > >
> > > > > > > > > 1. You cannot issue a blank ("dummy") request for sink
> > > connectors
> > > > > > > because a
> > > > > > > > > topic list/topic regex has to be supplied (otherwise the
> PUT
> > > > > endpoint
> > > > > > > > > returns a 500 response)
> > > > > > > > > 2. A dummy request still triggers custom validations by the
> > > > > connector,
> > > > > > > > > which may be best to avoid if we know for sure that the
> config
> > > > > isn't
> > > > > > > worth
> > > > > > > > > validating yet
> > > > > > > > > 3. It's more ergonomic and intuitive to be able to issue a
> GET
> > > > > request
> > > > > > > > > without having to give a dummy connector config
> > > > > > > > >
> > > > > > > > > With regards to 1, this is actually a bug in Connect (
> > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a
> fix
> > > > > already
> > > > > > > > > implemented and awaiting committer review (
> > > > > > > > > https://github.com/apache/kafka/pull/11369). I think it'd
> be
> > > > > better to
> > > > > > > > > focus on fixing this bug in general instead of
> implementing a
> > > new
> > > > > REST
> > > > > > > > > endpoint in order to allow people to work around it.
> > > > > > > > >
> > > > > > > > > With regards to 2, this is technically possible but I'm
> unsure
> > > > > it'd be
> > > > > > > too
> > > > > > > > > common out in the wild given that most validations that
> could
> > > be
> > > > > > > expensive
> > > > > > > > > would involve things like connecting to a database,
> checking
> > > if a
> > > > > cloud
> > > > > > > > > storage bucket exists, etc., none of which are possible
> without
> > > > > some
> > > > > > > > > configuration properties from the user (db hostname, bucket
> > > name,
> > > > > > > etc.).
> > > > > > > > >
> > > > > > > > > With regards to 3, I do agree that it'd be easier for
> people
> > > > > designing
> > > > > > > UIs
> > > > > > > > > to have a GET API to work against. I'm just not sure it's
> > > worth the
> > > > > > > > > additional implementation, testing, and maintenance
> burden. If
> > > it
> > > > > were
> > > > > > > > > possible to issue a PUT request without unexpected 500s for
> > > invalid
> > > > > > > > > configs, would that suffice? AFAICT it'd basically be as
> > > simple as
> > > > > > > issuing
> > > > > > > > > a PUT request with a dummy body consisting of nothing
> except
> > > the
> > > > > > > connector
> > > > > > > > > class (which at this point we might even make unnecessary
> and
> > > just
> > > > > > > > > automatically replace with the connector class from the
> URL)
> > > and
> > > > > then
> > > > > > > > > filtering the response to just grab the "definition" field
> of
> > > each
> > > > > > > element
> > > > > > > > > in the "configs" array in the response.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Folks,
> > > > > > > > > >
> > > > > > > > > > I too think this would be a very useful feature. Some of
> our
> > > > > > > management
> > > > > > > > > > applications would provide a wizard for creating
> connectors.
> > > In
> > > > > this
> > > > > > > > > > scenario the user basically would fill out a sample
> > > configuration
> > > > > > > > > generated
> > > > > > > > > > by the UI which would send it back to Connect for
> validation
> > > and
> > > > > > > > > eventually
> > > > > > > > > > create a new connector. The first part of this workflow
> can
> > > be
> > > > > > > enhanced
> > > > > > > > > if
> > > > > > > > > > we had an API that can return the configuration
> definition
> > > of the
> > > > > > > given
> > > > > > > > > > type of connector as the UI application would be able to
> > > > > generate a
> > > > > > > > > sample
> > > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > > The connector-plugins/{connectorType}/config/validate API
> > > > > essentially
> > > > > > > > > works
> > > > > > > > > > and returns the data that we need, however it is a HTTP
> PUT
> > > API
> > > > > that
> > > > > > > is a
> > > > > > > > > > bit unintuitive for a fetch-like functionality and also
> > > > > functionally
> > > > > > > > > > different as it validates the given (dummy) request. In
> case
> > > of
> > > > > sink
> > > > > > > > > > connectors one would need to also provide a topic name.
> > > > > > > > > >
> > > > > > > > > > A suggestion for the KIP: I think it can be useful to
> return
> > > the
> > > > > > > config
> > > > > > > > > > groups and the connector class' name similarly to the
> > > validate
> > > > > API
> > > > > > > just
> > > > > > > > > in
> > > > > > > > > > case any frontend needs them (and also the response
> would be
> > > more
> > > > > > > like
> > > > > > > > > the
> > > > > > > > > > validate API but simpler).
> > > > > > > > > >
> > > > > > > > > > Viktor
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > > ryannedolan@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I think it'd be worth adding a GET version, fwiw.
> Could be
> > > the
> > > > > same
> > > > > > > > > > handler
> > > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > > mickael.maison@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Chris,
> > > > > > > > > > > >
> > > > > > > > > > > > You're right, you can achieve the same functionality
> > > using
> > > > > the
> > > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > > In my mind it was only for validation once you have
> > > build a
> > > > > > > > > > > > configuration but when used with an empty
> configuration,
> > > it
> > > > > > > basically
> > > > > > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > > > > > >
> > > > > > > > > > > > I think it's a bit easier to use a GET endpoint but I
> > > don't
> > > > > > > think it
> > > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm wondering about the use case here. The
> motivation
> > > > > section
> > > > > > > > > states
> > > > > > > > > > > that
> > > > > > > > > > > > > "Connect does not provide a way to see what
> > > configurations
> > > > > a
> > > > > > > > > > connector
> > > > > > > > > > > > > requires. Instead users have to go look at the
> > > connector
> > > > > > > > > > documentation
> > > > > > > > > > > or
> > > > > > > > > > > > > in the worst case, look directly at the connector
> > > source
> > > > > > > code.",
> > > > > > > > > and
> > > > > > > > > > > that
> > > > > > > > > > > > > with this KIP, "users will be able to discover the
> > > required
> > > > > > > > > > > > configurations
> > > > > > > > > > > > > for connectors installed in a Connect cluster" and
> > > "tools
> > > > > will
> > > > > > > be
> > > > > > > > > > able
> > > > > > > > > > > to
> > > > > > > > > > > > > generate wizards for configuring and starting
> > > connectors".
> > > > > > > > > > > > >
> > > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > > endpoint not address these points? What will the
> > > > > newly-proposed
> > > > > > > > > > > endpoint
> > > > > > > > > > > > > allow users to do that they will not already be
> able
> > > to do
> > > > > > > with the
> > > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Chris
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I've created KIP-769 to expose connector
> > > configuration
> > > > > > > > > definitions
> > > > > > > > > > in
> > > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please take a look and let me know if you have
> any
> > > > > feedback.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
>

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Mickael Maison <mi...@gmail.com>]

Hi Chris,

Thanks again for the feedback! I've updated the KIP based on our last
discussions. I've decided to include the new endpoint for worker
plugins.

1. Yes I agree, it's best to gate the new behavior.
2. Yes, it was a remnant from the original proposal. I've now removed
the location field.

Thanks,
Mickael

On Tue, Nov 30, 2021 at 3:22 AM Chris Egerton
<ch...@confluent.io.invalid> wrote:
>
> Hi Mickael,
>
> I think that's a great idea! I especially like how we can establish the
> expectation that any plugin type that appears in the response from the GET
> /connector-plugins endpoint will have a corresponding GET
> /connector-plugins/<type>/config endpoint, but (if we decide to add them in
> the future), worker plugins won't be expected to expose this kind of
> information and the different root path helps give a decent hint about this.
>
> I also like the choice to return an empty ConfigDef from Converter::config
> instead of null.
>
> Two things come to mind:
>
> 1. We may want to gate this behind a URL query parameter (maybe something
> like "connectorsOnly") that defaults to the old behavior in order to avoid
> breaking existing tools such as programmatic UIs that use the endpoint
> today to discover the connectors that can be created by the user. We can
> even plan to change the default for that parameter to the newly-proposed
> behavior in the next major release, which should give people enough time to
> either adapt to the expanded response format or add the query parameter to
> their tooling.
>
> 2. The existing GET /connector-plugins endpoint doesn't contain information
> on the location of the plugin on the worker's file system. Do you think we
> should still include this info in the new response format? Correct me if
> I'm wrong but it seems it may have been proposed originally to help prevent
> already-addressed bugs in Connect classloading from striking; all else
> equal, I'd personally err on the side of leaving this info out or at least
> reducing permitted values for it to just "classpath" or "plugin path" in
> order to avoid leaking worker file paths into the REST API, which might
> bother super security-conscious users.
>
> Cheers,
>
> Chris
>
> On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Hi Chris,
> >
> > Yes to keep compatibility we want a default implementation for
> > Converter.configs(), I've updated the KIP.
> >
> > Regarding worker plugins, the use case you described seems valuable.
> > I'd prefer not mixing worker and connector plugins on the same
> > endpoint but I agree using /plugins and /worker-plugins could be
> > confusing.
> >
> > One alternative is to expose all connector-level plugins via the
> > existing /connector-plugins endpoint. In that case, we'd need to keep
> > the current JSON schema and not group plugins by type. As the current
> > schema already has a type field for each entry, we'll still be able to
> > tell them apart. Then we can have /worker-plugins and a relatively
> > clean API. What do you think?
> >
> > Thanks,
> > Mickael
> >
> > On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> > <ch...@confluent.io.invalid> wrote:
> > >
> > > Hi Mickael,
> > >
> > > I think one potential use case for exposing worker-level plugins is that
> > it
> > > may make it easier to determine whether a worker is set up correctly (the
> > > current alternative requires looking through log files and can be a
> > little
> > > tedious), and might even make it possible to automatically identify
> > > discrepancies within a cluster by diffing the contents of that endpoint
> > > across each worker. But I don't think this has to be addressed by the
> > > current KIP; the only thing that bothers me a little is that "plugins" is
> > > generic and it may confuse people down the road if we add an endpoint for
> > > worker-level plugins ("why is one just called 'plugins' and the other one
> > > is 'worker-plugins'?"). Probably not worth blocking on, though.
> > >
> > > Agreed that the suggestion for improved validation should be made on the
> > > KIP-802 thread.
> > >
> > > I also noticed that the newly-proposed config method for the Converter
> > > interface doesn't have a default implementation, making it
> > > backwards-incompatible. Should we add a default implementation that
> > returns
> > > either null or an empty ConfigDef?
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <mickael.maison@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Chris,
> > > >
> > > > 1. If we want to expose worker plugins, I think we should do it via a
> > > > separate endpoint. But to be honest, I'm not even sure I see strong
> > > > use cases for exposing them as they are either enabled or not and
> > > > can't be changed at runtime. So I'd prefer to stick to "connector
> > > > level" plugins in this KIP. Let me now if you have use cases, I'm open
> > > > to reconsider this choice.
> > > > I'll add that in the rejected alternatives section for now
> > > >
> > > > 2. I remembered seeing issues in the past with multiple plugin.path
> > > > entries but I tried today and I was able to mix and match plugins from
> > > > different paths. So my bad for getting confused.
> > > > Then I agree, it makes more sense to group them by plugin type.
> > > >
> > > > 3. Yes this should be covered in KIP-802:
> > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > > >
> > > > 4. No particular reason. We can support both formats like today. I've
> > > > updated the KIP
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > >
> > > >
> > > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > > <ch...@confluent.io.invalid> wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > I think the increase in scope here is great and the added value
> > certainly
> > > > > justifies the proposed changes. I have some thoughts but overall I
> > like
> > > > the
> > > > > direction this is going in now.
> > > > >
> > > > > 1. The new /plugins endpoint is described as containing "all plugins
> > that
> > > > > are Connectors, Transformations, Converters, HeaderConverters and
> > > > > Predicates". So essentially, it looks like we want to expose all
> > plugins
> > > > > that are configured on a per-connector basis, but exclude plugins
> > that
> > > > are
> > > > > configured on a per-worker basis (such as config providers and REST
> > > > > extensions). Do you think it may be valuable to expose information on
> > > > > worker-level plugins as well?
> > > > >
> > > > > 2. The description for the new /plugins endpoint also states that
> > > > "Plugins
> > > > > will be grouped by plugin.path. This will make it clear to users
> > what's
> > > > > available to use as it's not possible to use a Connector from one
> > path
> > > > with
> > > > > Transformations from another.". Is this true? I thought that
> > Connect's
> > > > > classloading made it possible to package
> > > > > converters/transformations/predicates completely independently from
> > each
> > > > > other, and to reference them from also-independently-packaged
> > connectors.
> > > > > If it turns out that this is the case, could we consider
> > restructuring
> > > > the
> > > > > response to be grouped by plugin type instead of by classloader?
> > There's
> > > > > also the ungrouped format proposed in KIP-494 (
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > > )
> > > > > which we might consider as well.
> > > > >
> > > > > 3. I think this can be left for a follow-up KIP if necessary, but I'm
> > > > > curious about your thoughts on adding new validate methods to all
> > > > > connector-level plugins that can be used similarly to how the
> > existing
> > > > > Connector::validate method (
> > > > >
> > > >
> > https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > > )
> > > > > is used. This would allow for plugins to perform validation that's
> > more
> > > > > sophisticated than what the ConfigDef is capable of, such as
> > validating
> > > > > combinations of properties like a hostname and credentials for
> > reaching
> > > > it.
> > > > > I know that at least Confluent's Avro, protobuf, and JSON schema
> > > > converters
> > > > > would benefit from this kind of feature. It's a little tangential to
> > this
> > > > > KIP (which at the moment is about discovering plugins and their
> > > > > configuration surfaces, as opposed to validating them), but I
> > figured I'd
> > > > > ask since we're going to be expanding the Converter interface and it
> > may
> > > > be
> > > > > useful to tackle this while we're in the neighborhood.
> > > > >
> > > > > 4. The description for the new /plugins/<type>/<name>/configdef
> > endpoint
> > > > > states that "Name must be the fully qualified class name of the
> > plugin".
> > > > > Any reason not to also support aliases (e.g.,
> > "FileStreamSinkConnector"
> > > > or
> > > > > "FileStreamSink" instead of
> > > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <
> > > > mickael.maison@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks all for the feedback!
> > > > > >
> > > > > > Chris,
> > > > > > I agree that fixing the current endpoint helps a lot. Thanks for
> > > > > > raising these JIRAs and submitting a PR!
> > > > > > However thinking about the issue further, I decided to expand the
> > > > > > scope of the KIP to cover all user-visible plugins.
> > > > > > In practice, users want to know about all available plugins not
> > only
> > > > > > connectors. This includes transformations, converters,
> > > > > > header_converters and predicates. As we also want to retrieve
> > > > > > configdef for these too, I think it makes sense to introduce a new
> > > > > > endpoint to do so. Alongside we obviously need a new endpoint for
> > > > > > listing all plugins.
> > > > > >
> > > > > > Gunnar,
> > > > > > I took a look at exposing valid values via the API. I think the
> > issue
> > > > > > is that Validators don't expose a way to retrieve valid values.
> > > > > > Changing validators will have an impact on all components so I'd
> > > > > > prefer to address this requirement in a separate KIP. I agree this
> > > > > > would be an interesting improvement and I'd happy to write a KIP
> > for
> > > > > > it too.
> > > > > >
> > > > > >
> > > > > > I have updated the KIP accordingly. Let me know if you have further
> > > > > > feedback.
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > > > > > <gu...@googlemail.com.invalid> wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I'm +1 for adding a GET endpoint for obtaining config
> > definitions. It
> > > > > > > always felt odd to me that one has to issue a PUT for that
> > purpose.
> > > > If
> > > > > > > nothing else, it'd be better in terms of discoverability of the
> > KC
> > > > REST
> > > > > > API.
> > > > > > >
> > > > > > > One additional feature request I'd have is to expose the valid
> > enum
> > > > > > > constants for enum-typed options. That'll help to display the
> > values
> > > > in a
> > > > > > > drop-down or via radio buttons in a UI, give us tab completion in
> > > > kcctl,
> > > > > > > etc.
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > --Gunnar
> > > > > > >
> > > > > > >
> > > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > > > > > > <ch...@confluent.io.invalid>:
> > > > > > >
> > > > > > > > Hi Viktor,
> > > > > > > >
> > > > > > > > It sounds like there are three major points here in favor of a
> > new
> > > > GET
> > > > > > > > endpoint for connector config defs.
> > > > > > > >
> > > > > > > > 1. You cannot issue a blank ("dummy") request for sink
> > connectors
> > > > > > because a
> > > > > > > > topic list/topic regex has to be supplied (otherwise the PUT
> > > > endpoint
> > > > > > > > returns a 500 response)
> > > > > > > > 2. A dummy request still triggers custom validations by the
> > > > connector,
> > > > > > > > which may be best to avoid if we know for sure that the config
> > > > isn't
> > > > > > worth
> > > > > > > > validating yet
> > > > > > > > 3. It's more ergonomic and intuitive to be able to issue a GET
> > > > request
> > > > > > > > without having to give a dummy connector config
> > > > > > > >
> > > > > > > > With regards to 1, this is actually a bug in Connect (
> > > > > > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix
> > > > already
> > > > > > > > implemented and awaiting committer review (
> > > > > > > > https://github.com/apache/kafka/pull/11369). I think it'd be
> > > > better to
> > > > > > > > focus on fixing this bug in general instead of implementing a
> > new
> > > > REST
> > > > > > > > endpoint in order to allow people to work around it.
> > > > > > > >
> > > > > > > > With regards to 2, this is technically possible but I'm unsure
> > > > it'd be
> > > > > > too
> > > > > > > > common out in the wild given that most validations that could
> > be
> > > > > > expensive
> > > > > > > > would involve things like connecting to a database, checking
> > if a
> > > > cloud
> > > > > > > > storage bucket exists, etc., none of which are possible without
> > > > some
> > > > > > > > configuration properties from the user (db hostname, bucket
> > name,
> > > > > > etc.).
> > > > > > > >
> > > > > > > > With regards to 3, I do agree that it'd be easier for people
> > > > designing
> > > > > > UIs
> > > > > > > > to have a GET API to work against. I'm just not sure it's
> > worth the
> > > > > > > > additional implementation, testing, and maintenance burden. If
> > it
> > > > were
> > > > > > > > possible to issue a PUT request without unexpected 500s for
> > invalid
> > > > > > > > configs, would that suffice? AFAICT it'd basically be as
> > simple as
> > > > > > issuing
> > > > > > > > a PUT request with a dummy body consisting of nothing except
> > the
> > > > > > connector
> > > > > > > > class (which at this point we might even make unnecessary and
> > just
> > > > > > > > automatically replace with the connector class from the URL)
> > and
> > > > then
> > > > > > > > filtering the response to just grab the "definition" field of
> > each
> > > > > > element
> > > > > > > > in the "configs" array in the response.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > > viktorsomogyi@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Folks,
> > > > > > > > >
> > > > > > > > > I too think this would be a very useful feature. Some of our
> > > > > > management
> > > > > > > > > applications would provide a wizard for creating connectors.
> > In
> > > > this
> > > > > > > > > scenario the user basically would fill out a sample
> > configuration
> > > > > > > > generated
> > > > > > > > > by the UI which would send it back to Connect for validation
> > and
> > > > > > > > eventually
> > > > > > > > > create a new connector. The first part of this workflow can
> > be
> > > > > > enhanced
> > > > > > > > if
> > > > > > > > > we had an API that can return the configuration definition
> > of the
> > > > > > given
> > > > > > > > > type of connector as the UI application would be able to
> > > > generate a
> > > > > > > > sample
> > > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > > The connector-plugins/{connectorType}/config/validate API
> > > > essentially
> > > > > > > > works
> > > > > > > > > and returns the data that we need, however it is a HTTP PUT
> > API
> > > > that
> > > > > > is a
> > > > > > > > > bit unintuitive for a fetch-like functionality and also
> > > > functionally
> > > > > > > > > different as it validates the given (dummy) request. In case
> > of
> > > > sink
> > > > > > > > > connectors one would need to also provide a topic name.
> > > > > > > > >
> > > > > > > > > A suggestion for the KIP: I think it can be useful to return
> > the
> > > > > > config
> > > > > > > > > groups and the connector class' name similarly to the
> > validate
> > > > API
> > > > > > just
> > > > > > > > in
> > > > > > > > > case any frontend needs them (and also the response would be
> > more
> > > > > > like
> > > > > > > > the
> > > > > > > > > validate API but simpler).
> > > > > > > > >
> > > > > > > > > Viktor
> > > > > > > > >
> > > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > > ryannedolan@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I think it'd be worth adding a GET version, fwiw. Could be
> > the
> > > > same
> > > > > > > > > handler
> > > > > > > > > > with just a different spelling maybe.
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > > mickael.maison@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Chris,
> > > > > > > > > > >
> > > > > > > > > > > You're right, you can achieve the same functionality
> > using
> > > > the
> > > > > > > > > > > existing validate endpoint.
> > > > > > > > > > > In my mind it was only for validation once you have
> > build a
> > > > > > > > > > > configuration but when used with an empty configuration,
> > it
> > > > > > basically
> > > > > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > > > > >
> > > > > > > > > > > I think it's a bit easier to use a GET endpoint but I
> > don't
> > > > > > think it
> > > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Mickael,
> > > > > > > > > > > >
> > > > > > > > > > > > I'm wondering about the use case here. The motivation
> > > > section
> > > > > > > > states
> > > > > > > > > > that
> > > > > > > > > > > > "Connect does not provide a way to see what
> > configurations
> > > > a
> > > > > > > > > connector
> > > > > > > > > > > > requires. Instead users have to go look at the
> > connector
> > > > > > > > > documentation
> > > > > > > > > > or
> > > > > > > > > > > > in the worst case, look directly at the connector
> > source
> > > > > > code.",
> > > > > > > > and
> > > > > > > > > > that
> > > > > > > > > > > > with this KIP, "users will be able to discover the
> > required
> > > > > > > > > > > configurations
> > > > > > > > > > > > for connectors installed in a Connect cluster" and
> > "tools
> > > > will
> > > > > > be
> > > > > > > > > able
> > > > > > > > > > to
> > > > > > > > > > > > generate wizards for configuring and starting
> > connectors".
> > > > > > > > > > > >
> > > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > > endpoint not address these points? What will the
> > > > newly-proposed
> > > > > > > > > > endpoint
> > > > > > > > > > > > allow users to do that they will not already be able
> > to do
> > > > > > with the
> > > > > > > > > > > > existing endpoint?
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > >
> > > > > > > > > > > > Chris
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I've created KIP-769 to expose connector
> > configuration
> > > > > > > > definitions
> > > > > > > > > in
> > > > > > > > > > > > > the Connect API
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please take a look and let me know if you have any
> > > > feedback.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Chris Egerton <ch...@confluent.io.INVALID>]

Hi Mickael,

I think that's a great idea! I especially like how we can establish the
expectation that any plugin type that appears in the response from the GET
/connector-plugins endpoint will have a corresponding GET
/connector-plugins/<type>/config endpoint, but (if we decide to add them in
the future), worker plugins won't be expected to expose this kind of
information and the different root path helps give a decent hint about this.

I also like the choice to return an empty ConfigDef from Converter::config
instead of null.

Two things come to mind:

1. We may want to gate this behind a URL query parameter (maybe something
like "connectorsOnly") that defaults to the old behavior in order to avoid
breaking existing tools such as programmatic UIs that use the endpoint
today to discover the connectors that can be created by the user. We can
even plan to change the default for that parameter to the newly-proposed
behavior in the next major release, which should give people enough time to
either adapt to the expanded response format or add the query parameter to
their tooling.

2. The existing GET /connector-plugins endpoint doesn't contain information
on the location of the plugin on the worker's file system. Do you think we
should still include this info in the new response format? Correct me if
I'm wrong but it seems it may have been proposed originally to help prevent
already-addressed bugs in Connect classloading from striking; all else
equal, I'd personally err on the side of leaving this info out or at least
reducing permitted values for it to just "classpath" or "plugin path" in
order to avoid leaking worker file paths into the REST API, which might
bother super security-conscious users.

Cheers,

Chris

On Mon, Nov 29, 2021 at 5:52 AM Mickael Maison <mi...@gmail.com>
wrote:

> Hi Chris,
>
> Yes to keep compatibility we want a default implementation for
> Converter.configs(), I've updated the KIP.
>
> Regarding worker plugins, the use case you described seems valuable.
> I'd prefer not mixing worker and connector plugins on the same
> endpoint but I agree using /plugins and /worker-plugins could be
> confusing.
>
> One alternative is to expose all connector-level plugins via the
> existing /connector-plugins endpoint. In that case, we'd need to keep
> the current JSON schema and not group plugins by type. As the current
> schema already has a type field for each entry, we'll still be able to
> tell them apart. Then we can have /worker-plugins and a relatively
> clean API. What do you think?
>
> Thanks,
> Mickael
>
> On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
> <ch...@confluent.io.invalid> wrote:
> >
> > Hi Mickael,
> >
> > I think one potential use case for exposing worker-level plugins is that
> it
> > may make it easier to determine whether a worker is set up correctly (the
> > current alternative requires looking through log files and can be a
> little
> > tedious), and might even make it possible to automatically identify
> > discrepancies within a cluster by diffing the contents of that endpoint
> > across each worker. But I don't think this has to be addressed by the
> > current KIP; the only thing that bothers me a little is that "plugins" is
> > generic and it may confuse people down the road if we add an endpoint for
> > worker-level plugins ("why is one just called 'plugins' and the other one
> > is 'worker-plugins'?"). Probably not worth blocking on, though.
> >
> > Agreed that the suggestion for improved validation should be made on the
> > KIP-802 thread.
> >
> > I also noticed that the newly-proposed config method for the Converter
> > interface doesn't have a default implementation, making it
> > backwards-incompatible. Should we add a default implementation that
> returns
> > either null or an empty ConfigDef?
> >
> > Cheers,
> >
> > Chris
> >
> > On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <mickael.maison@gmail.com
> >
> > wrote:
> >
> > > Hi Chris,
> > >
> > > 1. If we want to expose worker plugins, I think we should do it via a
> > > separate endpoint. But to be honest, I'm not even sure I see strong
> > > use cases for exposing them as they are either enabled or not and
> > > can't be changed at runtime. So I'd prefer to stick to "connector
> > > level" plugins in this KIP. Let me now if you have use cases, I'm open
> > > to reconsider this choice.
> > > I'll add that in the rejected alternatives section for now
> > >
> > > 2. I remembered seeing issues in the past with multiple plugin.path
> > > entries but I tried today and I was able to mix and match plugins from
> > > different paths. So my bad for getting confused.
> > > Then I agree, it makes more sense to group them by plugin type.
> > >
> > > 3. Yes this should be covered in KIP-802:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> > >
> > > 4. No particular reason. We can support both formats like today. I've
> > > updated the KIP
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > >
> > > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > > <ch...@confluent.io.invalid> wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > I think the increase in scope here is great and the added value
> certainly
> > > > justifies the proposed changes. I have some thoughts but overall I
> like
> > > the
> > > > direction this is going in now.
> > > >
> > > > 1. The new /plugins endpoint is described as containing "all plugins
> that
> > > > are Connectors, Transformations, Converters, HeaderConverters and
> > > > Predicates". So essentially, it looks like we want to expose all
> plugins
> > > > that are configured on a per-connector basis, but exclude plugins
> that
> > > are
> > > > configured on a per-worker basis (such as config providers and REST
> > > > extensions). Do you think it may be valuable to expose information on
> > > > worker-level plugins as well?
> > > >
> > > > 2. The description for the new /plugins endpoint also states that
> > > "Plugins
> > > > will be grouped by plugin.path. This will make it clear to users
> what's
> > > > available to use as it's not possible to use a Connector from one
> path
> > > with
> > > > Transformations from another.". Is this true? I thought that
> Connect's
> > > > classloading made it possible to package
> > > > converters/transformations/predicates completely independently from
> each
> > > > other, and to reference them from also-independently-packaged
> connectors.
> > > > If it turns out that this is the case, could we consider
> restructuring
> > > the
> > > > response to be grouped by plugin type instead of by classloader?
> There's
> > > > also the ungrouped format proposed in KIP-494 (
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > > )
> > > > which we might consider as well.
> > > >
> > > > 3. I think this can be left for a follow-up KIP if necessary, but I'm
> > > > curious about your thoughts on adding new validate methods to all
> > > > connector-level plugins that can be used similarly to how the
> existing
> > > > Connector::validate method (
> > > >
> > >
> https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > > )
> > > > is used. This would allow for plugins to perform validation that's
> more
> > > > sophisticated than what the ConfigDef is capable of, such as
> validating
> > > > combinations of properties like a hostname and credentials for
> reaching
> > > it.
> > > > I know that at least Confluent's Avro, protobuf, and JSON schema
> > > converters
> > > > would benefit from this kind of feature. It's a little tangential to
> this
> > > > KIP (which at the moment is about discovering plugins and their
> > > > configuration surfaces, as opposed to validating them), but I
> figured I'd
> > > > ask since we're going to be expanding the Converter interface and it
> may
> > > be
> > > > useful to tackle this while we're in the neighborhood.
> > > >
> > > > 4. The description for the new /plugins/<type>/<name>/configdef
> endpoint
> > > > states that "Name must be the fully qualified class name of the
> plugin".
> > > > Any reason not to also support aliases (e.g.,
> "FileStreamSinkConnector"
> > > or
> > > > "FileStreamSink" instead of
> > > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <
> > > mickael.maison@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks all for the feedback!
> > > > >
> > > > > Chris,
> > > > > I agree that fixing the current endpoint helps a lot. Thanks for
> > > > > raising these JIRAs and submitting a PR!
> > > > > However thinking about the issue further, I decided to expand the
> > > > > scope of the KIP to cover all user-visible plugins.
> > > > > In practice, users want to know about all available plugins not
> only
> > > > > connectors. This includes transformations, converters,
> > > > > header_converters and predicates. As we also want to retrieve
> > > > > configdef for these too, I think it makes sense to introduce a new
> > > > > endpoint to do so. Alongside we obviously need a new endpoint for
> > > > > listing all plugins.
> > > > >
> > > > > Gunnar,
> > > > > I took a look at exposing valid values via the API. I think the
> issue
> > > > > is that Validators don't expose a way to retrieve valid values.
> > > > > Changing validators will have an impact on all components so I'd
> > > > > prefer to address this requirement in a separate KIP. I agree this
> > > > > would be an interesting improvement and I'd happy to write a KIP
> for
> > > > > it too.
> > > > >
> > > > >
> > > > > I have updated the KIP accordingly. Let me know if you have further
> > > > > feedback.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > > > > <gu...@googlemail.com.invalid> wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I'm +1 for adding a GET endpoint for obtaining config
> definitions. It
> > > > > > always felt odd to me that one has to issue a PUT for that
> purpose.
> > > If
> > > > > > nothing else, it'd be better in terms of discoverability of the
> KC
> > > REST
> > > > > API.
> > > > > >
> > > > > > One additional feature request I'd have is to expose the valid
> enum
> > > > > > constants for enum-typed options. That'll help to display the
> values
> > > in a
> > > > > > drop-down or via radio buttons in a UI, give us tab completion in
> > > kcctl,
> > > > > > etc.
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > --Gunnar
> > > > > >
> > > > > >
> > > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > > > > > <ch...@confluent.io.invalid>:
> > > > > >
> > > > > > > Hi Viktor,
> > > > > > >
> > > > > > > It sounds like there are three major points here in favor of a
> new
> > > GET
> > > > > > > endpoint for connector config defs.
> > > > > > >
> > > > > > > 1. You cannot issue a blank ("dummy") request for sink
> connectors
> > > > > because a
> > > > > > > topic list/topic regex has to be supplied (otherwise the PUT
> > > endpoint
> > > > > > > returns a 500 response)
> > > > > > > 2. A dummy request still triggers custom validations by the
> > > connector,
> > > > > > > which may be best to avoid if we know for sure that the config
> > > isn't
> > > > > worth
> > > > > > > validating yet
> > > > > > > 3. It's more ergonomic and intuitive to be able to issue a GET
> > > request
> > > > > > > without having to give a dummy connector config
> > > > > > >
> > > > > > > With regards to 1, this is actually a bug in Connect (
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix
> > > already
> > > > > > > implemented and awaiting committer review (
> > > > > > > https://github.com/apache/kafka/pull/11369). I think it'd be
> > > better to
> > > > > > > focus on fixing this bug in general instead of implementing a
> new
> > > REST
> > > > > > > endpoint in order to allow people to work around it.
> > > > > > >
> > > > > > > With regards to 2, this is technically possible but I'm unsure
> > > it'd be
> > > > > too
> > > > > > > common out in the wild given that most validations that could
> be
> > > > > expensive
> > > > > > > would involve things like connecting to a database, checking
> if a
> > > cloud
> > > > > > > storage bucket exists, etc., none of which are possible without
> > > some
> > > > > > > configuration properties from the user (db hostname, bucket
> name,
> > > > > etc.).
> > > > > > >
> > > > > > > With regards to 3, I do agree that it'd be easier for people
> > > designing
> > > > > UIs
> > > > > > > to have a GET API to work against. I'm just not sure it's
> worth the
> > > > > > > additional implementation, testing, and maintenance burden. If
> it
> > > were
> > > > > > > possible to issue a PUT request without unexpected 500s for
> invalid
> > > > > > > configs, would that suffice? AFAICT it'd basically be as
> simple as
> > > > > issuing
> > > > > > > a PUT request with a dummy body consisting of nothing except
> the
> > > > > connector
> > > > > > > class (which at this point we might even make unnecessary and
> just
> > > > > > > automatically replace with the connector class from the URL)
> and
> > > then
> > > > > > > filtering the response to just grab the "definition" field of
> each
> > > > > element
> > > > > > > in the "configs" array in the response.
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > > viktorsomogyi@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Folks,
> > > > > > > >
> > > > > > > > I too think this would be a very useful feature. Some of our
> > > > > management
> > > > > > > > applications would provide a wizard for creating connectors.
> In
> > > this
> > > > > > > > scenario the user basically would fill out a sample
> configuration
> > > > > > > generated
> > > > > > > > by the UI which would send it back to Connect for validation
> and
> > > > > > > eventually
> > > > > > > > create a new connector. The first part of this workflow can
> be
> > > > > enhanced
> > > > > > > if
> > > > > > > > we had an API that can return the configuration definition
> of the
> > > > > given
> > > > > > > > type of connector as the UI application would be able to
> > > generate a
> > > > > > > sample
> > > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > > The connector-plugins/{connectorType}/config/validate API
> > > essentially
> > > > > > > works
> > > > > > > > and returns the data that we need, however it is a HTTP PUT
> API
> > > that
> > > > > is a
> > > > > > > > bit unintuitive for a fetch-like functionality and also
> > > functionally
> > > > > > > > different as it validates the given (dummy) request. In case
> of
> > > sink
> > > > > > > > connectors one would need to also provide a topic name.
> > > > > > > >
> > > > > > > > A suggestion for the KIP: I think it can be useful to return
> the
> > > > > config
> > > > > > > > groups and the connector class' name similarly to the
> validate
> > > API
> > > > > just
> > > > > > > in
> > > > > > > > case any frontend needs them (and also the response would be
> more
> > > > > like
> > > > > > > the
> > > > > > > > validate API but simpler).
> > > > > > > >
> > > > > > > > Viktor
> > > > > > > >
> > > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > > ryannedolan@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I think it'd be worth adding a GET version, fwiw. Could be
> the
> > > same
> > > > > > > > handler
> > > > > > > > > with just a different spelling maybe.
> > > > > > > > >
> > > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > > mickael.maison@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Chris,
> > > > > > > > > >
> > > > > > > > > > You're right, you can achieve the same functionality
> using
> > > the
> > > > > > > > > > existing validate endpoint.
> > > > > > > > > > In my mind it was only for validation once you have
> build a
> > > > > > > > > > configuration but when used with an empty configuration,
> it
> > > > > basically
> > > > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > > > >
> > > > > > > > > > I think it's a bit easier to use a GET endpoint but I
> don't
> > > > > think it
> > > > > > > > > > really warrants a different endpoint.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Mickael,
> > > > > > > > > > >
> > > > > > > > > > > I'm wondering about the use case here. The motivation
> > > section
> > > > > > > states
> > > > > > > > > that
> > > > > > > > > > > "Connect does not provide a way to see what
> configurations
> > > a
> > > > > > > > connector
> > > > > > > > > > > requires. Instead users have to go look at the
> connector
> > > > > > > > documentation
> > > > > > > > > or
> > > > > > > > > > > in the worst case, look directly at the connector
> source
> > > > > code.",
> > > > > > > and
> > > > > > > > > that
> > > > > > > > > > > with this KIP, "users will be able to discover the
> required
> > > > > > > > > > configurations
> > > > > > > > > > > for connectors installed in a Connect cluster" and
> "tools
> > > will
> > > > > be
> > > > > > > > able
> > > > > > > > > to
> > > > > > > > > > > generate wizards for configuring and starting
> connectors".
> > > > > > > > > > >
> > > > > > > > > > > Does the existing "PUT
> > > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > > endpoint not address these points? What will the
> > > newly-proposed
> > > > > > > > > endpoint
> > > > > > > > > > > allow users to do that they will not already be able
> to do
> > > > > with the
> > > > > > > > > > > existing endpoint?
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > Chris
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > > mickael.maison@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > I've created KIP-769 to expose connector
> configuration
> > > > > > > definitions
> > > > > > > > in
> > > > > > > > > > > > the Connect API
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > > >
> > > > > > > > > > > > Please take a look and let me know if you have any
> > > feedback.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
>

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Mickael Maison <mi...@gmail.com>]

Hi Chris,

Yes to keep compatibility we want a default implementation for
Converter.configs(), I've updated the KIP.

Regarding worker plugins, the use case you described seems valuable.
I'd prefer not mixing worker and connector plugins on the same
endpoint but I agree using /plugins and /worker-plugins could be
confusing.

One alternative is to expose all connector-level plugins via the
existing /connector-plugins endpoint. In that case, we'd need to keep
the current JSON schema and not group plugins by type. As the current
schema already has a type field for each entry, we'll still be able to
tell them apart. Then we can have /worker-plugins and a relatively
clean API. What do you think?

Thanks,
Mickael

On Sun, Nov 28, 2021 at 8:21 PM Chris Egerton
<ch...@confluent.io.invalid> wrote:
>
> Hi Mickael,
>
> I think one potential use case for exposing worker-level plugins is that it
> may make it easier to determine whether a worker is set up correctly (the
> current alternative requires looking through log files and can be a little
> tedious), and might even make it possible to automatically identify
> discrepancies within a cluster by diffing the contents of that endpoint
> across each worker. But I don't think this has to be addressed by the
> current KIP; the only thing that bothers me a little is that "plugins" is
> generic and it may confuse people down the road if we add an endpoint for
> worker-level plugins ("why is one just called 'plugins' and the other one
> is 'worker-plugins'?"). Probably not worth blocking on, though.
>
> Agreed that the suggestion for improved validation should be made on the
> KIP-802 thread.
>
> I also noticed that the newly-proposed config method for the Converter
> interface doesn't have a default implementation, making it
> backwards-incompatible. Should we add a default implementation that returns
> either null or an empty ConfigDef?
>
> Cheers,
>
> Chris
>
> On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Hi Chris,
> >
> > 1. If we want to expose worker plugins, I think we should do it via a
> > separate endpoint. But to be honest, I'm not even sure I see strong
> > use cases for exposing them as they are either enabled or not and
> > can't be changed at runtime. So I'd prefer to stick to "connector
> > level" plugins in this KIP. Let me now if you have use cases, I'm open
> > to reconsider this choice.
> > I'll add that in the rejected alternatives section for now
> >
> > 2. I remembered seeing issues in the past with multiple plugin.path
> > entries but I tried today and I was able to mix and match plugins from
> > different paths. So my bad for getting confused.
> > Then I agree, it makes more sense to group them by plugin type.
> >
> > 3. Yes this should be covered in KIP-802:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
> >
> > 4. No particular reason. We can support both formats like today. I've
> > updated the KIP
> >
> > Thanks,
> > Mickael
> >
> >
> >
> > On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> > <ch...@confluent.io.invalid> wrote:
> > >
> > > Hi Mickael,
> > >
> > > I think the increase in scope here is great and the added value certainly
> > > justifies the proposed changes. I have some thoughts but overall I like
> > the
> > > direction this is going in now.
> > >
> > > 1. The new /plugins endpoint is described as containing "all plugins that
> > > are Connectors, Transformations, Converters, HeaderConverters and
> > > Predicates". So essentially, it looks like we want to expose all plugins
> > > that are configured on a per-connector basis, but exclude plugins that
> > are
> > > configured on a per-worker basis (such as config providers and REST
> > > extensions). Do you think it may be valuable to expose information on
> > > worker-level plugins as well?
> > >
> > > 2. The description for the new /plugins endpoint also states that
> > "Plugins
> > > will be grouped by plugin.path. This will make it clear to users what's
> > > available to use as it's not possible to use a Connector from one path
> > with
> > > Transformations from another.". Is this true? I thought that Connect's
> > > classloading made it possible to package
> > > converters/transformations/predicates completely independently from each
> > > other, and to reference them from also-independently-packaged connectors.
> > > If it turns out that this is the case, could we consider restructuring
> > the
> > > response to be grouped by plugin type instead of by classloader? There's
> > > also the ungrouped format proposed in KIP-494 (
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> > )
> > > which we might consider as well.
> > >
> > > 3. I think this can be left for a follow-up KIP if necessary, but I'm
> > > curious about your thoughts on adding new validate methods to all
> > > connector-level plugins that can be used similarly to how the existing
> > > Connector::validate method (
> > >
> > https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> > )
> > > is used. This would allow for plugins to perform validation that's more
> > > sophisticated than what the ConfigDef is capable of, such as validating
> > > combinations of properties like a hostname and credentials for reaching
> > it.
> > > I know that at least Confluent's Avro, protobuf, and JSON schema
> > converters
> > > would benefit from this kind of feature. It's a little tangential to this
> > > KIP (which at the moment is about discovering plugins and their
> > > configuration surfaces, as opposed to validating them), but I figured I'd
> > > ask since we're going to be expanding the Converter interface and it may
> > be
> > > useful to tackle this while we're in the neighborhood.
> > >
> > > 4. The description for the new /plugins/<type>/<name>/configdef endpoint
> > > states that "Name must be the fully qualified class name of the plugin".
> > > Any reason not to also support aliases (e.g., "FileStreamSinkConnector"
> > or
> > > "FileStreamSink" instead of
> > > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <
> > mickael.maison@gmail.com>
> > > wrote:
> > >
> > > > Thanks all for the feedback!
> > > >
> > > > Chris,
> > > > I agree that fixing the current endpoint helps a lot. Thanks for
> > > > raising these JIRAs and submitting a PR!
> > > > However thinking about the issue further, I decided to expand the
> > > > scope of the KIP to cover all user-visible plugins.
> > > > In practice, users want to know about all available plugins not only
> > > > connectors. This includes transformations, converters,
> > > > header_converters and predicates. As we also want to retrieve
> > > > configdef for these too, I think it makes sense to introduce a new
> > > > endpoint to do so. Alongside we obviously need a new endpoint for
> > > > listing all plugins.
> > > >
> > > > Gunnar,
> > > > I took a look at exposing valid values via the API. I think the issue
> > > > is that Validators don't expose a way to retrieve valid values.
> > > > Changing validators will have an impact on all components so I'd
> > > > prefer to address this requirement in a separate KIP. I agree this
> > > > would be an interesting improvement and I'd happy to write a KIP for
> > > > it too.
> > > >
> > > >
> > > > I have updated the KIP accordingly. Let me know if you have further
> > > > feedback.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > > > <gu...@googlemail.com.invalid> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I'm +1 for adding a GET endpoint for obtaining config definitions. It
> > > > > always felt odd to me that one has to issue a PUT for that purpose.
> > If
> > > > > nothing else, it'd be better in terms of discoverability of the KC
> > REST
> > > > API.
> > > > >
> > > > > One additional feature request I'd have is to expose the valid enum
> > > > > constants for enum-typed options. That'll help to display the values
> > in a
> > > > > drop-down or via radio buttons in a UI, give us tab completion in
> > kcctl,
> > > > > etc.
> > > > >
> > > > > Best,
> > > > >
> > > > > --Gunnar
> > > > >
> > > > >
> > > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > > > > <ch...@confluent.io.invalid>:
> > > > >
> > > > > > Hi Viktor,
> > > > > >
> > > > > > It sounds like there are three major points here in favor of a new
> > GET
> > > > > > endpoint for connector config defs.
> > > > > >
> > > > > > 1. You cannot issue a blank ("dummy") request for sink connectors
> > > > because a
> > > > > > topic list/topic regex has to be supplied (otherwise the PUT
> > endpoint
> > > > > > returns a 500 response)
> > > > > > 2. A dummy request still triggers custom validations by the
> > connector,
> > > > > > which may be best to avoid if we know for sure that the config
> > isn't
> > > > worth
> > > > > > validating yet
> > > > > > 3. It's more ergonomic and intuitive to be able to issue a GET
> > request
> > > > > > without having to give a dummy connector config
> > > > > >
> > > > > > With regards to 1, this is actually a bug in Connect (
> > > > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix
> > already
> > > > > > implemented and awaiting committer review (
> > > > > > https://github.com/apache/kafka/pull/11369). I think it'd be
> > better to
> > > > > > focus on fixing this bug in general instead of implementing a new
> > REST
> > > > > > endpoint in order to allow people to work around it.
> > > > > >
> > > > > > With regards to 2, this is technically possible but I'm unsure
> > it'd be
> > > > too
> > > > > > common out in the wild given that most validations that could be
> > > > expensive
> > > > > > would involve things like connecting to a database, checking if a
> > cloud
> > > > > > storage bucket exists, etc., none of which are possible without
> > some
> > > > > > configuration properties from the user (db hostname, bucket name,
> > > > etc.).
> > > > > >
> > > > > > With regards to 3, I do agree that it'd be easier for people
> > designing
> > > > UIs
> > > > > > to have a GET API to work against. I'm just not sure it's worth the
> > > > > > additional implementation, testing, and maintenance burden. If it
> > were
> > > > > > possible to issue a PUT request without unexpected 500s for invalid
> > > > > > configs, would that suffice? AFAICT it'd basically be as simple as
> > > > issuing
> > > > > > a PUT request with a dummy body consisting of nothing except the
> > > > connector
> > > > > > class (which at this point we might even make unnecessary and just
> > > > > > automatically replace with the connector class from the URL) and
> > then
> > > > > > filtering the response to just grab the "definition" field of each
> > > > element
> > > > > > in the "configs" array in the response.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > > viktorsomogyi@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Folks,
> > > > > > >
> > > > > > > I too think this would be a very useful feature. Some of our
> > > > management
> > > > > > > applications would provide a wizard for creating connectors. In
> > this
> > > > > > > scenario the user basically would fill out a sample configuration
> > > > > > generated
> > > > > > > by the UI which would send it back to Connect for validation and
> > > > > > eventually
> > > > > > > create a new connector. The first part of this workflow can be
> > > > enhanced
> > > > > > if
> > > > > > > we had an API that can return the configuration definition of the
> > > > given
> > > > > > > type of connector as the UI application would be able to
> > generate a
> > > > > > sample
> > > > > > > for the user based on that (nicely drawn diagram:
> > > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > > The connector-plugins/{connectorType}/config/validate API
> > essentially
> > > > > > works
> > > > > > > and returns the data that we need, however it is a HTTP PUT API
> > that
> > > > is a
> > > > > > > bit unintuitive for a fetch-like functionality and also
> > functionally
> > > > > > > different as it validates the given (dummy) request. In case of
> > sink
> > > > > > > connectors one would need to also provide a topic name.
> > > > > > >
> > > > > > > A suggestion for the KIP: I think it can be useful to return the
> > > > config
> > > > > > > groups and the connector class' name similarly to the validate
> > API
> > > > just
> > > > > > in
> > > > > > > case any frontend needs them (and also the response would be more
> > > > like
> > > > > > the
> > > > > > > validate API but simpler).
> > > > > > >
> > > > > > > Viktor
> > > > > > >
> > > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> > ryannedolan@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I think it'd be worth adding a GET version, fwiw. Could be the
> > same
> > > > > > > handler
> > > > > > > > with just a different spelling maybe.
> > > > > > > >
> > > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > > mickael.maison@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Chris,
> > > > > > > > >
> > > > > > > > > You're right, you can achieve the same functionality using
> > the
> > > > > > > > > existing validate endpoint.
> > > > > > > > > In my mind it was only for validation once you have build a
> > > > > > > > > configuration but when used with an empty configuration, it
> > > > basically
> > > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > > >
> > > > > > > > > I think it's a bit easier to use a GET endpoint but I don't
> > > > think it
> > > > > > > > > really warrants a different endpoint.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Mickael,
> > > > > > > > > >
> > > > > > > > > > I'm wondering about the use case here. The motivation
> > section
> > > > > > states
> > > > > > > > that
> > > > > > > > > > "Connect does not provide a way to see what configurations
> > a
> > > > > > > connector
> > > > > > > > > > requires. Instead users have to go look at the connector
> > > > > > > documentation
> > > > > > > > or
> > > > > > > > > > in the worst case, look directly at the connector source
> > > > code.",
> > > > > > and
> > > > > > > > that
> > > > > > > > > > with this KIP, "users will be able to discover the required
> > > > > > > > > configurations
> > > > > > > > > > for connectors installed in a Connect cluster" and "tools
> > will
> > > > be
> > > > > > > able
> > > > > > > > to
> > > > > > > > > > generate wizards for configuring and starting connectors".
> > > > > > > > > >
> > > > > > > > > > Does the existing "PUT
> > > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > > endpoint not address these points? What will the
> > newly-proposed
> > > > > > > > endpoint
> > > > > > > > > > allow users to do that they will not already be able to do
> > > > with the
> > > > > > > > > > existing endpoint?
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > >
> > > > > > > > > > Chris
> > > > > > > > > >
> > > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > > mickael.maison@gmail.com
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > I've created KIP-769 to expose connector configuration
> > > > > > definitions
> > > > > > > in
> > > > > > > > > > > the Connect API
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > > >
> > > > > > > > > > > Please take a look and let me know if you have any
> > feedback.
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Chris Egerton <ch...@confluent.io.INVALID>]

Hi Mickael,

I think one potential use case for exposing worker-level plugins is that it
may make it easier to determine whether a worker is set up correctly (the
current alternative requires looking through log files and can be a little
tedious), and might even make it possible to automatically identify
discrepancies within a cluster by diffing the contents of that endpoint
across each worker. But I don't think this has to be addressed by the
current KIP; the only thing that bothers me a little is that "plugins" is
generic and it may confuse people down the road if we add an endpoint for
worker-level plugins ("why is one just called 'plugins' and the other one
is 'worker-plugins'?"). Probably not worth blocking on, though.

Agreed that the suggestion for improved validation should be made on the
KIP-802 thread.

I also noticed that the newly-proposed config method for the Converter
interface doesn't have a default implementation, making it
backwards-incompatible. Should we add a default implementation that returns
either null or an empty ConfigDef?

Cheers,

Chris

On Fri, Nov 26, 2021 at 8:35 AM Mickael Maison <mi...@gmail.com>
wrote:

> Hi Chris,
>
> 1. If we want to expose worker plugins, I think we should do it via a
> separate endpoint. But to be honest, I'm not even sure I see strong
> use cases for exposing them as they are either enabled or not and
> can't be changed at runtime. So I'd prefer to stick to "connector
> level" plugins in this KIP. Let me now if you have use cases, I'm open
> to reconsider this choice.
> I'll add that in the rejected alternatives section for now
>
> 2. I remembered seeing issues in the past with multiple plugin.path
> entries but I tried today and I was able to mix and match plugins from
> different paths. So my bad for getting confused.
> Then I agree, it makes more sense to group them by plugin type.
>
> 3. Yes this should be covered in KIP-802:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options
>
> 4. No particular reason. We can support both formats like today. I've
> updated the KIP
>
> Thanks,
> Mickael
>
>
>
> On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
> <ch...@confluent.io.invalid> wrote:
> >
> > Hi Mickael,
> >
> > I think the increase in scope here is great and the added value certainly
> > justifies the proposed changes. I have some thoughts but overall I like
> the
> > direction this is going in now.
> >
> > 1. The new /plugins endpoint is described as containing "all plugins that
> > are Connectors, Transformations, Converters, HeaderConverters and
> > Predicates". So essentially, it looks like we want to expose all plugins
> > that are configured on a per-connector basis, but exclude plugins that
> are
> > configured on a per-worker basis (such as config providers and REST
> > extensions). Do you think it may be valuable to expose information on
> > worker-level plugins as well?
> >
> > 2. The description for the new /plugins endpoint also states that
> "Plugins
> > will be grouped by plugin.path. This will make it clear to users what's
> > available to use as it's not possible to use a Connector from one path
> with
> > Transformations from another.". Is this true? I thought that Connect's
> > classloading made it possible to package
> > converters/transformations/predicates completely independently from each
> > other, and to reference them from also-independently-packaged connectors.
> > If it turns out that this is the case, could we consider restructuring
> the
> > response to be grouped by plugin type instead of by classloader? There's
> > also the ungrouped format proposed in KIP-494 (
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150
> )
> > which we might consider as well.
> >
> > 3. I think this can be left for a follow-up KIP if necessary, but I'm
> > curious about your thoughts on adding new validate methods to all
> > connector-level plugins that can be used similarly to how the existing
> > Connector::validate method (
> >
> https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146
> )
> > is used. This would allow for plugins to perform validation that's more
> > sophisticated than what the ConfigDef is capable of, such as validating
> > combinations of properties like a hostname and credentials for reaching
> it.
> > I know that at least Confluent's Avro, protobuf, and JSON schema
> converters
> > would benefit from this kind of feature. It's a little tangential to this
> > KIP (which at the moment is about discovering plugins and their
> > configuration surfaces, as opposed to validating them), but I figured I'd
> > ask since we're going to be expanding the Converter interface and it may
> be
> > useful to tackle this while we're in the neighborhood.
> >
> > 4. The description for the new /plugins/<type>/<name>/configdef endpoint
> > states that "Name must be the fully qualified class name of the plugin".
> > Any reason not to also support aliases (e.g., "FileStreamSinkConnector"
> or
> > "FileStreamSink" instead of
> > "org.apache.kafka.connect.file.FileStreamSinkConnector")?
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <
> mickael.maison@gmail.com>
> > wrote:
> >
> > > Thanks all for the feedback!
> > >
> > > Chris,
> > > I agree that fixing the current endpoint helps a lot. Thanks for
> > > raising these JIRAs and submitting a PR!
> > > However thinking about the issue further, I decided to expand the
> > > scope of the KIP to cover all user-visible plugins.
> > > In practice, users want to know about all available plugins not only
> > > connectors. This includes transformations, converters,
> > > header_converters and predicates. As we also want to retrieve
> > > configdef for these too, I think it makes sense to introduce a new
> > > endpoint to do so. Alongside we obviously need a new endpoint for
> > > listing all plugins.
> > >
> > > Gunnar,
> > > I took a look at exposing valid values via the API. I think the issue
> > > is that Validators don't expose a way to retrieve valid values.
> > > Changing validators will have an impact on all components so I'd
> > > prefer to address this requirement in a separate KIP. I agree this
> > > would be an interesting improvement and I'd happy to write a KIP for
> > > it too.
> > >
> > >
> > > I have updated the KIP accordingly. Let me know if you have further
> > > feedback.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > > <gu...@googlemail.com.invalid> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I'm +1 for adding a GET endpoint for obtaining config definitions. It
> > > > always felt odd to me that one has to issue a PUT for that purpose.
> If
> > > > nothing else, it'd be better in terms of discoverability of the KC
> REST
> > > API.
> > > >
> > > > One additional feature request I'd have is to expose the valid enum
> > > > constants for enum-typed options. That'll help to display the values
> in a
> > > > drop-down or via radio buttons in a UI, give us tab completion in
> kcctl,
> > > > etc.
> > > >
> > > > Best,
> > > >
> > > > --Gunnar
> > > >
> > > >
> > > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > > > <ch...@confluent.io.invalid>:
> > > >
> > > > > Hi Viktor,
> > > > >
> > > > > It sounds like there are three major points here in favor of a new
> GET
> > > > > endpoint for connector config defs.
> > > > >
> > > > > 1. You cannot issue a blank ("dummy") request for sink connectors
> > > because a
> > > > > topic list/topic regex has to be supplied (otherwise the PUT
> endpoint
> > > > > returns a 500 response)
> > > > > 2. A dummy request still triggers custom validations by the
> connector,
> > > > > which may be best to avoid if we know for sure that the config
> isn't
> > > worth
> > > > > validating yet
> > > > > 3. It's more ergonomic and intuitive to be able to issue a GET
> request
> > > > > without having to give a dummy connector config
> > > > >
> > > > > With regards to 1, this is actually a bug in Connect (
> > > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix
> already
> > > > > implemented and awaiting committer review (
> > > > > https://github.com/apache/kafka/pull/11369). I think it'd be
> better to
> > > > > focus on fixing this bug in general instead of implementing a new
> REST
> > > > > endpoint in order to allow people to work around it.
> > > > >
> > > > > With regards to 2, this is technically possible but I'm unsure
> it'd be
> > > too
> > > > > common out in the wild given that most validations that could be
> > > expensive
> > > > > would involve things like connecting to a database, checking if a
> cloud
> > > > > storage bucket exists, etc., none of which are possible without
> some
> > > > > configuration properties from the user (db hostname, bucket name,
> > > etc.).
> > > > >
> > > > > With regards to 3, I do agree that it'd be easier for people
> designing
> > > UIs
> > > > > to have a GET API to work against. I'm just not sure it's worth the
> > > > > additional implementation, testing, and maintenance burden. If it
> were
> > > > > possible to issue a PUT request without unexpected 500s for invalid
> > > > > configs, would that suffice? AFAICT it'd basically be as simple as
> > > issuing
> > > > > a PUT request with a dummy body consisting of nothing except the
> > > connector
> > > > > class (which at this point we might even make unnecessary and just
> > > > > automatically replace with the connector class from the URL) and
> then
> > > > > filtering the response to just grab the "definition" field of each
> > > element
> > > > > in the "configs" array in the response.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > > viktorsomogyi@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Folks,
> > > > > >
> > > > > > I too think this would be a very useful feature. Some of our
> > > management
> > > > > > applications would provide a wizard for creating connectors. In
> this
> > > > > > scenario the user basically would fill out a sample configuration
> > > > > generated
> > > > > > by the UI which would send it back to Connect for validation and
> > > > > eventually
> > > > > > create a new connector. The first part of this workflow can be
> > > enhanced
> > > > > if
> > > > > > we had an API that can return the configuration definition of the
> > > given
> > > > > > type of connector as the UI application would be able to
> generate a
> > > > > sample
> > > > > > for the user based on that (nicely drawn diagram:
> > > > > > https://imgur.com/a/7S1Xwm5).
> > > > > > The connector-plugins/{connectorType}/config/validate API
> essentially
> > > > > works
> > > > > > and returns the data that we need, however it is a HTTP PUT API
> that
> > > is a
> > > > > > bit unintuitive for a fetch-like functionality and also
> functionally
> > > > > > different as it validates the given (dummy) request. In case of
> sink
> > > > > > connectors one would need to also provide a topic name.
> > > > > >
> > > > > > A suggestion for the KIP: I think it can be useful to return the
> > > config
> > > > > > groups and the connector class' name similarly to the validate
> API
> > > just
> > > > > in
> > > > > > case any frontend needs them (and also the response would be more
> > > like
> > > > > the
> > > > > > validate API but simpler).
> > > > > >
> > > > > > Viktor
> > > > > >
> > > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <
> ryannedolan@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I think it'd be worth adding a GET version, fwiw. Could be the
> same
> > > > > > handler
> > > > > > > with just a different spelling maybe.
> > > > > > >
> > > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > > mickael.maison@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Chris,
> > > > > > > >
> > > > > > > > You're right, you can achieve the same functionality using
> the
> > > > > > > > existing validate endpoint.
> > > > > > > > In my mind it was only for validation once you have build a
> > > > > > > > configuration but when used with an empty configuration, it
> > > basically
> > > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > > >
> > > > > > > > I think it's a bit easier to use a GET endpoint but I don't
> > > think it
> > > > > > > > really warrants a different endpoint.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > > >
> > > > > > > > > Hi Mickael,
> > > > > > > > >
> > > > > > > > > I'm wondering about the use case here. The motivation
> section
> > > > > states
> > > > > > > that
> > > > > > > > > "Connect does not provide a way to see what configurations
> a
> > > > > > connector
> > > > > > > > > requires. Instead users have to go look at the connector
> > > > > > documentation
> > > > > > > or
> > > > > > > > > in the worst case, look directly at the connector source
> > > code.",
> > > > > and
> > > > > > > that
> > > > > > > > > with this KIP, "users will be able to discover the required
> > > > > > > > configurations
> > > > > > > > > for connectors installed in a Connect cluster" and "tools
> will
> > > be
> > > > > > able
> > > > > > > to
> > > > > > > > > generate wizards for configuring and starting connectors".
> > > > > > > > >
> > > > > > > > > Does the existing "PUT
> > > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > > endpoint not address these points? What will the
> newly-proposed
> > > > > > > endpoint
> > > > > > > > > allow users to do that they will not already be able to do
> > > with the
> > > > > > > > > existing endpoint?
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > Chris
> > > > > > > > >
> > > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > > mickael.maison@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I've created KIP-769 to expose connector configuration
> > > > > definitions
> > > > > > in
> > > > > > > > > > the Connect API
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > > >
> > > > > > > > > > Please take a look and let me know if you have any
> feedback.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
>

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Mickael Maison <mi...@gmail.com>]

Hi Chris,

1. If we want to expose worker plugins, I think we should do it via a
separate endpoint. But to be honest, I'm not even sure I see strong
use cases for exposing them as they are either enabled or not and
can't be changed at runtime. So I'd prefer to stick to "connector
level" plugins in this KIP. Let me now if you have use cases, I'm open
to reconsider this choice.
I'll add that in the rejected alternatives section for now

2. I remembered seeing issues in the past with multiple plugin.path
entries but I tried today and I was able to mix and match plugins from
different paths. So my bad for getting confused.
Then I agree, it makes more sense to group them by plugin type.

3. Yes this should be covered in KIP-802:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-802%3A+Validation+Support+for+Kafka+Connect+SMT+Options

4. No particular reason. We can support both formats like today. I've
updated the KIP

Thanks,
Mickael



On Tue, Nov 23, 2021 at 6:40 PM Chris Egerton
<ch...@confluent.io.invalid> wrote:
>
> Hi Mickael,
>
> I think the increase in scope here is great and the added value certainly
> justifies the proposed changes. I have some thoughts but overall I like the
> direction this is going in now.
>
> 1. The new /plugins endpoint is described as containing "all plugins that
> are Connectors, Transformations, Converters, HeaderConverters and
> Predicates". So essentially, it looks like we want to expose all plugins
> that are configured on a per-connector basis, but exclude plugins that are
> configured on a per-worker basis (such as config providers and REST
> extensions). Do you think it may be valuable to expose information on
> worker-level plugins as well?
>
> 2. The description for the new /plugins endpoint also states that "Plugins
> will be grouped by plugin.path. This will make it clear to users what's
> available to use as it's not possible to use a Connector from one path with
> Transformations from another.". Is this true? I thought that Connect's
> classloading made it possible to package
> converters/transformations/predicates completely independently from each
> other, and to reference them from also-independently-packaged connectors.
> If it turns out that this is the case, could we consider restructuring the
> response to be grouped by plugin type instead of by classloader? There's
> also the ungrouped format proposed in KIP-494 (
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150)
> which we might consider as well.
>
> 3. I think this can be left for a follow-up KIP if necessary, but I'm
> curious about your thoughts on adding new validate methods to all
> connector-level plugins that can be used similarly to how the existing
> Connector::validate method (
> https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146)
> is used. This would allow for plugins to perform validation that's more
> sophisticated than what the ConfigDef is capable of, such as validating
> combinations of properties like a hostname and credentials for reaching it.
> I know that at least Confluent's Avro, protobuf, and JSON schema converters
> would benefit from this kind of feature. It's a little tangential to this
> KIP (which at the moment is about discovering plugins and their
> configuration surfaces, as opposed to validating them), but I figured I'd
> ask since we're going to be expanding the Converter interface and it may be
> useful to tackle this while we're in the neighborhood.
>
> 4. The description for the new /plugins/<type>/<name>/configdef endpoint
> states that "Name must be the fully qualified class name of the plugin".
> Any reason not to also support aliases (e.g., "FileStreamSinkConnector" or
> "FileStreamSink" instead of
> "org.apache.kafka.connect.file.FileStreamSinkConnector")?
>
> Cheers,
>
> Chris
>
> On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Thanks all for the feedback!
> >
> > Chris,
> > I agree that fixing the current endpoint helps a lot. Thanks for
> > raising these JIRAs and submitting a PR!
> > However thinking about the issue further, I decided to expand the
> > scope of the KIP to cover all user-visible plugins.
> > In practice, users want to know about all available plugins not only
> > connectors. This includes transformations, converters,
> > header_converters and predicates. As we also want to retrieve
> > configdef for these too, I think it makes sense to introduce a new
> > endpoint to do so. Alongside we obviously need a new endpoint for
> > listing all plugins.
> >
> > Gunnar,
> > I took a look at exposing valid values via the API. I think the issue
> > is that Validators don't expose a way to retrieve valid values.
> > Changing validators will have an impact on all components so I'd
> > prefer to address this requirement in a separate KIP. I agree this
> > would be an interesting improvement and I'd happy to write a KIP for
> > it too.
> >
> >
> > I have updated the KIP accordingly. Let me know if you have further
> > feedback.
> >
> > Thanks,
> > Mickael
> >
> > On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> > <gu...@googlemail.com.invalid> wrote:
> > >
> > > Hi,
> > >
> > > I'm +1 for adding a GET endpoint for obtaining config definitions. It
> > > always felt odd to me that one has to issue a PUT for that purpose. If
> > > nothing else, it'd be better in terms of discoverability of the KC REST
> > API.
> > >
> > > One additional feature request I'd have is to expose the valid enum
> > > constants for enum-typed options. That'll help to display the values in a
> > > drop-down or via radio buttons in a UI, give us tab completion in kcctl,
> > > etc.
> > >
> > > Best,
> > >
> > > --Gunnar
> > >
> > >
> > > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > > <ch...@confluent.io.invalid>:
> > >
> > > > Hi Viktor,
> > > >
> > > > It sounds like there are three major points here in favor of a new GET
> > > > endpoint for connector config defs.
> > > >
> > > > 1. You cannot issue a blank ("dummy") request for sink connectors
> > because a
> > > > topic list/topic regex has to be supplied (otherwise the PUT endpoint
> > > > returns a 500 response)
> > > > 2. A dummy request still triggers custom validations by the connector,
> > > > which may be best to avoid if we know for sure that the config isn't
> > worth
> > > > validating yet
> > > > 3. It's more ergonomic and intuitive to be able to issue a GET request
> > > > without having to give a dummy connector config
> > > >
> > > > With regards to 1, this is actually a bug in Connect (
> > > > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix already
> > > > implemented and awaiting committer review (
> > > > https://github.com/apache/kafka/pull/11369). I think it'd be better to
> > > > focus on fixing this bug in general instead of implementing a new REST
> > > > endpoint in order to allow people to work around it.
> > > >
> > > > With regards to 2, this is technically possible but I'm unsure it'd be
> > too
> > > > common out in the wild given that most validations that could be
> > expensive
> > > > would involve things like connecting to a database, checking if a cloud
> > > > storage bucket exists, etc., none of which are possible without some
> > > > configuration properties from the user (db hostname, bucket name,
> > etc.).
> > > >
> > > > With regards to 3, I do agree that it'd be easier for people designing
> > UIs
> > > > to have a GET API to work against. I'm just not sure it's worth the
> > > > additional implementation, testing, and maintenance burden. If it were
> > > > possible to issue a PUT request without unexpected 500s for invalid
> > > > configs, would that suffice? AFAICT it'd basically be as simple as
> > issuing
> > > > a PUT request with a dummy body consisting of nothing except the
> > connector
> > > > class (which at this point we might even make unnecessary and just
> > > > automatically replace with the connector class from the URL) and then
> > > > filtering the response to just grab the "definition" field of each
> > element
> > > > in the "configs" array in the response.
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > > viktorsomogyi@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Folks,
> > > > >
> > > > > I too think this would be a very useful feature. Some of our
> > management
> > > > > applications would provide a wizard for creating connectors. In this
> > > > > scenario the user basically would fill out a sample configuration
> > > > generated
> > > > > by the UI which would send it back to Connect for validation and
> > > > eventually
> > > > > create a new connector. The first part of this workflow can be
> > enhanced
> > > > if
> > > > > we had an API that can return the configuration definition of the
> > given
> > > > > type of connector as the UI application would be able to generate a
> > > > sample
> > > > > for the user based on that (nicely drawn diagram:
> > > > > https://imgur.com/a/7S1Xwm5).
> > > > > The connector-plugins/{connectorType}/config/validate API essentially
> > > > works
> > > > > and returns the data that we need, however it is a HTTP PUT API that
> > is a
> > > > > bit unintuitive for a fetch-like functionality and also functionally
> > > > > different as it validates the given (dummy) request. In case of sink
> > > > > connectors one would need to also provide a topic name.
> > > > >
> > > > > A suggestion for the KIP: I think it can be useful to return the
> > config
> > > > > groups and the connector class' name similarly to the validate API
> > just
> > > > in
> > > > > case any frontend needs them (and also the response would be more
> > like
> > > > the
> > > > > validate API but simpler).
> > > > >
> > > > > Viktor
> > > > >
> > > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <ry...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I think it'd be worth adding a GET version, fwiw. Could be the same
> > > > > handler
> > > > > > with just a different spelling maybe.
> > > > > >
> > > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> > mickael.maison@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Chris,
> > > > > > >
> > > > > > > You're right, you can achieve the same functionality using the
> > > > > > > existing validate endpoint.
> > > > > > > In my mind it was only for validation once you have build a
> > > > > > > configuration but when used with an empty configuration, it
> > basically
> > > > > > > serves the same purpose as the proposed new endpoint.
> > > > > > >
> > > > > > > I think it's a bit easier to use a GET endpoint but I don't
> > think it
> > > > > > > really warrants a different endpoint.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > > >
> > > > > > > > Hi Mickael,
> > > > > > > >
> > > > > > > > I'm wondering about the use case here. The motivation section
> > > > states
> > > > > > that
> > > > > > > > "Connect does not provide a way to see what configurations a
> > > > > connector
> > > > > > > > requires. Instead users have to go look at the connector
> > > > > documentation
> > > > > > or
> > > > > > > > in the worst case, look directly at the connector source
> > code.",
> > > > and
> > > > > > that
> > > > > > > > with this KIP, "users will be able to discover the required
> > > > > > > configurations
> > > > > > > > for connectors installed in a Connect cluster" and "tools will
> > be
> > > > > able
> > > > > > to
> > > > > > > > generate wizards for configuring and starting connectors".
> > > > > > > >
> > > > > > > > Does the existing "PUT
> > > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > > endpoint not address these points? What will the newly-proposed
> > > > > > endpoint
> > > > > > > > allow users to do that they will not already be able to do
> > with the
> > > > > > > > existing endpoint?
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > > mickael.maison@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I've created KIP-769 to expose connector configuration
> > > > definitions
> > > > > in
> > > > > > > > > the Connect API
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > > >
> > > > > > > > > Please take a look and let me know if you have any feedback.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Chris Egerton <ch...@confluent.io.INVALID>]

Hi Mickael,

I think the increase in scope here is great and the added value certainly
justifies the proposed changes. I have some thoughts but overall I like the
direction this is going in now.

1. The new /plugins endpoint is described as containing "all plugins that
are Connectors, Transformations, Converters, HeaderConverters and
Predicates". So essentially, it looks like we want to expose all plugins
that are configured on a per-connector basis, but exclude plugins that are
configured on a per-worker basis (such as config providers and REST
extensions). Do you think it may be valuable to expose information on
worker-level plugins as well?

2. The description for the new /plugins endpoint also states that "Plugins
will be grouped by plugin.path. This will make it clear to users what's
available to use as it's not possible to use a Connector from one path with
Transformations from another.". Is this true? I thought that Connect's
classloading made it possible to package
converters/transformations/predicates completely independently from each
other, and to reference them from also-independently-packaged connectors.
If it turns out that this is the case, could we consider restructuring the
response to be grouped by plugin type instead of by classloader? There's
also the ungrouped format proposed in KIP-494 (
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150)
which we might consider as well.

3. I think this can be left for a follow-up KIP if necessary, but I'm
curious about your thoughts on adding new validate methods to all
connector-level plugins that can be used similarly to how the existing
Connector::validate method (
https://github.com/apache/kafka/blob/1e0916580f16b99b911b0ed36e9740dcaeef520e/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146)
is used. This would allow for plugins to perform validation that's more
sophisticated than what the ConfigDef is capable of, such as validating
combinations of properties like a hostname and credentials for reaching it.
I know that at least Confluent's Avro, protobuf, and JSON schema converters
would benefit from this kind of feature. It's a little tangential to this
KIP (which at the moment is about discovering plugins and their
configuration surfaces, as opposed to validating them), but I figured I'd
ask since we're going to be expanding the Converter interface and it may be
useful to tackle this while we're in the neighborhood.

4. The description for the new /plugins/<type>/<name>/configdef endpoint
states that "Name must be the fully qualified class name of the plugin".
Any reason not to also support aliases (e.g., "FileStreamSinkConnector" or
"FileStreamSink" instead of
"org.apache.kafka.connect.file.FileStreamSinkConnector")?

Cheers,

Chris

On Tue, Nov 23, 2021 at 12:07 PM Mickael Maison <mi...@gmail.com>
wrote:

> Thanks all for the feedback!
>
> Chris,
> I agree that fixing the current endpoint helps a lot. Thanks for
> raising these JIRAs and submitting a PR!
> However thinking about the issue further, I decided to expand the
> scope of the KIP to cover all user-visible plugins.
> In practice, users want to know about all available plugins not only
> connectors. This includes transformations, converters,
> header_converters and predicates. As we also want to retrieve
> configdef for these too, I think it makes sense to introduce a new
> endpoint to do so. Alongside we obviously need a new endpoint for
> listing all plugins.
>
> Gunnar,
> I took a look at exposing valid values via the API. I think the issue
> is that Validators don't expose a way to retrieve valid values.
> Changing validators will have an impact on all components so I'd
> prefer to address this requirement in a separate KIP. I agree this
> would be an interesting improvement and I'd happy to write a KIP for
> it too.
>
>
> I have updated the KIP accordingly. Let me know if you have further
> feedback.
>
> Thanks,
> Mickael
>
> On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
> <gu...@googlemail.com.invalid> wrote:
> >
> > Hi,
> >
> > I'm +1 for adding a GET endpoint for obtaining config definitions. It
> > always felt odd to me that one has to issue a PUT for that purpose. If
> > nothing else, it'd be better in terms of discoverability of the KC REST
> API.
> >
> > One additional feature request I'd have is to expose the valid enum
> > constants for enum-typed options. That'll help to display the values in a
> > drop-down or via radio buttons in a UI, give us tab completion in kcctl,
> > etc.
> >
> > Best,
> >
> > --Gunnar
> >
> >
> > Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> > <ch...@confluent.io.invalid>:
> >
> > > Hi Viktor,
> > >
> > > It sounds like there are three major points here in favor of a new GET
> > > endpoint for connector config defs.
> > >
> > > 1. You cannot issue a blank ("dummy") request for sink connectors
> because a
> > > topic list/topic regex has to be supplied (otherwise the PUT endpoint
> > > returns a 500 response)
> > > 2. A dummy request still triggers custom validations by the connector,
> > > which may be best to avoid if we know for sure that the config isn't
> worth
> > > validating yet
> > > 3. It's more ergonomic and intuitive to be able to issue a GET request
> > > without having to give a dummy connector config
> > >
> > > With regards to 1, this is actually a bug in Connect (
> > > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix already
> > > implemented and awaiting committer review (
> > > https://github.com/apache/kafka/pull/11369). I think it'd be better to
> > > focus on fixing this bug in general instead of implementing a new REST
> > > endpoint in order to allow people to work around it.
> > >
> > > With regards to 2, this is technically possible but I'm unsure it'd be
> too
> > > common out in the wild given that most validations that could be
> expensive
> > > would involve things like connecting to a database, checking if a cloud
> > > storage bucket exists, etc., none of which are possible without some
> > > configuration properties from the user (db hostname, bucket name,
> etc.).
> > >
> > > With regards to 3, I do agree that it'd be easier for people designing
> UIs
> > > to have a GET API to work against. I'm just not sure it's worth the
> > > additional implementation, testing, and maintenance burden. If it were
> > > possible to issue a PUT request without unexpected 500s for invalid
> > > configs, would that suffice? AFAICT it'd basically be as simple as
> issuing
> > > a PUT request with a dummy body consisting of nothing except the
> connector
> > > class (which at this point we might even make unnecessary and just
> > > automatically replace with the connector class from the URL) and then
> > > filtering the response to just grab the "definition" field of each
> element
> > > in the "configs" array in the response.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > > viktorsomogyi@gmail.com>
> > > wrote:
> > >
> > > > Hi Folks,
> > > >
> > > > I too think this would be a very useful feature. Some of our
> management
> > > > applications would provide a wizard for creating connectors. In this
> > > > scenario the user basically would fill out a sample configuration
> > > generated
> > > > by the UI which would send it back to Connect for validation and
> > > eventually
> > > > create a new connector. The first part of this workflow can be
> enhanced
> > > if
> > > > we had an API that can return the configuration definition of the
> given
> > > > type of connector as the UI application would be able to generate a
> > > sample
> > > > for the user based on that (nicely drawn diagram:
> > > > https://imgur.com/a/7S1Xwm5).
> > > > The connector-plugins/{connectorType}/config/validate API essentially
> > > works
> > > > and returns the data that we need, however it is a HTTP PUT API that
> is a
> > > > bit unintuitive for a fetch-like functionality and also functionally
> > > > different as it validates the given (dummy) request. In case of sink
> > > > connectors one would need to also provide a topic name.
> > > >
> > > > A suggestion for the KIP: I think it can be useful to return the
> config
> > > > groups and the connector class' name similarly to the validate API
> just
> > > in
> > > > case any frontend needs them (and also the response would be more
> like
> > > the
> > > > validate API but simpler).
> > > >
> > > > Viktor
> > > >
> > > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <ry...@gmail.com>
> > > > wrote:
> > > >
> > > > > I think it'd be worth adding a GET version, fwiw. Could be the same
> > > > handler
> > > > > with just a different spelling maybe.
> > > > >
> > > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <
> mickael.maison@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Chris,
> > > > > >
> > > > > > You're right, you can achieve the same functionality using the
> > > > > > existing validate endpoint.
> > > > > > In my mind it was only for validation once you have build a
> > > > > > configuration but when used with an empty configuration, it
> basically
> > > > > > serves the same purpose as the proposed new endpoint.
> > > > > >
> > > > > > I think it's a bit easier to use a GET endpoint but I don't
> think it
> > > > > > really warrants a different endpoint.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > > <ch...@confluent.io.invalid> wrote:
> > > > > > >
> > > > > > > Hi Mickael,
> > > > > > >
> > > > > > > I'm wondering about the use case here. The motivation section
> > > states
> > > > > that
> > > > > > > "Connect does not provide a way to see what configurations a
> > > > connector
> > > > > > > requires. Instead users have to go look at the connector
> > > > documentation
> > > > > or
> > > > > > > in the worst case, look directly at the connector source
> code.",
> > > and
> > > > > that
> > > > > > > with this KIP, "users will be able to discover the required
> > > > > > configurations
> > > > > > > for connectors installed in a Connect cluster" and "tools will
> be
> > > > able
> > > > > to
> > > > > > > generate wizards for configuring and starting connectors".
> > > > > > >
> > > > > > > Does the existing "PUT
> > > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > > endpoint not address these points? What will the newly-proposed
> > > > > endpoint
> > > > > > > allow users to do that they will not already be able to do
> with the
> > > > > > > existing endpoint?
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > > mickael.maison@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I've created KIP-769 to expose connector configuration
> > > definitions
> > > > in
> > > > > > > > the Connect API
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > > >
> > > > > > > > Please take a look and let me know if you have any feedback.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Mickael Maison <mi...@gmail.com>]

Thanks all for the feedback!

Chris,
I agree that fixing the current endpoint helps a lot. Thanks for
raising these JIRAs and submitting a PR!
However thinking about the issue further, I decided to expand the
scope of the KIP to cover all user-visible plugins.
In practice, users want to know about all available plugins not only
connectors. This includes transformations, converters,
header_converters and predicates. As we also want to retrieve
configdef for these too, I think it makes sense to introduce a new
endpoint to do so. Alongside we obviously need a new endpoint for
listing all plugins.

Gunnar,
I took a look at exposing valid values via the API. I think the issue
is that Validators don't expose a way to retrieve valid values.
Changing validators will have an impact on all components so I'd
prefer to address this requirement in a separate KIP. I agree this
would be an interesting improvement and I'd happy to write a KIP for
it too.


I have updated the KIP accordingly. Let me know if you have further feedback.

Thanks,
Mickael

On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
<gu...@googlemail.com.invalid> wrote:
>
> Hi,
>
> I'm +1 for adding a GET endpoint for obtaining config definitions. It
> always felt odd to me that one has to issue a PUT for that purpose. If
> nothing else, it'd be better in terms of discoverability of the KC REST API.
>
> One additional feature request I'd have is to expose the valid enum
> constants for enum-typed options. That'll help to display the values in a
> drop-down or via radio buttons in a UI, give us tab completion in kcctl,
> etc.
>
> Best,
>
> --Gunnar
>
>
> Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> <ch...@confluent.io.invalid>:
>
> > Hi Viktor,
> >
> > It sounds like there are three major points here in favor of a new GET
> > endpoint for connector config defs.
> >
> > 1. You cannot issue a blank ("dummy") request for sink connectors because a
> > topic list/topic regex has to be supplied (otherwise the PUT endpoint
> > returns a 500 response)
> > 2. A dummy request still triggers custom validations by the connector,
> > which may be best to avoid if we know for sure that the config isn't worth
> > validating yet
> > 3. It's more ergonomic and intuitive to be able to issue a GET request
> > without having to give a dummy connector config
> >
> > With regards to 1, this is actually a bug in Connect (
> > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix already
> > implemented and awaiting committer review (
> > https://github.com/apache/kafka/pull/11369). I think it'd be better to
> > focus on fixing this bug in general instead of implementing a new REST
> > endpoint in order to allow people to work around it.
> >
> > With regards to 2, this is technically possible but I'm unsure it'd be too
> > common out in the wild given that most validations that could be expensive
> > would involve things like connecting to a database, checking if a cloud
> > storage bucket exists, etc., none of which are possible without some
> > configuration properties from the user (db hostname, bucket name, etc.).
> >
> > With regards to 3, I do agree that it'd be easier for people designing UIs
> > to have a GET API to work against. I'm just not sure it's worth the
> > additional implementation, testing, and maintenance burden. If it were
> > possible to issue a PUT request without unexpected 500s for invalid
> > configs, would that suffice? AFAICT it'd basically be as simple as issuing
> > a PUT request with a dummy body consisting of nothing except the connector
> > class (which at this point we might even make unnecessary and just
> > automatically replace with the connector class from the URL) and then
> > filtering the response to just grab the "definition" field of each element
> > in the "configs" array in the response.
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > viktorsomogyi@gmail.com>
> > wrote:
> >
> > > Hi Folks,
> > >
> > > I too think this would be a very useful feature. Some of our management
> > > applications would provide a wizard for creating connectors. In this
> > > scenario the user basically would fill out a sample configuration
> > generated
> > > by the UI which would send it back to Connect for validation and
> > eventually
> > > create a new connector. The first part of this workflow can be enhanced
> > if
> > > we had an API that can return the configuration definition of the given
> > > type of connector as the UI application would be able to generate a
> > sample
> > > for the user based on that (nicely drawn diagram:
> > > https://imgur.com/a/7S1Xwm5).
> > > The connector-plugins/{connectorType}/config/validate API essentially
> > works
> > > and returns the data that we need, however it is a HTTP PUT API that is a
> > > bit unintuitive for a fetch-like functionality and also functionally
> > > different as it validates the given (dummy) request. In case of sink
> > > connectors one would need to also provide a topic name.
> > >
> > > A suggestion for the KIP: I think it can be useful to return the config
> > > groups and the connector class' name similarly to the validate API just
> > in
> > > case any frontend needs them (and also the response would be more like
> > the
> > > validate API but simpler).
> > >
> > > Viktor
> > >
> > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <ry...@gmail.com>
> > > wrote:
> > >
> > > > I think it'd be worth adding a GET version, fwiw. Could be the same
> > > handler
> > > > with just a different spelling maybe.
> > > >
> > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <mickael.maison@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Chris,
> > > > >
> > > > > You're right, you can achieve the same functionality using the
> > > > > existing validate endpoint.
> > > > > In my mind it was only for validation once you have build a
> > > > > configuration but when used with an empty configuration, it basically
> > > > > serves the same purpose as the proposed new endpoint.
> > > > >
> > > > > I think it's a bit easier to use a GET endpoint but I don't think it
> > > > > really warrants a different endpoint.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > <ch...@confluent.io.invalid> wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > I'm wondering about the use case here. The motivation section
> > states
> > > > that
> > > > > > "Connect does not provide a way to see what configurations a
> > > connector
> > > > > > requires. Instead users have to go look at the connector
> > > documentation
> > > > or
> > > > > > in the worst case, look directly at the connector source code.",
> > and
> > > > that
> > > > > > with this KIP, "users will be able to discover the required
> > > > > configurations
> > > > > > for connectors installed in a Connect cluster" and "tools will be
> > > able
> > > > to
> > > > > > generate wizards for configuring and starting connectors".
> > > > > >
> > > > > > Does the existing "PUT
> > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > endpoint not address these points? What will the newly-proposed
> > > > endpoint
> > > > > > allow users to do that they will not already be able to do with the
> > > > > > existing endpoint?
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > mickael.maison@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I've created KIP-769 to expose connector configuration
> > definitions
> > > in
> > > > > > > the Connect API
> > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > >
> > > > > > > Please take a look and let me know if you have any feedback.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > >
> > > >
> > >
> >

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Gunnar Morling <gu...@googlemail.com.INVALID>]

Hi,

I'm +1 for adding a GET endpoint for obtaining config definitions. It
always felt odd to me that one has to issue a PUT for that purpose. If
nothing else, it'd be better in terms of discoverability of the KC REST API.

One additional feature request I'd have is to expose the valid enum
constants for enum-typed options. That'll help to display the values in a
drop-down or via radio buttons in a UI, give us tab completion in kcctl,
etc.

Best,

--Gunnar


Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
<ch...@confluent.io.invalid>:

> Hi Viktor,
>
> It sounds like there are three major points here in favor of a new GET
> endpoint for connector config defs.
>
> 1. You cannot issue a blank ("dummy") request for sink connectors because a
> topic list/topic regex has to be supplied (otherwise the PUT endpoint
> returns a 500 response)
> 2. A dummy request still triggers custom validations by the connector,
> which may be best to avoid if we know for sure that the config isn't worth
> validating yet
> 3. It's more ergonomic and intuitive to be able to issue a GET request
> without having to give a dummy connector config
>
> With regards to 1, this is actually a bug in Connect (
> https://issues.apache.org/jira/browse/KAFKA-13327) with a fix already
> implemented and awaiting committer review (
> https://github.com/apache/kafka/pull/11369). I think it'd be better to
> focus on fixing this bug in general instead of implementing a new REST
> endpoint in order to allow people to work around it.
>
> With regards to 2, this is technically possible but I'm unsure it'd be too
> common out in the wild given that most validations that could be expensive
> would involve things like connecting to a database, checking if a cloud
> storage bucket exists, etc., none of which are possible without some
> configuration properties from the user (db hostname, bucket name, etc.).
>
> With regards to 3, I do agree that it'd be easier for people designing UIs
> to have a GET API to work against. I'm just not sure it's worth the
> additional implementation, testing, and maintenance burden. If it were
> possible to issue a PUT request without unexpected 500s for invalid
> configs, would that suffice? AFAICT it'd basically be as simple as issuing
> a PUT request with a dummy body consisting of nothing except the connector
> class (which at this point we might even make unnecessary and just
> automatically replace with the connector class from the URL) and then
> filtering the response to just grab the "definition" field of each element
> in the "configs" array in the response.
>
> Cheers,
>
> Chris
>
> On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> viktorsomogyi@gmail.com>
> wrote:
>
> > Hi Folks,
> >
> > I too think this would be a very useful feature. Some of our management
> > applications would provide a wizard for creating connectors. In this
> > scenario the user basically would fill out a sample configuration
> generated
> > by the UI which would send it back to Connect for validation and
> eventually
> > create a new connector. The first part of this workflow can be enhanced
> if
> > we had an API that can return the configuration definition of the given
> > type of connector as the UI application would be able to generate a
> sample
> > for the user based on that (nicely drawn diagram:
> > https://imgur.com/a/7S1Xwm5).
> > The connector-plugins/{connectorType}/config/validate API essentially
> works
> > and returns the data that we need, however it is a HTTP PUT API that is a
> > bit unintuitive for a fetch-like functionality and also functionally
> > different as it validates the given (dummy) request. In case of sink
> > connectors one would need to also provide a topic name.
> >
> > A suggestion for the KIP: I think it can be useful to return the config
> > groups and the connector class' name similarly to the validate API just
> in
> > case any frontend needs them (and also the response would be more like
> the
> > validate API but simpler).
> >
> > Viktor
> >
> > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <ry...@gmail.com>
> > wrote:
> >
> > > I think it'd be worth adding a GET version, fwiw. Could be the same
> > handler
> > > with just a different spelling maybe.
> > >
> > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <mickael.maison@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Chris,
> > > >
> > > > You're right, you can achieve the same functionality using the
> > > > existing validate endpoint.
> > > > In my mind it was only for validation once you have build a
> > > > configuration but when used with an empty configuration, it basically
> > > > serves the same purpose as the proposed new endpoint.
> > > >
> > > > I think it's a bit easier to use a GET endpoint but I don't think it
> > > > really warrants a different endpoint.
> > > >
> > > > Thanks
> > > >
> > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > <ch...@confluent.io.invalid> wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > I'm wondering about the use case here. The motivation section
> states
> > > that
> > > > > "Connect does not provide a way to see what configurations a
> > connector
> > > > > requires. Instead users have to go look at the connector
> > documentation
> > > or
> > > > > in the worst case, look directly at the connector source code.",
> and
> > > that
> > > > > with this KIP, "users will be able to discover the required
> > > > configurations
> > > > > for connectors installed in a Connect cluster" and "tools will be
> > able
> > > to
> > > > > generate wizards for configuring and starting connectors".
> > > > >
> > > > > Does the existing "PUT
> > > > /connector-plugins/{connector-type}/config/validate"
> > > > > endpoint not address these points? What will the newly-proposed
> > > endpoint
> > > > > allow users to do that they will not already be able to do with the
> > > > > existing endpoint?
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > mickael.maison@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I've created KIP-769 to expose connector configuration
> definitions
> > in
> > > > > > the Connect API
> > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > >
> > > > > > Please take a look and let me know if you have any feedback.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > >
> > >
> >
>

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Chris Egerton <ch...@confluent.io.INVALID>]

Hi Viktor,

It sounds like there are three major points here in favor of a new GET
endpoint for connector config defs.

1. You cannot issue a blank ("dummy") request for sink connectors because a
topic list/topic regex has to be supplied (otherwise the PUT endpoint
returns a 500 response)
2. A dummy request still triggers custom validations by the connector,
which may be best to avoid if we know for sure that the config isn't worth
validating yet
3. It's more ergonomic and intuitive to be able to issue a GET request
without having to give a dummy connector config

With regards to 1, this is actually a bug in Connect (
https://issues.apache.org/jira/browse/KAFKA-13327) with a fix already
implemented and awaiting committer review (
https://github.com/apache/kafka/pull/11369). I think it'd be better to
focus on fixing this bug in general instead of implementing a new REST
endpoint in order to allow people to work around it.

With regards to 2, this is technically possible but I'm unsure it'd be too
common out in the wild given that most validations that could be expensive
would involve things like connecting to a database, checking if a cloud
storage bucket exists, etc., none of which are possible without some
configuration properties from the user (db hostname, bucket name, etc.).

With regards to 3, I do agree that it'd be easier for people designing UIs
to have a GET API to work against. I'm just not sure it's worth the
additional implementation, testing, and maintenance burden. If it were
possible to issue a PUT request without unexpected 500s for invalid
configs, would that suffice? AFAICT it'd basically be as simple as issuing
a PUT request with a dummy body consisting of nothing except the connector
class (which at this point we might even make unnecessary and just
automatically replace with the connector class from the URL) and then
filtering the response to just grab the "definition" field of each element
in the "configs" array in the response.

Cheers,

Chris

On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <vi...@gmail.com>
wrote:

> Hi Folks,
>
> I too think this would be a very useful feature. Some of our management
> applications would provide a wizard for creating connectors. In this
> scenario the user basically would fill out a sample configuration generated
> by the UI which would send it back to Connect for validation and eventually
> create a new connector. The first part of this workflow can be enhanced if
> we had an API that can return the configuration definition of the given
> type of connector as the UI application would be able to generate a sample
> for the user based on that (nicely drawn diagram:
> https://imgur.com/a/7S1Xwm5).
> The connector-plugins/{connectorType}/config/validate API essentially works
> and returns the data that we need, however it is a HTTP PUT API that is a
> bit unintuitive for a fetch-like functionality and also functionally
> different as it validates the given (dummy) request. In case of sink
> connectors one would need to also provide a topic name.
>
> A suggestion for the KIP: I think it can be useful to return the config
> groups and the connector class' name similarly to the validate API just in
> case any frontend needs them (and also the response would be more like the
> validate API but simpler).
>
> Viktor
>
> On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <ry...@gmail.com>
> wrote:
>
> > I think it'd be worth adding a GET version, fwiw. Could be the same
> handler
> > with just a different spelling maybe.
> >
> > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <mi...@gmail.com>
> > wrote:
> >
> > > Hi Chris,
> > >
> > > You're right, you can achieve the same functionality using the
> > > existing validate endpoint.
> > > In my mind it was only for validation once you have build a
> > > configuration but when used with an empty configuration, it basically
> > > serves the same purpose as the proposed new endpoint.
> > >
> > > I think it's a bit easier to use a GET endpoint but I don't think it
> > > really warrants a different endpoint.
> > >
> > > Thanks
> > >
> > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > <ch...@confluent.io.invalid> wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > I'm wondering about the use case here. The motivation section states
> > that
> > > > "Connect does not provide a way to see what configurations a
> connector
> > > > requires. Instead users have to go look at the connector
> documentation
> > or
> > > > in the worst case, look directly at the connector source code.", and
> > that
> > > > with this KIP, "users will be able to discover the required
> > > configurations
> > > > for connectors installed in a Connect cluster" and "tools will be
> able
> > to
> > > > generate wizards for configuring and starting connectors".
> > > >
> > > > Does the existing "PUT
> > > /connector-plugins/{connector-type}/config/validate"
> > > > endpoint not address these points? What will the newly-proposed
> > endpoint
> > > > allow users to do that they will not already be able to do with the
> > > > existing endpoint?
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > mickael.maison@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I've created KIP-769 to expose connector configuration definitions
> in
> > > > > the Connect API
> > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > >
> > > > > Please take a look and let me know if you have any feedback.
> > > > >
> > > > > Thanks
> > > > >
> > >
> >
>