You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Ashwin <ap...@confluent.io.INVALID> on 2022/06/22 08:25:30 UTC
[DISCUSS] KIP-850: REST API for filtering Connector plugins by type
Hello Folks,
I have picked up a newbie task and request you provide your valuable
feedback.
KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-850%3A+REST+API+for+filtering+Connector+plugins+by+type
Initial PR: https://github.com/apache/kafka/pull/12330
Though https://issues.apache.org/jira/browse/KAFKA-4279 was for an endpoint
to list converter plugins, I propose that we implement a generic "filter"
API which will list connector plugins matched by type (one of
SOURCE,SINK,CONVERTER,HEADER_CONVERTER).
Thanks,
Ashwin
Re: [DISCUSS] KIP-850: REST API for filtering Connector plugins by type
Posted by Ashwin <ap...@confluent.io.INVALID>.
No worries Chris - I understand what you are getting at.
I will pursue updating the current REST API.
Cheers,
Ashwin
On Fri, Jun 24, 2022 at 8:57 AM Chris Egerton <fe...@gmail.com>
wrote:
> Hi Ashwin,
>
> I just want to be clear that I do think there might be some value to being
> able to address the edge case you've raised; I just want to understand
> exactly what users might do with that information. Driving a programmatic
> UI with a dropdown menu for selecting header/key/value converters could be
> one such use case, for example.
>
> If we do decide to tweak the REST API to expose multiple types for classes
> that implement multiple plugins, we'll still need a KIP for that, and I'd
> be happy to review if you wanted to pursue that. I'm also not dead-set
> against the filter API you've proposed, I just don't know of any cases yet
> where the implementation cost would be outweighed by potential benefit to
> users.
>
> I don't want to sound unwelcoming, especially since this is a newbie task.
> Hope this has been helpful so far, and thank you for your interest in
> contributing to Kafka Connect!
>
> Cheers,
>
> Chris
>
> On Thu, Jun 23, 2022 at 11:16 PM Ashwin <ap...@confluent.io.invalid>
> wrote:
>
> > Hi Chris,
> >
> > Sure, if JIRA is obsolete and does not add value, I am fine with
> abandoning
> > this change.
> > I will close the JIRA as well.
> > Thanks for helping out!
> >
> > Ashwin
> >
> > On Fri, Jun 24, 2022 at 4:23 AM Chris Egerton <fe...@gmail.com>
> > wrote:
> >
> > > Hi Ashwin,
> > >
> > > Thanks for the clarification, which is certainly useful. However, it's
> > not
> > > an actual use case. What will users try to do now that would have been
> > > difficult/impossible in the future because of this ambiguity? I ask
> > because
> > > the Jira ticket linked in the KIP is very old and may be obsolete, and
> > > there seems to be a much simpler way to address the edge case of a
> class
> > > implementing multiple plugin interfaces: we can just expose each type
> in
> > > the REST API, instead of picking one at random.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Thu, Jun 23, 2022 at 12:55 AM Ashwin <ap...@confluent.io.invalid>
> > > wrote:
> > >
> > > > Hi Chris,
> > > >
> > > > Thanks for the quick response.
> > > >
> > > > > It seems pretty straightforward to get this via client-side
> filtering
> > > >
> > > > I assume that you meant GET /collector-plugins + jq filtering on
> type
> > > > attribute.
> > > >
> > > > One issue that I see with that approach is that we cannot filter
> > plugins
> > > if
> > > > they implement more than one of Converter, HeaderConverter, Sink,
> > Source
> > > > interfaces.
> > > >
> > > > The type field in PluginDesc field is set based on the order of enum
> > > values
> > > > in PluginType -
> > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/22534424d22e5a4d4d512b0ac1b5f312b867a5bf/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/PluginType.java#L49
> > > > .
> > > >
> > > > Eg: org.apache.kafka.connect.json.JsonConverter implements both
> > Converter
> > > > and HeaderConverter interfaces, but the type value is always set to
> > > > Converter as that value is defined first in PluginType enum.
> > > >
> > > > The new API will list JsonConverter when we query either for
> > > > HeaderConverter or Converter plugin types.
> > > >
> > > > Thanks,
> > > > Ashwin
> > > >
> > > >
> > > > On Thu, Jun 23, 2022 at 12:03 AM Chris Egerton <
> > fearthecellos@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Ashwin,
> > > > >
> > > > > Can you provide a use case for this new behavior? It seems pretty
> > > > > straightforward to get this via client-side filtering, I'm a little
> > > > > skeptical that we need to implement it server-side.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Wed, Jun 22, 2022 at 4:25 AM Ashwin
> <apankaj@confluent.io.invalid
> > >
> > > > > wrote:
> > > > >
> > > > > > Hello Folks,
> > > > > >
> > > > > > I have picked up a newbie task and request you provide your
> > valuable
> > > > > > feedback.
> > > > > >
> > > > > > KIP:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-850%3A+REST+API+for+filtering+Connector+plugins+by+type
> > > > > > Initial PR: https://github.com/apache/kafka/pull/12330
> > > > > >
> > > > > > Though https://issues.apache.org/jira/browse/KAFKA-4279 was for
> an
> > > > > > endpoint
> > > > > > to list converter plugins, I propose that we implement a generic
> > > > "filter"
> > > > > > API which will list connector plugins matched by type (one of
> > > > > > SOURCE,SINK,CONVERTER,HEADER_CONVERTER).
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Ashwin
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-850: REST API for filtering Connector plugins by type
Posted by Chris Egerton <fe...@gmail.com>.
Hi Ashwin,
I just want to be clear that I do think there might be some value to being
able to address the edge case you've raised; I just want to understand
exactly what users might do with that information. Driving a programmatic
UI with a dropdown menu for selecting header/key/value converters could be
one such use case, for example.
If we do decide to tweak the REST API to expose multiple types for classes
that implement multiple plugins, we'll still need a KIP for that, and I'd
be happy to review if you wanted to pursue that. I'm also not dead-set
against the filter API you've proposed, I just don't know of any cases yet
where the implementation cost would be outweighed by potential benefit to
users.
I don't want to sound unwelcoming, especially since this is a newbie task.
Hope this has been helpful so far, and thank you for your interest in
contributing to Kafka Connect!
Cheers,
Chris
On Thu, Jun 23, 2022 at 11:16 PM Ashwin <ap...@confluent.io.invalid>
wrote:
> Hi Chris,
>
> Sure, if JIRA is obsolete and does not add value, I am fine with abandoning
> this change.
> I will close the JIRA as well.
> Thanks for helping out!
>
> Ashwin
>
> On Fri, Jun 24, 2022 at 4:23 AM Chris Egerton <fe...@gmail.com>
> wrote:
>
> > Hi Ashwin,
> >
> > Thanks for the clarification, which is certainly useful. However, it's
> not
> > an actual use case. What will users try to do now that would have been
> > difficult/impossible in the future because of this ambiguity? I ask
> because
> > the Jira ticket linked in the KIP is very old and may be obsolete, and
> > there seems to be a much simpler way to address the edge case of a class
> > implementing multiple plugin interfaces: we can just expose each type in
> > the REST API, instead of picking one at random.
> >
> > Cheers,
> >
> > Chris
> >
> > On Thu, Jun 23, 2022 at 12:55 AM Ashwin <ap...@confluent.io.invalid>
> > wrote:
> >
> > > Hi Chris,
> > >
> > > Thanks for the quick response.
> > >
> > > > It seems pretty straightforward to get this via client-side filtering
> > >
> > > I assume that you meant GET /collector-plugins + jq filtering on type
> > > attribute.
> > >
> > > One issue that I see with that approach is that we cannot filter
> plugins
> > if
> > > they implement more than one of Converter, HeaderConverter, Sink,
> Source
> > > interfaces.
> > >
> > > The type field in PluginDesc field is set based on the order of enum
> > values
> > > in PluginType -
> > >
> > >
> >
> https://github.com/apache/kafka/blob/22534424d22e5a4d4d512b0ac1b5f312b867a5bf/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/PluginType.java#L49
> > > .
> > >
> > > Eg: org.apache.kafka.connect.json.JsonConverter implements both
> Converter
> > > and HeaderConverter interfaces, but the type value is always set to
> > > Converter as that value is defined first in PluginType enum.
> > >
> > > The new API will list JsonConverter when we query either for
> > > HeaderConverter or Converter plugin types.
> > >
> > > Thanks,
> > > Ashwin
> > >
> > >
> > > On Thu, Jun 23, 2022 at 12:03 AM Chris Egerton <
> fearthecellos@gmail.com>
> > > wrote:
> > >
> > > > Hi Ashwin,
> > > >
> > > > Can you provide a use case for this new behavior? It seems pretty
> > > > straightforward to get this via client-side filtering, I'm a little
> > > > skeptical that we need to implement it server-side.
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Wed, Jun 22, 2022 at 4:25 AM Ashwin <apankaj@confluent.io.invalid
> >
> > > > wrote:
> > > >
> > > > > Hello Folks,
> > > > >
> > > > > I have picked up a newbie task and request you provide your
> valuable
> > > > > feedback.
> > > > >
> > > > > KIP:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-850%3A+REST+API+for+filtering+Connector+plugins+by+type
> > > > > Initial PR: https://github.com/apache/kafka/pull/12330
> > > > >
> > > > > Though https://issues.apache.org/jira/browse/KAFKA-4279 was for an
> > > > > endpoint
> > > > > to list converter plugins, I propose that we implement a generic
> > > "filter"
> > > > > API which will list connector plugins matched by type (one of
> > > > > SOURCE,SINK,CONVERTER,HEADER_CONVERTER).
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Ashwin
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-850: REST API for filtering Connector plugins by type
Posted by Ashwin <ap...@confluent.io.INVALID>.
Hi Chris,
Sure, if JIRA is obsolete and does not add value, I am fine with abandoning
this change.
I will close the JIRA as well.
Thanks for helping out!
Ashwin
On Fri, Jun 24, 2022 at 4:23 AM Chris Egerton <fe...@gmail.com>
wrote:
> Hi Ashwin,
>
> Thanks for the clarification, which is certainly useful. However, it's not
> an actual use case. What will users try to do now that would have been
> difficult/impossible in the future because of this ambiguity? I ask because
> the Jira ticket linked in the KIP is very old and may be obsolete, and
> there seems to be a much simpler way to address the edge case of a class
> implementing multiple plugin interfaces: we can just expose each type in
> the REST API, instead of picking one at random.
>
> Cheers,
>
> Chris
>
> On Thu, Jun 23, 2022 at 12:55 AM Ashwin <ap...@confluent.io.invalid>
> wrote:
>
> > Hi Chris,
> >
> > Thanks for the quick response.
> >
> > > It seems pretty straightforward to get this via client-side filtering
> >
> > I assume that you meant GET /collector-plugins + jq filtering on type
> > attribute.
> >
> > One issue that I see with that approach is that we cannot filter plugins
> if
> > they implement more than one of Converter, HeaderConverter, Sink, Source
> > interfaces.
> >
> > The type field in PluginDesc field is set based on the order of enum
> values
> > in PluginType -
> >
> >
> https://github.com/apache/kafka/blob/22534424d22e5a4d4d512b0ac1b5f312b867a5bf/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/PluginType.java#L49
> > .
> >
> > Eg: org.apache.kafka.connect.json.JsonConverter implements both Converter
> > and HeaderConverter interfaces, but the type value is always set to
> > Converter as that value is defined first in PluginType enum.
> >
> > The new API will list JsonConverter when we query either for
> > HeaderConverter or Converter plugin types.
> >
> > Thanks,
> > Ashwin
> >
> >
> > On Thu, Jun 23, 2022 at 12:03 AM Chris Egerton <fe...@gmail.com>
> > wrote:
> >
> > > Hi Ashwin,
> > >
> > > Can you provide a use case for this new behavior? It seems pretty
> > > straightforward to get this via client-side filtering, I'm a little
> > > skeptical that we need to implement it server-side.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Wed, Jun 22, 2022 at 4:25 AM Ashwin <ap...@confluent.io.invalid>
> > > wrote:
> > >
> > > > Hello Folks,
> > > >
> > > > I have picked up a newbie task and request you provide your valuable
> > > > feedback.
> > > >
> > > > KIP:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-850%3A+REST+API+for+filtering+Connector+plugins+by+type
> > > > Initial PR: https://github.com/apache/kafka/pull/12330
> > > >
> > > > Though https://issues.apache.org/jira/browse/KAFKA-4279 was for an
> > > > endpoint
> > > > to list converter plugins, I propose that we implement a generic
> > "filter"
> > > > API which will list connector plugins matched by type (one of
> > > > SOURCE,SINK,CONVERTER,HEADER_CONVERTER).
> > > >
> > > >
> > > > Thanks,
> > > > Ashwin
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-850: REST API for filtering Connector plugins by type
Posted by Chris Egerton <fe...@gmail.com>.
Hi Ashwin,
Thanks for the clarification, which is certainly useful. However, it's not
an actual use case. What will users try to do now that would have been
difficult/impossible in the future because of this ambiguity? I ask because
the Jira ticket linked in the KIP is very old and may be obsolete, and
there seems to be a much simpler way to address the edge case of a class
implementing multiple plugin interfaces: we can just expose each type in
the REST API, instead of picking one at random.
Cheers,
Chris
On Thu, Jun 23, 2022 at 12:55 AM Ashwin <ap...@confluent.io.invalid>
wrote:
> Hi Chris,
>
> Thanks for the quick response.
>
> > It seems pretty straightforward to get this via client-side filtering
>
> I assume that you meant GET /collector-plugins + jq filtering on type
> attribute.
>
> One issue that I see with that approach is that we cannot filter plugins if
> they implement more than one of Converter, HeaderConverter, Sink, Source
> interfaces.
>
> The type field in PluginDesc field is set based on the order of enum values
> in PluginType -
>
> https://github.com/apache/kafka/blob/22534424d22e5a4d4d512b0ac1b5f312b867a5bf/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/PluginType.java#L49
> .
>
> Eg: org.apache.kafka.connect.json.JsonConverter implements both Converter
> and HeaderConverter interfaces, but the type value is always set to
> Converter as that value is defined first in PluginType enum.
>
> The new API will list JsonConverter when we query either for
> HeaderConverter or Converter plugin types.
>
> Thanks,
> Ashwin
>
>
> On Thu, Jun 23, 2022 at 12:03 AM Chris Egerton <fe...@gmail.com>
> wrote:
>
> > Hi Ashwin,
> >
> > Can you provide a use case for this new behavior? It seems pretty
> > straightforward to get this via client-side filtering, I'm a little
> > skeptical that we need to implement it server-side.
> >
> > Cheers,
> >
> > Chris
> >
> > On Wed, Jun 22, 2022 at 4:25 AM Ashwin <ap...@confluent.io.invalid>
> > wrote:
> >
> > > Hello Folks,
> > >
> > > I have picked up a newbie task and request you provide your valuable
> > > feedback.
> > >
> > > KIP:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-850%3A+REST+API+for+filtering+Connector+plugins+by+type
> > > Initial PR: https://github.com/apache/kafka/pull/12330
> > >
> > > Though https://issues.apache.org/jira/browse/KAFKA-4279 was for an
> > > endpoint
> > > to list converter plugins, I propose that we implement a generic
> "filter"
> > > API which will list connector plugins matched by type (one of
> > > SOURCE,SINK,CONVERTER,HEADER_CONVERTER).
> > >
> > >
> > > Thanks,
> > > Ashwin
> > >
> >
>
Re: [DISCUSS] KIP-850: REST API for filtering Connector plugins by type
Posted by Ashwin <ap...@confluent.io.INVALID>.
Hi Chris,
Thanks for the quick response.
> It seems pretty straightforward to get this via client-side filtering
I assume that you meant GET /collector-plugins + jq filtering on type
attribute.
One issue that I see with that approach is that we cannot filter plugins if
they implement more than one of Converter, HeaderConverter, Sink, Source
interfaces.
The type field in PluginDesc field is set based on the order of enum values
in PluginType -
https://github.com/apache/kafka/blob/22534424d22e5a4d4d512b0ac1b5f312b867a5bf/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/PluginType.java#L49
.
Eg: org.apache.kafka.connect.json.JsonConverter implements both Converter
and HeaderConverter interfaces, but the type value is always set to
Converter as that value is defined first in PluginType enum.
The new API will list JsonConverter when we query either for
HeaderConverter or Converter plugin types.
Thanks,
Ashwin
On Thu, Jun 23, 2022 at 12:03 AM Chris Egerton <fe...@gmail.com>
wrote:
> Hi Ashwin,
>
> Can you provide a use case for this new behavior? It seems pretty
> straightforward to get this via client-side filtering, I'm a little
> skeptical that we need to implement it server-side.
>
> Cheers,
>
> Chris
>
> On Wed, Jun 22, 2022 at 4:25 AM Ashwin <ap...@confluent.io.invalid>
> wrote:
>
> > Hello Folks,
> >
> > I have picked up a newbie task and request you provide your valuable
> > feedback.
> >
> > KIP:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-850%3A+REST+API+for+filtering+Connector+plugins+by+type
> > Initial PR: https://github.com/apache/kafka/pull/12330
> >
> > Though https://issues.apache.org/jira/browse/KAFKA-4279 was for an
> > endpoint
> > to list converter plugins, I propose that we implement a generic "filter"
> > API which will list connector plugins matched by type (one of
> > SOURCE,SINK,CONVERTER,HEADER_CONVERTER).
> >
> >
> > Thanks,
> > Ashwin
> >
>
Re: [DISCUSS] KIP-850: REST API for filtering Connector plugins by type
Posted by Chris Egerton <fe...@gmail.com>.
Hi Ashwin,
Can you provide a use case for this new behavior? It seems pretty
straightforward to get this via client-side filtering, I'm a little
skeptical that we need to implement it server-side.
Cheers,
Chris
On Wed, Jun 22, 2022 at 4:25 AM Ashwin <ap...@confluent.io.invalid> wrote:
> Hello Folks,
>
> I have picked up a newbie task and request you provide your valuable
> feedback.
>
> KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-850%3A+REST+API+for+filtering+Connector+plugins+by+type
> Initial PR: https://github.com/apache/kafka/pull/12330
>
> Though https://issues.apache.org/jira/browse/KAFKA-4279 was for an
> endpoint
> to list converter plugins, I propose that we implement a generic "filter"
> API which will list connector plugins matched by type (one of
> SOURCE,SINK,CONVERTER,HEADER_CONVERTER).
>
>
> Thanks,
> Ashwin
>