What happened to KIP-158?

[Tom Bentley <tb...@redhat.com>]

Hi,

KIP-158 proposed a way to allow source connectors a way to configure their
topics prior to creation. This was discussed, but the vote didn't get
enough commiter votes at the time (though I think a couple of the +1s came
from people who are now committers).

Meanwhile in KIP-382 (MirrorMaker 2.0) Ryanne made a throwaway comment
about possibly proposing a KIP which would provide a generic way for Kafka
Connect to expose AdminClient functionality to connectors, but I don't
think that KIP was ever created.

Is there a current plan about how to bring this functionality to
connectors? If not what's the best way forward, proceed with KIP-158, or
try for something more generic such as could be used by MM2?

Kind regards,

Tom

Re: What happened to KIP-158?

[Konstantine Karantasis <ko...@confluent.io>]

Hi Tom.

Good timing for your question.
I'll be revisiting KIP-158 and I'll bring an updated proposal for
discussion this quarter.
As always, it'll be ideal to strike a good balance between flexibility and
conciseness on what we expose to Connect's developers and operators.

Stay tuned.
Konstantine


On Mon, Nov 25, 2019 at 9:48 AM Ryanne Dolan <ry...@gmail.com> wrote:

> Hey Tom. The approach I ended up taking with MM2 is to construct an
> AdminClient based on properties in the connector config, which requires
> that such properties exist in two places (the worker config and the
> connector config). I still believe that, ideally, a connector implementer
> should have access to an admin client based on the worker config, or
> perhaps just the properties (so that an admin client can be constructed).
> Otherwise, as you point out, there is no way for a connector to control
> topics without this workaround.
>
> Ryanne
>
> On Mon, Nov 25, 2019, 5:18 AM Tom Bentley <tb...@redhat.com> wrote:
>
> > Hi,
> >
> > KIP-158 proposed a way to allow source connectors a way to configure
> their
> > topics prior to creation. This was discussed, but the vote didn't get
> > enough commiter votes at the time (though I think a couple of the +1s
> came
> > from people who are now committers).
> >
> > Meanwhile in KIP-382 (MirrorMaker 2.0) Ryanne made a throwaway comment
> > about possibly proposing a KIP which would provide a generic way for
> Kafka
> > Connect to expose AdminClient functionality to connectors, but I don't
> > think that KIP was ever created.
> >
> > Is there a current plan about how to bring this functionality to
> > connectors? If not what's the best way forward, proceed with KIP-158, or
> > try for something more generic such as could be used by MM2?
> >
> > Kind regards,
> >
> > Tom
> >
>

Re: What happened to KIP-158?

[Ryanne Dolan <ry...@gmail.com>]

Hey Tom. The approach I ended up taking with MM2 is to construct an
AdminClient based on properties in the connector config, which requires
that such properties exist in two places (the worker config and the
connector config). I still believe that, ideally, a connector implementer
should have access to an admin client based on the worker config, or
perhaps just the properties (so that an admin client can be constructed).
Otherwise, as you point out, there is no way for a connector to control
topics without this workaround.

Ryanne

On Mon, Nov 25, 2019, 5:18 AM Tom Bentley <tb...@redhat.com> wrote:

> Hi,
>
> KIP-158 proposed a way to allow source connectors a way to configure their
> topics prior to creation. This was discussed, but the vote didn't get
> enough commiter votes at the time (though I think a couple of the +1s came
> from people who are now committers).
>
> Meanwhile in KIP-382 (MirrorMaker 2.0) Ryanne made a throwaway comment
> about possibly proposing a KIP which would provide a generic way for Kafka
> Connect to expose AdminClient functionality to connectors, but I don't
> think that KIP was ever created.
>
> Is there a current plan about how to bring this functionality to
> connectors? If not what's the best way forward, proceed with KIP-158, or
> try for something more generic such as could be used by MM2?
>
> Kind regards,
>
> Tom
>