[DISCUSS] KIP-795: Add public APIs for AbstractCoordinator

["Hector Geraldino (BLOOMBERG/ 919 3RD A)" <hg...@bloomberg.net>]

Hi Kafka devs,

I would like to start the discussion of  KIP-795: Add public APIs for AbstractCoordinator

https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator

Looking forward for some feedback from the community.

Regards,
Hector

Re: [DISCUSS] KIP-795: Add public APIs for AbstractCoordinator

[Tom Bentley <tb...@redhat.com>]

Hi Hector,

Thanks for the KIP.

At a high level, I think the question to be answered by the community is
"Should Kafka really be providing this kind of cluster management API?".
While Kafka clients need this to provide their functionality it's a
different thing to expose that as a public API of the project, which is
otherwise about providing a distributed event log/data streaming
platform/<insert your favourite definition of Kafka here>. Having a public
API brings a significant commitment for API compatibility, which could
impair the ability of the project to change the API in order to make
improvements to the Kafka clients. The current AbstractCoordinator not
being a supported API means we don't currently have to reason about
compatibility here. So I think it would help the motivation section of the
KIP to describe in a bit more detail the use case(s) you have for
implementing your own coordinators. For example, are these applications
using Kafka otherwise, or just to leverage this API? And what alternatives
to implementing your own coordinators did you consider, and why did you
reject them?

From a technical point of view, there are a number of issues I think would
need addressing in order to do something like this:

1. There probably ought to be a way to ensure that protocolTypes() don't
collide, or at least reduce the chances of a collision. While probably
unlikely in practice the consequences of different protocols having the
same name could be pretty confusing to debug.
2. JoinGroupRequestData and JoinGroupResponseData are not public classes
(none of the *RequestData or *ResponseData classes are, intentionally), so
there would have to be an abstraction for them.
3. It's all well and good having an interface that anyone can implement,
but there is no supported Kafka API which takes an instance as a parameter
(i.e. where do you plug your implementation in without having to use a
bunch of other non-public Kafka APIs?) I assume in your current usage
you're having to make use of other non-supported client APIs to make use of
your coordinator. The KIP isn't really a complete piece of work without a
way to use a custom implementation, in my opinion. It would be confusing if
it looked like we were encouraging people to use those other non-supported
APIs by making this coordinator public.

Kind regards,

Tom




On Mon, Nov 8, 2021 at 2:01 PM Hector Geraldino (BLOOMBERG/ 919 3RD A) <
hgeraldino@bloomberg.net> wrote:

> Hi Kafka devs,
>
> I would like to start the discussion of  KIP-795: Add public APIs for
> AbstractCoordinator
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-795%3A+Add+public+APIs+for+AbstractCoordinator
>
> Looking forward for some feedback from the community.
>
> Regards,
> Hector