You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Ismael Juma <is...@juma.me.uk> on 2023/01/03 16:17:24 UTC
[DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Hi all,
I would like to start a discussion regarding the removal of very old client
protocol API versions in Apache Kafka 4.0 to improve maintainability &
supportability of Kafka. Please take a look at the proposal:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0
Ismael
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Jose,
I updated the KIP to include a new metric for deprecated request api
versions and also a new attribute in the request log to make it easy to
find such entries,
Thanks,
Ismael
On Thu, Jan 12, 2023 at 1:03 AM Ismael Juma <is...@juma.me.uk> wrote:
> Hi Jose,
>
> I think it's reasonable to add more user-friendly metrics as you
> described. I'll update the KIP soon with that. I'll try to define them in a
> way where they track deprecated protocols for the next major release. That
> way, they can be useful even after AK 4.0 is released.
>
> Ismael
>
> On Wed, Jan 11, 2023 at 12:34 PM José Armando García Sancio
> <js...@confluent.io.invalid> wrote:
>
>> Thanks Ismael.
>>
>> > The following metrics are used to determine both questions:
>> > >
>> > > - Client name and version:
>> > >
>> kafka.server:clientSoftwareName=(client-software-name),clientSoftwareVersion=(client-software-version),listener=(listener),networkProcessor=(processor-index),type=(type)
>> > > - Request name and version:
>> > >
>> kafka.network:type=RequestMetrics,name=RequestsPerSec,request=(api-name),version=(api-version)}
>> > >
>> > >
>> > Are you suggesting that this is too complicated and hence we should add
>> a
>> > metric that tracks AK 4.0 support explicitly?
>>
>> Correct. It doesn't look trivial for the users to implement this check
>> against the RequestMetrics. I was wondering if it is worth it for
>> Kafka to implement this for them and expose a simple metric that they
>> can check.
>>
>> --
>> -José
>>
>
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Jose,
I think it's reasonable to add more user-friendly metrics as you described.
I'll update the KIP soon with that. I'll try to define them in a way where
they track deprecated protocols for the next major release. That way, they
can be useful even after AK 4.0 is released.
Ismael
On Wed, Jan 11, 2023 at 12:34 PM José Armando García Sancio
<js...@confluent.io.invalid> wrote:
> Thanks Ismael.
>
> > The following metrics are used to determine both questions:
> > >
> > > - Client name and version:
> > >
> kafka.server:clientSoftwareName=(client-software-name),clientSoftwareVersion=(client-software-version),listener=(listener),networkProcessor=(processor-index),type=(type)
> > > - Request name and version:
> > >
> kafka.network:type=RequestMetrics,name=RequestsPerSec,request=(api-name),version=(api-version)}
> > >
> > >
> > Are you suggesting that this is too complicated and hence we should add a
> > metric that tracks AK 4.0 support explicitly?
>
> Correct. It doesn't look trivial for the users to implement this check
> against the RequestMetrics. I was wondering if it is worth it for
> Kafka to implement this for them and expose a simple metric that they
> can check.
>
> --
> -José
>
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Posted by José Armando García Sancio <js...@confluent.io.INVALID>.
Thanks Ismael.
> The following metrics are used to determine both questions:
> >
> > - Client name and version:
> > kafka.server:clientSoftwareName=(client-software-name),clientSoftwareVersion=(client-software-version),listener=(listener),networkProcessor=(processor-index),type=(type)
> > - Request name and version:
> > kafka.network:type=RequestMetrics,name=RequestsPerSec,request=(api-name),version=(api-version)}
> >
> >
> Are you suggesting that this is too complicated and hence we should add a
> metric that tracks AK 4.0 support explicitly?
Correct. It doesn't look trivial for the users to implement this check
against the RequestMetrics. I was wondering if it is worth it for
Kafka to implement this for them and expose a simple metric that they
can check.
--
-José
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Jose,
The KIP describes a couple of existing metrics that can be used to answer
that question:
The following metrics are used to determine both questions:
>
> - Client name and version:
> kafka.server:clientSoftwareName=(client-software-name),clientSoftwareVersion=(client-software-version),listener=(listener),networkProcessor=(processor-index),type=(type)
> - Request name and version:
> kafka.network:type=RequestMetrics,name=RequestsPerSec,request=(api-name),version=(api-version)}
>
>
Are you suggesting that this is too complicated and hence we should add a
metric that tracks AK 4.0 support explicitly?
Ismael
On Tue, Jan 10, 2023 at 10:33 AM José Armando García Sancio
<js...@confluent.io.invalid> wrote:
> Hi Ismael,
>
> Thanks for the improvement.
>
> I haven't been following the discussion in detail so it is possible
> that this was already discussed.
>
> If a user upgrades to Apache Kafka 4.0 it is possible for some of
> their clients to stop working because the request's version would not
> be a version that Kafka 4.0 supports. Should we add metrics or some
> other mechanism that the user can monitor to determine if it is safe
> to upgrade Kafka to 4.0. For example, the metrics could report if a
> Kafka broker received a request or response in the past 7 days that
> would not be supported by Kafka 4.0.
>
> Thanks
> --
> -José
>
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Posted by José Armando García Sancio <js...@confluent.io.INVALID>.
Hi Ismael,
Thanks for the improvement.
I haven't been following the discussion in detail so it is possible
that this was already discussed.
If a user upgrades to Apache Kafka 4.0 it is possible for some of
their clients to stop working because the request's version would not
be a version that Kafka 4.0 supports. Should we add metrics or some
other mechanism that the user can monitor to determine if it is safe
to upgrade Kafka to 4.0. For example, the metrics could report if a
Kafka broker received a request or response in the past 7 days that
would not be supported by Kafka 4.0.
Thanks
--
-José
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Viktor,
Thanks for the feedback. See inline.
I was gonna ask about whether you think it would be useful to automate
> protocol deprecation but I see you'll have a separate KIP for that so I'll
> just wait :).
>
With regards to the follow-up KIP, if you are interested in submitting
that, please go ahead. My high level idea was something like the following:
Starting with Apache Kafka 5.0, the broker only keeps the latest protocol
API version from the first release in previous major version series or a
minor release that is at least n years old, whichever is oldest. Reasonable
values for `n` could be `3` or `4`.
Let's assume `n` is 3 and we release Apache Kafka 5.0 4 years after Apache
Kafka 4.0, then we'd keep the highest protocol API version for each
protocol from Apache Kafka 4.0. However, if Apache Kafka 5.0 was released 2
years after Apache Kafka 4.0, then we'd pick the relevant 3.x release as
the baseline instead.
If you do decide to write the KIP, feel free to suggest what you think is
best, of course. With that said, let's leave further discussions on this
future KIP idea to the future KIP thread. :)
I see that in multiple places clients have older than baseline
> implementation. Would it make sense in your opinion to also document that
> those protocols below the baseline will be deprecated in 4.0 and possibly
> go away in next versions (with maybe the deprecation/removal automation)
>
I considered that, but I thought it would be handled more cleanly by the
mechanical policy of the future KIP while this KIP focuses only on what
happens in 4.0.
Ismael
On Wed, Jan 4, 2023 at 7:42 AM Viktor Somogyi-Vass
<vi...@cloudera.com.invalid> wrote:
> Hi Ismael,
>
> I think this is a good idea, it can also simplify the code quite a bit.
> I was gonna ask about whether you think it would be useful to automate
> protocol deprecation but I see you'll have a separate KIP for that so I'll
> just wait :).
>
> I see that in multiple places clients have older than baseline
> implementation. Would it make sense in your opinion to also document that
> those protocols below the baseline will be deprecated in 4.0 and possibly
> go away in next versions (with maybe the deprecation/removal automation)?
>
> Viktor
>
> On Tue, Jan 3, 2023 at 5:18 PM Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi all,
> >
> > I would like to start a discussion regarding the removal of very old
> client
> > protocol API versions in Apache Kafka 4.0 to improve maintainability &
> > supportability of Kafka. Please take a look at the proposal:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0
> >
> > Ismael
> >
>
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Posted by Viktor Somogyi-Vass <vi...@cloudera.com.INVALID>.
Hi Ismael,
I think this is a good idea, it can also simplify the code quite a bit.
I was gonna ask about whether you think it would be useful to automate
protocol deprecation but I see you'll have a separate KIP for that so I'll
just wait :).
I see that in multiple places clients have older than baseline
implementation. Would it make sense in your opinion to also document that
those protocols below the baseline will be deprecated in 4.0 and possibly
go away in next versions (with maybe the deprecation/removal automation)?
Viktor
On Tue, Jan 3, 2023 at 5:18 PM Ismael Juma <is...@juma.me.uk> wrote:
> Hi all,
>
> I would like to start a discussion regarding the removal of very old client
> protocol API versions in Apache Kafka 4.0 to improve maintainability &
> supportability of Kafka. Please take a look at the proposal:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0
>
> Ismael
>
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Posted by Ismael Juma <is...@juma.me.uk>.
I started a vote thread for this since I addressed the comments so far and
it doesn't seem like there were any major concerns.
Ismael
On Tue, Jan 3, 2023 at 8:17 AM Ismael Juma <is...@juma.me.uk> wrote:
> Hi all,
>
> I would like to start a discussion regarding the removal of very old
> client protocol API versions in Apache Kafka 4.0 to improve maintainability
> & supportability of Kafka. Please take a look at the proposal:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0
>
> Ismael
>
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Jun,
I updated the KIP to include the inter-broker protocol version deprecation.
Ismael
On Mon, Jan 9, 2023 at 3:53 PM Jun Rao <ju...@confluent.io.invalid> wrote:
> Hi, Ismael,
>
> Thanks for the reply.
>
> 2. Yes, that makes sense.
>
> Jun
>
> On Mon, Jan 9, 2023 at 11:26 AM Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Jun,
> >
> > Thanks for the feedback.
> >
> > 1. Good question. I was looking at the IBP issue as more of an inter
> broker
> > protocol communication thing and hence I thought we'd cover that as part
> of
> > the KIP that modernizes that aspect (yet to be written). I filed a JIRA
> > recently that shares some of the thinking:
> > https://issues.apache.org/jira/browse/KAFKA-14552. I was hoping we would
> > tackle the details of that as part of the zk deprecation/removal. You're
> > right though that we need to deprecate IBP before 2.1 as part of this KIP
> > since we are removing some of the protocol API versions that would be
> > required for that to work. I'll update the KIP to include this.
> >
> > 2. I believe they are all introduced after 2.1, yes. This is what I used
> as
> > the source for 2.1:
> >
> >
> https://github.com/apache/kafka/blob/2.1/clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java#L189
> > .
> > Note that I excluded inter broker protocol APIs like LeaderAndIsr,
> > StopReplica and others for the reason I explained in `1`. Does that make
> > sense?
> >
> > Ismael
> >
> > On Mon, Jan 9, 2023 at 11:01 AM Jun Rao <ju...@confluent.io.invalid>
> wrote:
> >
> > > Hi, Ismael,
> > >
> > > Thanks for the KIP. A couple of comments.
> > >
> > > 1. Should we deprecate IBPs before 2.1 too?
> > >
> > > 2. There seem to be more requests than those listed in the KIP. Are
> they
> > > all introduced after 2.1?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > >
> > > On Sat, Jan 7, 2023 at 5:13 PM Ismael Juma <is...@juma.me.uk> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I updated the KIP to make it explicit that the Java clients shipped
> > with
> > > > Apache Kafka 4.0 would only be guaranteed to work with brokers
> running
> > > > Apache Kafka 2.1 or newer, since they too would no longer have access
> > to
> > > > the removed protocol APIs. Thanks to Jason for asking about this
> aspect
> > > > offline.
> > > >
> > > > I will leave this KIP open for discussion for another couple or weeks
> > or
> > > > so. Please do share your thoughts given the high impact of the
> change.
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Jan 3, 2023 at 8:17 AM Ismael Juma <is...@juma.me.uk>
> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I would like to start a discussion regarding the removal of very
> old
> > > > > client protocol API versions in Apache Kafka 4.0 to improve
> > > > maintainability
> > > > > & supportability of Kafka. Please take a look at the proposal:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0
> > > > >
> > > > > Ismael
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Posted by Jun Rao <ju...@confluent.io.INVALID>.
Hi, Ismael,
Thanks for the reply.
2. Yes, that makes sense.
Jun
On Mon, Jan 9, 2023 at 11:26 AM Ismael Juma <is...@juma.me.uk> wrote:
> Hi Jun,
>
> Thanks for the feedback.
>
> 1. Good question. I was looking at the IBP issue as more of an inter broker
> protocol communication thing and hence I thought we'd cover that as part of
> the KIP that modernizes that aspect (yet to be written). I filed a JIRA
> recently that shares some of the thinking:
> https://issues.apache.org/jira/browse/KAFKA-14552. I was hoping we would
> tackle the details of that as part of the zk deprecation/removal. You're
> right though that we need to deprecate IBP before 2.1 as part of this KIP
> since we are removing some of the protocol API versions that would be
> required for that to work. I'll update the KIP to include this.
>
> 2. I believe they are all introduced after 2.1, yes. This is what I used as
> the source for 2.1:
>
> https://github.com/apache/kafka/blob/2.1/clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java#L189
> .
> Note that I excluded inter broker protocol APIs like LeaderAndIsr,
> StopReplica and others for the reason I explained in `1`. Does that make
> sense?
>
> Ismael
>
> On Mon, Jan 9, 2023 at 11:01 AM Jun Rao <ju...@confluent.io.invalid> wrote:
>
> > Hi, Ismael,
> >
> > Thanks for the KIP. A couple of comments.
> >
> > 1. Should we deprecate IBPs before 2.1 too?
> >
> > 2. There seem to be more requests than those listed in the KIP. Are they
> > all introduced after 2.1?
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Sat, Jan 7, 2023 at 5:13 PM Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi all,
> > >
> > > I updated the KIP to make it explicit that the Java clients shipped
> with
> > > Apache Kafka 4.0 would only be guaranteed to work with brokers running
> > > Apache Kafka 2.1 or newer, since they too would no longer have access
> to
> > > the removed protocol APIs. Thanks to Jason for asking about this aspect
> > > offline.
> > >
> > > I will leave this KIP open for discussion for another couple or weeks
> or
> > > so. Please do share your thoughts given the high impact of the change.
> > >
> > > Ismael
> > >
> > > On Tue, Jan 3, 2023 at 8:17 AM Ismael Juma <is...@juma.me.uk> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I would like to start a discussion regarding the removal of very old
> > > > client protocol API versions in Apache Kafka 4.0 to improve
> > > maintainability
> > > > & supportability of Kafka. Please take a look at the proposal:
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0
> > > >
> > > > Ismael
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Jun,
Thanks for the feedback.
1. Good question. I was looking at the IBP issue as more of an inter broker
protocol communication thing and hence I thought we'd cover that as part of
the KIP that modernizes that aspect (yet to be written). I filed a JIRA
recently that shares some of the thinking:
https://issues.apache.org/jira/browse/KAFKA-14552. I was hoping we would
tackle the details of that as part of the zk deprecation/removal. You're
right though that we need to deprecate IBP before 2.1 as part of this KIP
since we are removing some of the protocol API versions that would be
required for that to work. I'll update the KIP to include this.
2. I believe they are all introduced after 2.1, yes. This is what I used as
the source for 2.1:
https://github.com/apache/kafka/blob/2.1/clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java#L189.
Note that I excluded inter broker protocol APIs like LeaderAndIsr,
StopReplica and others for the reason I explained in `1`. Does that make
sense?
Ismael
On Mon, Jan 9, 2023 at 11:01 AM Jun Rao <ju...@confluent.io.invalid> wrote:
> Hi, Ismael,
>
> Thanks for the KIP. A couple of comments.
>
> 1. Should we deprecate IBPs before 2.1 too?
>
> 2. There seem to be more requests than those listed in the KIP. Are they
> all introduced after 2.1?
>
> Thanks,
>
> Jun
>
>
>
> On Sat, Jan 7, 2023 at 5:13 PM Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi all,
> >
> > I updated the KIP to make it explicit that the Java clients shipped with
> > Apache Kafka 4.0 would only be guaranteed to work with brokers running
> > Apache Kafka 2.1 or newer, since they too would no longer have access to
> > the removed protocol APIs. Thanks to Jason for asking about this aspect
> > offline.
> >
> > I will leave this KIP open for discussion for another couple or weeks or
> > so. Please do share your thoughts given the high impact of the change.
> >
> > Ismael
> >
> > On Tue, Jan 3, 2023 at 8:17 AM Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi all,
> > >
> > > I would like to start a discussion regarding the removal of very old
> > > client protocol API versions in Apache Kafka 4.0 to improve
> > maintainability
> > > & supportability of Kafka. Please take a look at the proposal:
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0
> > >
> > > Ismael
> > >
> >
>
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Posted by Jun Rao <ju...@confluent.io.INVALID>.
Hi, Ismael,
Thanks for the KIP. A couple of comments.
1. Should we deprecate IBPs before 2.1 too?
2. There seem to be more requests than those listed in the KIP. Are they
all introduced after 2.1?
Thanks,
Jun
On Sat, Jan 7, 2023 at 5:13 PM Ismael Juma <is...@juma.me.uk> wrote:
> Hi all,
>
> I updated the KIP to make it explicit that the Java clients shipped with
> Apache Kafka 4.0 would only be guaranteed to work with brokers running
> Apache Kafka 2.1 or newer, since they too would no longer have access to
> the removed protocol APIs. Thanks to Jason for asking about this aspect
> offline.
>
> I will leave this KIP open for discussion for another couple or weeks or
> so. Please do share your thoughts given the high impact of the change.
>
> Ismael
>
> On Tue, Jan 3, 2023 at 8:17 AM Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi all,
> >
> > I would like to start a discussion regarding the removal of very old
> > client protocol API versions in Apache Kafka 4.0 to improve
> maintainability
> > & supportability of Kafka. Please take a look at the proposal:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0
> >
> > Ismael
> >
>
Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0
Posted by Ismael Juma <is...@juma.me.uk>.
Hi all,
I updated the KIP to make it explicit that the Java clients shipped with
Apache Kafka 4.0 would only be guaranteed to work with brokers running
Apache Kafka 2.1 or newer, since they too would no longer have access to
the removed protocol APIs. Thanks to Jason for asking about this aspect
offline.
I will leave this KIP open for discussion for another couple or weeks or
so. Please do share your thoughts given the high impact of the change.
Ismael
On Tue, Jan 3, 2023 at 8:17 AM Ismael Juma <is...@juma.me.uk> wrote:
> Hi all,
>
> I would like to start a discussion regarding the removal of very old
> client protocol API versions in Apache Kafka 4.0 to improve maintainability
> & supportability of Kafka. Please take a look at the proposal:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0
>
> Ismael
>