You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Kamal Chandraprakash <ka...@gmail.com> on 2025/02/03 17:35:19 UTC
Re: Tagged fields constraint to start from tagId: 0
Bumping up on this thread.
In a forked Kafka repo, if we want to introduce a new tagged field. Which
approach should be taken?
Is it fine to relax the constraint in StructSpec
<https://github.com/apache/kafka/blob/trunk/generator/src/main/java/org/apache/kafka/message/StructSpec.java#L76>
to
start with a very large tag-id like 10000 for custom
tagged fields to not clash with future tagged fields being introduced
upstream? TIA.
Thanks,
Kamal
On Thu, Jan 23, 2025 at 11:47 PM Kamal Chandraprakash <
kamal.chandraprakash@gmail.com> wrote:
> Hi all,
>
> In KIP-482
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields#KIP482:TheKafkaProtocolshouldSupportOptionalTaggedFields-FlexibleVersions>,
> we have introduced support for optional tagged fields when the
> API-message supports "flexibleVersions" feature. One of the motivations of
> the KIP is to support attaching extra information to a message without
> requiring a version bump for everyone who has to read that message.
>
> In StructSpec
> <https://github.com/apache/kafka/blob/trunk/generator/src/main/java/org/apache/kafka/message/StructSpec.java#L76>,
> there is a constraint to start the tagId of the tagged field from 0 and
> should be contiguous. Once the user introduces a custom taggedField that
> starts from 0, there is a risk of protocol divergence between the server
> and client when the upstream/Apache introduces a new taggedField in future
> that starts from 0.
>
> Once a tag-id gets introduced, it cannot be used for anything else. We
> want to propagate a custom trace-id in the API-message between
> server/client and want to use a large tagId like 10001 (2 bytes varint) to
> avoid conflict with the future API message specification.
>
> We would like to understand the reason behind the tagId constraint to
> start from 0 and contiguous. One use-case of having this constraint ensures
> that the tag-id starts from 0 and won't be random. If it is random, then
> the user won't be sure in defining the unique tagged-ids for their
> user-defined fields.
>
> I’d appreciate it if you could take a look and share your thoughts or
> feedback.
>
> Thanks,
> Kamal
>
>
>
>
Re: Tagged fields constraint to start from tagId: 0
Posted by Kamal Chandraprakash <ka...@gmail.com>.
Got it. Thanks for clearing that up!
--
Kamal
On Tue, Feb 4, 2025 at 4:40 PM David Jacot <dj...@confluent.io.invalid>
wrote:
> Hi Kamal,
>
> It is fine to relax the constraint in StructSpec for your internal tags.
> For instance, you could allow your internal tags to start from id 10000. It
> is very unlikely that we will ever have tags as part of the official
> protocol reaching that.
>
> Best,
> David
>
> On Mon, Feb 3, 2025 at 6:36 PM Kamal Chandraprakash <
> kamal.chandraprakash@gmail.com> wrote:
>
> > Bumping up on this thread.
> >
> > In a forked Kafka repo, if we want to introduce a new tagged field. Which
> > approach should be taken?
> > Is it fine to relax the constraint in StructSpec
> > <
> >
> https://github.com/apache/kafka/blob/trunk/generator/src/main/java/org/apache/kafka/message/StructSpec.java#L76
> > >
> > to
> > start with a very large tag-id like 10000 for custom
> > tagged fields to not clash with future tagged fields being introduced
> > upstream? TIA.
> >
> > Thanks,
> > Kamal
> >
> > On Thu, Jan 23, 2025 at 11:47 PM Kamal Chandraprakash <
> > kamal.chandraprakash@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > In KIP-482
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields#KIP482:TheKafkaProtocolshouldSupportOptionalTaggedFields-FlexibleVersions
> > >,
> > > we have introduced support for optional tagged fields when the
> > > API-message supports "flexibleVersions" feature. One of the motivations
> > of
> > > the KIP is to support attaching extra information to a message without
> > > requiring a version bump for everyone who has to read that message.
> > >
> > > In StructSpec
> > > <
> >
> https://github.com/apache/kafka/blob/trunk/generator/src/main/java/org/apache/kafka/message/StructSpec.java#L76
> > >,
> > > there is a constraint to start the tagId of the tagged field from 0 and
> > > should be contiguous. Once the user introduces a custom taggedField
> that
> > > starts from 0, there is a risk of protocol divergence between the
> server
> > > and client when the upstream/Apache introduces a new taggedField in
> > future
> > > that starts from 0.
> > >
> > > Once a tag-id gets introduced, it cannot be used for anything else. We
> > > want to propagate a custom trace-id in the API-message between
> > > server/client and want to use a large tagId like 10001 (2 bytes varint)
> > to
> > > avoid conflict with the future API message specification.
> > >
> > > We would like to understand the reason behind the tagId constraint to
> > > start from 0 and contiguous. One use-case of having this constraint
> > ensures
> > > that the tag-id starts from 0 and won't be random. If it is random,
> then
> > > the user won't be sure in defining the unique tagged-ids for their
> > > user-defined fields.
> > >
> > > I’d appreciate it if you could take a look and share your thoughts or
> > > feedback.
> > >
> > > Thanks,
> > > Kamal
> > >
> > >
> > >
> > >
> >
>
Re: Tagged fields constraint to start from tagId: 0
Posted by David Jacot <dj...@confluent.io.INVALID>.
Hi Kamal,
It is fine to relax the constraint in StructSpec for your internal tags.
For instance, you could allow your internal tags to start from id 10000. It
is very unlikely that we will ever have tags as part of the official
protocol reaching that.
Best,
David
On Mon, Feb 3, 2025 at 6:36 PM Kamal Chandraprakash <
kamal.chandraprakash@gmail.com> wrote:
> Bumping up on this thread.
>
> In a forked Kafka repo, if we want to introduce a new tagged field. Which
> approach should be taken?
> Is it fine to relax the constraint in StructSpec
> <
> https://github.com/apache/kafka/blob/trunk/generator/src/main/java/org/apache/kafka/message/StructSpec.java#L76
> >
> to
> start with a very large tag-id like 10000 for custom
> tagged fields to not clash with future tagged fields being introduced
> upstream? TIA.
>
> Thanks,
> Kamal
>
> On Thu, Jan 23, 2025 at 11:47 PM Kamal Chandraprakash <
> kamal.chandraprakash@gmail.com> wrote:
>
> > Hi all,
> >
> > In KIP-482
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields#KIP482:TheKafkaProtocolshouldSupportOptionalTaggedFields-FlexibleVersions
> >,
> > we have introduced support for optional tagged fields when the
> > API-message supports "flexibleVersions" feature. One of the motivations
> of
> > the KIP is to support attaching extra information to a message without
> > requiring a version bump for everyone who has to read that message.
> >
> > In StructSpec
> > <
> https://github.com/apache/kafka/blob/trunk/generator/src/main/java/org/apache/kafka/message/StructSpec.java#L76
> >,
> > there is a constraint to start the tagId of the tagged field from 0 and
> > should be contiguous. Once the user introduces a custom taggedField that
> > starts from 0, there is a risk of protocol divergence between the server
> > and client when the upstream/Apache introduces a new taggedField in
> future
> > that starts from 0.
> >
> > Once a tag-id gets introduced, it cannot be used for anything else. We
> > want to propagate a custom trace-id in the API-message between
> > server/client and want to use a large tagId like 10001 (2 bytes varint)
> to
> > avoid conflict with the future API message specification.
> >
> > We would like to understand the reason behind the tagId constraint to
> > start from 0 and contiguous. One use-case of having this constraint
> ensures
> > that the tag-id starts from 0 and won't be random. If it is random, then
> > the user won't be sure in defining the unique tagged-ids for their
> > user-defined fields.
> >
> > I’d appreciate it if you could take a look and share your thoughts or
> > feedback.
> >
> > Thanks,
> > Kamal
> >
> >
> >
> >
>