You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Colin McCabe <cm...@apache.org> on 2019/09/03 17:16:09 UTC
[VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields
Hi all,
I'd like to start the vote for KIP-482: The Kafka Protocol should Support Optional Tagged Fields.
KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields
Discussion thread here: https://lists.apache.org/thread.html/cdc801ae886491b73ef7efecac7ef81b24382f8b6b025899ee343f7a@%3Cdev.kafka.apache.org%3E
best,
Colin
Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields
Posted by Colin McCabe <cm...@apache.org>.
On Sat, Sep 28, 2019, at 17:49, Magnus Edenhill wrote:
> Den mån 23 sep. 2019 kl 14:42 skrev Colin McCabe <cm...@apache.org>:
>
> > On Fri, Sep 20, 2019, at 18:05, Jun Rao wrote:
> > > 101. We already use varInt in the message format. I assume that the
> > > protocol uses the same varInt representation?
> >
> > It uses a slightly different varint representation. Basically, the
> > difference is that the existing representation uses serpentine encoding to
> > make representing negative numbers more efficient, at the cost of making
> > positive numbers less efficient. Since tags (and lengths) can't be
> > negative, there is no need for serpentine encoding, and we can be more
> > efficient without it.
> >
>
> While I don't see anything technically wrong with the proposed custom
> varint encoding, it does
> come at a price since it prevents client developers from using an existing,
> tested, and optimized zigzag varint implementation,
> and it makes the Kafka protocol more complex by now having 4 ways to encode
> integers.
>
> I'm not strongly opposed, but unless there is an actual efficiency gain in
> using the custom encoding,
> I'd see us using the existing zigzag varint encoding instead, or having the
> protocol semantics state
> that a zero-sized tag value is the same as null, or that omission of a tag
> means null.
>
> //Magnus
>
Hi Magnus,
I don't really think of this as a custom encoding. Protobuf supports both varints that use zigzag and varints that don't, for example. The call the latter "unsigned varints." Zigzag encoding reserves half of the bit patterns for negative integers, and sometimes you know that something isn't going to be negative-- like a length.
There is a big efficiency gain from not using zigzag encoding when we don't need negative numbers. It allows us to encode 2x as many integers before moving to using more bytes. Going from memory without checking the code, it's something like we have to start using a two byte encoding when values get larger than 63 when using zigzag encoding, but we can go up to 127 without it.
It's should be simple to implement both variations. Basically the code for implementing varints with zigzag encoding is:
> public static void writeVarint(int value, DataOutput out) throws IOException {
> writeUnsignedVarint((value << 1) ^ (value >> 31), out);
> }
You can zigzag encode an int with "(value << 1) ^ (value >> 31)" And then you just do the normal variable-length encoding. This is not something that needs a lot of testing or optimization because it's just a single bit-munging step that you either do or don't do, like choosing to negative the int or something. Decoding is similar -- just a simple bit manipulation expression that you either do or omit.
We can't make omission of a tag mean null because a lot of datatypes in the Kafka protocol just aren't nullable. int8, int16, int32, int64, UUID, etc. Even for datatypes that are nullable, the Kafka protocol lets us choose whether null is even an allowed value, let alone the default. Keep in mind that existing fields can be converted to tagged fields in new message versions, so we have to work with the existing semantics.
best,
Colin
Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields
Posted by Magnus Edenhill <ma...@edenhill.se>.
Den mån 23 sep. 2019 kl 14:42 skrev Colin McCabe <cm...@apache.org>:
> On Fri, Sep 20, 2019, at 18:05, Jun Rao wrote:
> > 101. We already use varInt in the message format. I assume that the
> > protocol uses the same varInt representation?
>
> It uses a slightly different varint representation. Basically, the
> difference is that the existing representation uses serpentine encoding to
> make representing negative numbers more efficient, at the cost of making
> positive numbers less efficient. Since tags (and lengths) can't be
> negative, there is no need for serpentine encoding, and we can be more
> efficient without it.
>
While I don't see anything technically wrong with the proposed custom
varint encoding, it does
come at a price since it prevents client developers from using an existing,
tested, and optimized zigzag varint implementation,
and it makes the Kafka protocol more complex by now having 4 ways to encode
integers.
I'm not strongly opposed, but unless there is an actual efficiency gain in
using the custom encoding,
I'd see us using the existing zigzag varint encoding instead, or having the
protocol semantics state
that a zero-sized tag value is the same as null, or that omission of a tag
means null.
//Magnus
Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields
Posted by Colin McCabe <cm...@apache.org>.
On Fri, Sep 20, 2019, at 18:05, Jun Rao wrote:
> Hi, Colin,
>
> Thanks for the KIP. Overall, looks good to me too. A couple of minor
> comments.
>
> 100. Does the tag number need to be 31-bit? It seems that 15-bit could be
> enough.
I think 15 bit would probably be enough, but there is no extra overhead to allowing 31 bits when using a variable-length integer.
>
> 101. We already use varInt in the message format. I assume that the
> protocol uses the same varInt representation?
It uses a slightly different varint representation. Basically, the difference is that the existing representation uses serpentine encoding to make representing negative numbers more efficient, at the cost of making positive numbers less efficient. Since tags (and lengths) can't be negative, there is no need for serpentine encoding, and we can be more efficient without it.
best,
Colin
>
> Jun
>
>
>
>
> On Fri, Sep 6, 2019 at 3:06 PM Colin McCabe <cm...@apache.org> wrote:
>
> > Hi all,
> >
> > The KIP-482 vote has passed. Thanks to everyone who voted or discussed
> > the issue.
> >
> > There were 3 non-binding +1 votes from David Jacot, Jose Armando Garcia
> > Sancio, and Satish Duggana.
> > There were 3 binding +1 votes from Harsha Chintalapani, David Arthur, and
> > Jason Gustafson.
> >
> > best,
> > Colin
> >
> > On Fri, Sep 6, 2019, at 13:36, Jason Gustafson wrote:
> > > +1 Thanks Colin. This is really going to help with compatibility.
> > >
> > > -Jason
> > >
> > > On Wed, Sep 4, 2019 at 1:34 PM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Wed, Sep 4, 2019, at 13:01, Jason Gustafson wrote:
> > > > > Hi Colin,
> > > > >
> > > > > Just a couple questions.
> > > > >
> > > > > 1. I think we discussed that we would do a lazy version bump of all
> > > > > protocols in order to get flexible version support. Can you add that
> > to
> > > > the
> > > > > KIP?
> > > >
> > > > Good point. Added.
> > > >
> > > > > 2. The doc mentions a bump to the request and response header
> > formats to
> > > > > version 1. Currently there is no formal header version. It wasn't
> > clear
> > > > to
> > > > > me if you were suggesting that we create a header version as part of
> > the
> > > > > schema or if this was just an informal way to refer to the header in
> > > > > "flexible version" requests. Can you clarify?
> > > >
> > > > I think we should have a formal header version. However, we can deduce
> > > > which header version we should use based on the apiKey and apiVersion,
> > so
> > > > no changes will be needed to what is sent over the wire.
> > > >
> > > > Having a new header version will let us add new fields to request and
> > > > response headers. In particular, having a flexible header version
> > will let
> > > > us add tagged fields, which will be useful for adding things like a
> > message
> > > > traceID.
> > > > As another example, ThrottleTimeMs would have made more sense in the
> > > > response header than as an addition to every message.
> > > >
> > > > cheers,
> > > > Colin
> > > >
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > > On Wed, Sep 4, 2019 at 8:14 AM David Arthur <mu...@gmail.com>
> > wrote:
> > > > >
> > > > > > +1 binding.
> > > > > >
> > > > > > Thanks for the KIP, Colin!
> > > > > >
> > > > > > -David
> > > > > >
> > > > > > On Wed, Sep 4, 2019 at 5:40 AM Harsha Chintalapani <
> > kafka@harsha.io>
> > > > > > wrote:
> > > > > >
> > > > > > > LGTM. +1 (binding)
> > > > > > > -Harsha
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Sep 04, 2019 at 1:46 AM, Satish Duggana <
> > > > > > satish.duggana@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 (non-binding) Thanks for the nice KIP.
> > > > > > > >
> > > > > > > > You may want to update the KIP saying that optional tagged
> > fields
> > > > do
> > > > > > not
> > > > > > > > support complex types(or structs).
> > > > > > > >
> > > > > > > > On Wed, Sep 4, 2019 at 3:43 AM Jose Armando Garcia Sancio
> > > > > > > > <js...@confluent.io> wrote:
> > > > > > > >
> > > > > > > > +1 (non-binding)
> > > > > > > >
> > > > > > > > Looking forward to this improvement.
> > > > > > > >
> > > > > > > > On Tue, Sep 3, 2019 at 12:49 PM David Jacot <
> > djacot@confluent.io>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > +1 (non-binding)
> > > > > > > >
> > > > > > > > Thank for the KIP. Great addition to the Kafka protocol!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > David
> > > > > > > >
> > > > > > > > Le mar. 3 sept. 2019 à 19:17, Colin McCabe <cm...@apache.org>
> > a
> > > > > > écrit
> > > > > > > :
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I'd like to start the vote for KIP-482: The Kafka Protocol
> > should
> > > > > > Support
> > > > > > > > Optional Tagged Fields.
> > > > > > > >
> > > > > > > > KIP:
> > > > > > > >
> > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > > >
> > KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields
> > > > > > > >
> > > > > > > > Discussion thread here:
> > > > > > > >
> > > > > > > > https://lists.apache.org/thread.html/
> > > > > > > > cdc801ae886491b73ef7efecac7ef81b24382f8b6b025899ee343f7a@
> > > > %3Cdev.kafka.
> > > > > > > > apache.org%3E
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > > --
> > > > > > > > -Jose
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > David Arthur
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields
Posted by Jun Rao <ju...@confluent.io>.
Hi, Colin,
Thanks for the KIP. Overall, looks good to me too. A couple of minor
comments.
100. Does the tag number need to be 31-bit? It seems that 15-bit could be
enough.
101. We already use varInt in the message format. I assume that the
protocol uses the same varInt representation?
Jun
On Fri, Sep 6, 2019 at 3:06 PM Colin McCabe <cm...@apache.org> wrote:
> Hi all,
>
> The KIP-482 vote has passed. Thanks to everyone who voted or discussed
> the issue.
>
> There were 3 non-binding +1 votes from David Jacot, Jose Armando Garcia
> Sancio, and Satish Duggana.
> There were 3 binding +1 votes from Harsha Chintalapani, David Arthur, and
> Jason Gustafson.
>
> best,
> Colin
>
> On Fri, Sep 6, 2019, at 13:36, Jason Gustafson wrote:
> > +1 Thanks Colin. This is really going to help with compatibility.
> >
> > -Jason
> >
> > On Wed, Sep 4, 2019 at 1:34 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Wed, Sep 4, 2019, at 13:01, Jason Gustafson wrote:
> > > > Hi Colin,
> > > >
> > > > Just a couple questions.
> > > >
> > > > 1. I think we discussed that we would do a lazy version bump of all
> > > > protocols in order to get flexible version support. Can you add that
> to
> > > the
> > > > KIP?
> > >
> > > Good point. Added.
> > >
> > > > 2. The doc mentions a bump to the request and response header
> formats to
> > > > version 1. Currently there is no formal header version. It wasn't
> clear
> > > to
> > > > me if you were suggesting that we create a header version as part of
> the
> > > > schema or if this was just an informal way to refer to the header in
> > > > "flexible version" requests. Can you clarify?
> > >
> > > I think we should have a formal header version. However, we can deduce
> > > which header version we should use based on the apiKey and apiVersion,
> so
> > > no changes will be needed to what is sent over the wire.
> > >
> > > Having a new header version will let us add new fields to request and
> > > response headers. In particular, having a flexible header version
> will let
> > > us add tagged fields, which will be useful for adding things like a
> message
> > > traceID.
> > > As another example, ThrottleTimeMs would have made more sense in the
> > > response header than as an addition to every message.
> > >
> > > cheers,
> > > Colin
> > >
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > > On Wed, Sep 4, 2019 at 8:14 AM David Arthur <mu...@gmail.com>
> wrote:
> > > >
> > > > > +1 binding.
> > > > >
> > > > > Thanks for the KIP, Colin!
> > > > >
> > > > > -David
> > > > >
> > > > > On Wed, Sep 4, 2019 at 5:40 AM Harsha Chintalapani <
> kafka@harsha.io>
> > > > > wrote:
> > > > >
> > > > > > LGTM. +1 (binding)
> > > > > > -Harsha
> > > > > >
> > > > > >
> > > > > > On Wed, Sep 04, 2019 at 1:46 AM, Satish Duggana <
> > > > > satish.duggana@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > +1 (non-binding) Thanks for the nice KIP.
> > > > > > >
> > > > > > > You may want to update the KIP saying that optional tagged
> fields
> > > do
> > > > > not
> > > > > > > support complex types(or structs).
> > > > > > >
> > > > > > > On Wed, Sep 4, 2019 at 3:43 AM Jose Armando Garcia Sancio
> > > > > > > <js...@confluent.io> wrote:
> > > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > Looking forward to this improvement.
> > > > > > >
> > > > > > > On Tue, Sep 3, 2019 at 12:49 PM David Jacot <
> djacot@confluent.io>
> > > > > wrote:
> > > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > Thank for the KIP. Great addition to the Kafka protocol!
> > > > > > >
> > > > > > > Best,
> > > > > > > David
> > > > > > >
> > > > > > > Le mar. 3 sept. 2019 à 19:17, Colin McCabe <cm...@apache.org>
> a
> > > > > écrit
> > > > > > :
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'd like to start the vote for KIP-482: The Kafka Protocol
> should
> > > > > Support
> > > > > > > Optional Tagged Fields.
> > > > > > >
> > > > > > > KIP:
> > > > > > >
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > >
> KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields
> > > > > > >
> > > > > > > Discussion thread here:
> > > > > > >
> > > > > > > https://lists.apache.org/thread.html/
> > > > > > > cdc801ae886491b73ef7efecac7ef81b24382f8b6b025899ee343f7a@
> > > %3Cdev.kafka.
> > > > > > > apache.org%3E
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > --
> > > > > > > -Jose
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > David Arthur
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields
Posted by Colin McCabe <cm...@apache.org>.
Hi all,
The KIP-482 vote has passed. Thanks to everyone who voted or discussed the issue.
There were 3 non-binding +1 votes from David Jacot, Jose Armando Garcia Sancio, and Satish Duggana.
There were 3 binding +1 votes from Harsha Chintalapani, David Arthur, and Jason Gustafson.
best,
Colin
On Fri, Sep 6, 2019, at 13:36, Jason Gustafson wrote:
> +1 Thanks Colin. This is really going to help with compatibility.
>
> -Jason
>
> On Wed, Sep 4, 2019 at 1:34 PM Colin McCabe <cm...@apache.org> wrote:
>
> > On Wed, Sep 4, 2019, at 13:01, Jason Gustafson wrote:
> > > Hi Colin,
> > >
> > > Just a couple questions.
> > >
> > > 1. I think we discussed that we would do a lazy version bump of all
> > > protocols in order to get flexible version support. Can you add that to
> > the
> > > KIP?
> >
> > Good point. Added.
> >
> > > 2. The doc mentions a bump to the request and response header formats to
> > > version 1. Currently there is no formal header version. It wasn't clear
> > to
> > > me if you were suggesting that we create a header version as part of the
> > > schema or if this was just an informal way to refer to the header in
> > > "flexible version" requests. Can you clarify?
> >
> > I think we should have a formal header version. However, we can deduce
> > which header version we should use based on the apiKey and apiVersion, so
> > no changes will be needed to what is sent over the wire.
> >
> > Having a new header version will let us add new fields to request and
> > response headers. In particular, having a flexible header version will let
> > us add tagged fields, which will be useful for adding things like a message
> > traceID.
> > As another example, ThrottleTimeMs would have made more sense in the
> > response header than as an addition to every message.
> >
> > cheers,
> > Colin
> >
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Wed, Sep 4, 2019 at 8:14 AM David Arthur <mu...@gmail.com> wrote:
> > >
> > > > +1 binding.
> > > >
> > > > Thanks for the KIP, Colin!
> > > >
> > > > -David
> > > >
> > > > On Wed, Sep 4, 2019 at 5:40 AM Harsha Chintalapani <ka...@harsha.io>
> > > > wrote:
> > > >
> > > > > LGTM. +1 (binding)
> > > > > -Harsha
> > > > >
> > > > >
> > > > > On Wed, Sep 04, 2019 at 1:46 AM, Satish Duggana <
> > > > satish.duggana@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > +1 (non-binding) Thanks for the nice KIP.
> > > > > >
> > > > > > You may want to update the KIP saying that optional tagged fields
> > do
> > > > not
> > > > > > support complex types(or structs).
> > > > > >
> > > > > > On Wed, Sep 4, 2019 at 3:43 AM Jose Armando Garcia Sancio
> > > > > > <js...@confluent.io> wrote:
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Looking forward to this improvement.
> > > > > >
> > > > > > On Tue, Sep 3, 2019 at 12:49 PM David Jacot <dj...@confluent.io>
> > > > wrote:
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Thank for the KIP. Great addition to the Kafka protocol!
> > > > > >
> > > > > > Best,
> > > > > > David
> > > > > >
> > > > > > Le mar. 3 sept. 2019 à 19:17, Colin McCabe <cm...@apache.org> a
> > > > écrit
> > > > > :
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to start the vote for KIP-482: The Kafka Protocol should
> > > > Support
> > > > > > Optional Tagged Fields.
> > > > > >
> > > > > > KIP:
> > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields
> > > > > >
> > > > > > Discussion thread here:
> > > > > >
> > > > > > https://lists.apache.org/thread.html/
> > > > > > cdc801ae886491b73ef7efecac7ef81b24382f8b6b025899ee343f7a@
> > %3Cdev.kafka.
> > > > > > apache.org%3E
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > --
> > > > > > -Jose
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > David Arthur
> > > >
> > >
> >
>
Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields
Posted by Jason Gustafson <ja...@confluent.io>.
+1 Thanks Colin. This is really going to help with compatibility.
-Jason
On Wed, Sep 4, 2019 at 1:34 PM Colin McCabe <cm...@apache.org> wrote:
> On Wed, Sep 4, 2019, at 13:01, Jason Gustafson wrote:
> > Hi Colin,
> >
> > Just a couple questions.
> >
> > 1. I think we discussed that we would do a lazy version bump of all
> > protocols in order to get flexible version support. Can you add that to
> the
> > KIP?
>
> Good point. Added.
>
> > 2. The doc mentions a bump to the request and response header formats to
> > version 1. Currently there is no formal header version. It wasn't clear
> to
> > me if you were suggesting that we create a header version as part of the
> > schema or if this was just an informal way to refer to the header in
> > "flexible version" requests. Can you clarify?
>
> I think we should have a formal header version. However, we can deduce
> which header version we should use based on the apiKey and apiVersion, so
> no changes will be needed to what is sent over the wire.
>
> Having a new header version will let us add new fields to request and
> response headers. In particular, having a flexible header version will let
> us add tagged fields, which will be useful for adding things like a message
> traceID.
> As another example, ThrottleTimeMs would have made more sense in the
> response header than as an addition to every message.
>
> cheers,
> Colin
>
> >
> > Thanks,
> > Jason
> >
> > On Wed, Sep 4, 2019 at 8:14 AM David Arthur <mu...@gmail.com> wrote:
> >
> > > +1 binding.
> > >
> > > Thanks for the KIP, Colin!
> > >
> > > -David
> > >
> > > On Wed, Sep 4, 2019 at 5:40 AM Harsha Chintalapani <ka...@harsha.io>
> > > wrote:
> > >
> > > > LGTM. +1 (binding)
> > > > -Harsha
> > > >
> > > >
> > > > On Wed, Sep 04, 2019 at 1:46 AM, Satish Duggana <
> > > satish.duggana@gmail.com>
> > > > wrote:
> > > >
> > > > > +1 (non-binding) Thanks for the nice KIP.
> > > > >
> > > > > You may want to update the KIP saying that optional tagged fields
> do
> > > not
> > > > > support complex types(or structs).
> > > > >
> > > > > On Wed, Sep 4, 2019 at 3:43 AM Jose Armando Garcia Sancio
> > > > > <js...@confluent.io> wrote:
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Looking forward to this improvement.
> > > > >
> > > > > On Tue, Sep 3, 2019 at 12:49 PM David Jacot <dj...@confluent.io>
> > > wrote:
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Thank for the KIP. Great addition to the Kafka protocol!
> > > > >
> > > > > Best,
> > > > > David
> > > > >
> > > > > Le mar. 3 sept. 2019 à 19:17, Colin McCabe <cm...@apache.org> a
> > > écrit
> > > > :
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to start the vote for KIP-482: The Kafka Protocol should
> > > Support
> > > > > Optional Tagged Fields.
> > > > >
> > > > > KIP:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields
> > > > >
> > > > > Discussion thread here:
> > > > >
> > > > > https://lists.apache.org/thread.html/
> > > > > cdc801ae886491b73ef7efecac7ef81b24382f8b6b025899ee343f7a@
> %3Cdev.kafka.
> > > > > apache.org%3E
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > --
> > > > > -Jose
> > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > David Arthur
> > >
> >
>
Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields
Posted by Colin McCabe <cm...@apache.org>.
On Wed, Sep 4, 2019, at 13:01, Jason Gustafson wrote:
> Hi Colin,
>
> Just a couple questions.
>
> 1. I think we discussed that we would do a lazy version bump of all
> protocols in order to get flexible version support. Can you add that to the
> KIP?
Good point. Added.
> 2. The doc mentions a bump to the request and response header formats to
> version 1. Currently there is no formal header version. It wasn't clear to
> me if you were suggesting that we create a header version as part of the
> schema or if this was just an informal way to refer to the header in
> "flexible version" requests. Can you clarify?
I think we should have a formal header version. However, we can deduce which header version we should use based on the apiKey and apiVersion, so no changes will be needed to what is sent over the wire.
Having a new header version will let us add new fields to request and response headers. In particular, having a flexible header version will let us add tagged fields, which will be useful for adding things like a message traceID.
As another example, ThrottleTimeMs would have made more sense in the response header than as an addition to every message.
cheers,
Colin
>
> Thanks,
> Jason
>
> On Wed, Sep 4, 2019 at 8:14 AM David Arthur <mu...@gmail.com> wrote:
>
> > +1 binding.
> >
> > Thanks for the KIP, Colin!
> >
> > -David
> >
> > On Wed, Sep 4, 2019 at 5:40 AM Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > > LGTM. +1 (binding)
> > > -Harsha
> > >
> > >
> > > On Wed, Sep 04, 2019 at 1:46 AM, Satish Duggana <
> > satish.duggana@gmail.com>
> > > wrote:
> > >
> > > > +1 (non-binding) Thanks for the nice KIP.
> > > >
> > > > You may want to update the KIP saying that optional tagged fields do
> > not
> > > > support complex types(or structs).
> > > >
> > > > On Wed, Sep 4, 2019 at 3:43 AM Jose Armando Garcia Sancio
> > > > <js...@confluent.io> wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Looking forward to this improvement.
> > > >
> > > > On Tue, Sep 3, 2019 at 12:49 PM David Jacot <dj...@confluent.io>
> > wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thank for the KIP. Great addition to the Kafka protocol!
> > > >
> > > > Best,
> > > > David
> > > >
> > > > Le mar. 3 sept. 2019 à 19:17, Colin McCabe <cm...@apache.org> a
> > écrit
> > > :
> > > >
> > > > Hi all,
> > > >
> > > > I'd like to start the vote for KIP-482: The Kafka Protocol should
> > Support
> > > > Optional Tagged Fields.
> > > >
> > > > KIP:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields
> > > >
> > > > Discussion thread here:
> > > >
> > > > https://lists.apache.org/thread.html/
> > > > cdc801ae886491b73ef7efecac7ef81b24382f8b6b025899ee343f7a@%3Cdev.kafka.
> > > > apache.org%3E
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > --
> > > > -Jose
> > > >
> > > >
> > >
> >
> >
> > --
> > David Arthur
> >
>
Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields
Posted by Jason Gustafson <ja...@confluent.io>.
Hi Colin,
Just a couple questions.
1. I think we discussed that we would do a lazy version bump of all
protocols in order to get flexible version support. Can you add that to the
KIP?
2. The doc mentions a bump to the request and response header formats to
version 1. Currently there is no formal header version. It wasn't clear to
me if you were suggesting that we create a header version as part of the
schema or if this was just an informal way to refer to the header in
"flexible version" requests. Can you clarify?
Thanks,
Jason
On Wed, Sep 4, 2019 at 8:14 AM David Arthur <mu...@gmail.com> wrote:
> +1 binding.
>
> Thanks for the KIP, Colin!
>
> -David
>
> On Wed, Sep 4, 2019 at 5:40 AM Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> > LGTM. +1 (binding)
> > -Harsha
> >
> >
> > On Wed, Sep 04, 2019 at 1:46 AM, Satish Duggana <
> satish.duggana@gmail.com>
> > wrote:
> >
> > > +1 (non-binding) Thanks for the nice KIP.
> > >
> > > You may want to update the KIP saying that optional tagged fields do
> not
> > > support complex types(or structs).
> > >
> > > On Wed, Sep 4, 2019 at 3:43 AM Jose Armando Garcia Sancio
> > > <js...@confluent.io> wrote:
> > >
> > > +1 (non-binding)
> > >
> > > Looking forward to this improvement.
> > >
> > > On Tue, Sep 3, 2019 at 12:49 PM David Jacot <dj...@confluent.io>
> wrote:
> > >
> > > +1 (non-binding)
> > >
> > > Thank for the KIP. Great addition to the Kafka protocol!
> > >
> > > Best,
> > > David
> > >
> > > Le mar. 3 sept. 2019 à 19:17, Colin McCabe <cm...@apache.org> a
> écrit
> > :
> > >
> > > Hi all,
> > >
> > > I'd like to start the vote for KIP-482: The Kafka Protocol should
> Support
> > > Optional Tagged Fields.
> > >
> > > KIP:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields
> > >
> > > Discussion thread here:
> > >
> > > https://lists.apache.org/thread.html/
> > > cdc801ae886491b73ef7efecac7ef81b24382f8b6b025899ee343f7a@%3Cdev.kafka.
> > > apache.org%3E
> > >
> > > best,
> > > Colin
> > >
> > > --
> > > -Jose
> > >
> > >
> >
>
>
> --
> David Arthur
>
Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields
Posted by David Arthur <mu...@gmail.com>.
+1 binding.
Thanks for the KIP, Colin!
-David
On Wed, Sep 4, 2019 at 5:40 AM Harsha Chintalapani <ka...@harsha.io> wrote:
> LGTM. +1 (binding)
> -Harsha
>
>
> On Wed, Sep 04, 2019 at 1:46 AM, Satish Duggana <sa...@gmail.com>
> wrote:
>
> > +1 (non-binding) Thanks for the nice KIP.
> >
> > You may want to update the KIP saying that optional tagged fields do not
> > support complex types(or structs).
> >
> > On Wed, Sep 4, 2019 at 3:43 AM Jose Armando Garcia Sancio
> > <js...@confluent.io> wrote:
> >
> > +1 (non-binding)
> >
> > Looking forward to this improvement.
> >
> > On Tue, Sep 3, 2019 at 12:49 PM David Jacot <dj...@confluent.io> wrote:
> >
> > +1 (non-binding)
> >
> > Thank for the KIP. Great addition to the Kafka protocol!
> >
> > Best,
> > David
> >
> > Le mar. 3 sept. 2019 à 19:17, Colin McCabe <cm...@apache.org> a écrit
> :
> >
> > Hi all,
> >
> > I'd like to start the vote for KIP-482: The Kafka Protocol should Support
> > Optional Tagged Fields.
> >
> > KIP:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields
> >
> > Discussion thread here:
> >
> > https://lists.apache.org/thread.html/
> > cdc801ae886491b73ef7efecac7ef81b24382f8b6b025899ee343f7a@%3Cdev.kafka.
> > apache.org%3E
> >
> > best,
> > Colin
> >
> > --
> > -Jose
> >
> >
>
--
David Arthur
Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields
Posted by Harsha Chintalapani <ka...@harsha.io>.
LGTM. +1 (binding)
-Harsha
On Wed, Sep 04, 2019 at 1:46 AM, Satish Duggana <sa...@gmail.com>
wrote:
> +1 (non-binding) Thanks for the nice KIP.
>
> You may want to update the KIP saying that optional tagged fields do not
> support complex types(or structs).
>
> On Wed, Sep 4, 2019 at 3:43 AM Jose Armando Garcia Sancio
> <js...@confluent.io> wrote:
>
> +1 (non-binding)
>
> Looking forward to this improvement.
>
> On Tue, Sep 3, 2019 at 12:49 PM David Jacot <dj...@confluent.io> wrote:
>
> +1 (non-binding)
>
> Thank for the KIP. Great addition to the Kafka protocol!
>
> Best,
> David
>
> Le mar. 3 sept. 2019 à 19:17, Colin McCabe <cm...@apache.org> a écrit :
>
> Hi all,
>
> I'd like to start the vote for KIP-482: The Kafka Protocol should Support
> Optional Tagged Fields.
>
> KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/
> KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields
>
> Discussion thread here:
>
> https://lists.apache.org/thread.html/
> cdc801ae886491b73ef7efecac7ef81b24382f8b6b025899ee343f7a@%3Cdev.kafka.
> apache.org%3E
>
> best,
> Colin
>
> --
> -Jose
>
>
Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields
Posted by Satish Duggana <sa...@gmail.com>.
+1 (non-binding) Thanks for the nice KIP.
You may want to update the KIP saying that optional tagged fields do
not support complex types(or structs).
On Wed, Sep 4, 2019 at 3:43 AM Jose Armando Garcia Sancio
<js...@confluent.io> wrote:
>
> +1 (non-binding)
>
> Looking forward to this improvement.
>
> On Tue, Sep 3, 2019 at 12:49 PM David Jacot <dj...@confluent.io> wrote:
>
> > +1 (non-binding)
> >
> > Thank for the KIP. Great addition to the Kafka protocol!
> >
> > Best,
> > David
> >
> > Le mar. 3 sept. 2019 à 19:17, Colin McCabe <cm...@apache.org> a écrit :
> >
> > > Hi all,
> > >
> > > I'd like to start the vote for KIP-482: The Kafka Protocol should Support
> > > Optional Tagged Fields.
> > >
> > > KIP:
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields
> > >
> > > Discussion thread here:
> > >
> > https://lists.apache.org/thread.html/cdc801ae886491b73ef7efecac7ef81b24382f8b6b025899ee343f7a@%3Cdev.kafka.apache.org%3E
> > >
> > > best,
> > > Colin
> > >
> >
>
>
> --
> -Jose
Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields
Posted by Jose Armando Garcia Sancio <js...@confluent.io>.
+1 (non-binding)
Looking forward to this improvement.
On Tue, Sep 3, 2019 at 12:49 PM David Jacot <dj...@confluent.io> wrote:
> +1 (non-binding)
>
> Thank for the KIP. Great addition to the Kafka protocol!
>
> Best,
> David
>
> Le mar. 3 sept. 2019 à 19:17, Colin McCabe <cm...@apache.org> a écrit :
>
> > Hi all,
> >
> > I'd like to start the vote for KIP-482: The Kafka Protocol should Support
> > Optional Tagged Fields.
> >
> > KIP:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields
> >
> > Discussion thread here:
> >
> https://lists.apache.org/thread.html/cdc801ae886491b73ef7efecac7ef81b24382f8b6b025899ee343f7a@%3Cdev.kafka.apache.org%3E
> >
> > best,
> > Colin
> >
>
--
-Jose
Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields
Posted by David Jacot <dj...@confluent.io>.
+1 (non-binding)
Thank for the KIP. Great addition to the Kafka protocol!
Best,
David
Le mar. 3 sept. 2019 à 19:17, Colin McCabe <cm...@apache.org> a écrit :
> Hi all,
>
> I'd like to start the vote for KIP-482: The Kafka Protocol should Support
> Optional Tagged Fields.
>
> KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields
>
> Discussion thread here:
> https://lists.apache.org/thread.html/cdc801ae886491b73ef7efecac7ef81b24382f8b6b025899ee343f7a@%3Cdev.kafka.apache.org%3E
>
> best,
> Colin
>