You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Luke Chen <sh...@gmail.com> on 2023/05/12 07:03:40 UTC
[DISCUSS] KIP-931: Flag to ignore unused message attribute field
Hi all,
I'd like to start a discussion for the KIP-931: Flag to ignore unused
message attribute field. This KIP is to add a flag in the batch header to
indicate if messages inside the batch have attribute field or not, to
reduce the message size, thus, save network traffic and storage size (and
money, of course).
Please check the link for more detail:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field
Any feedback is welcome.
Thank you.
Luke
Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field
Posted by Divij Vaidya <di...@gmail.com>.
Hey Luke
Thank you for another great idea. Though, I agree with David's concern
here. The benefits of the space savings outweigh the cost incurred for
message conversion here. As an example, for a compressed workload, we will
have to perform decompression -> change format -> compress in new format ->
decompress to validate the batch header/data for every batch. The whole
process of message conversion is very expensive in Kafka.
Having said that, we definitely can improve the current message format.
1. About deployment timeline: May I suggest that v3 message format should
be kept as opt-in until we release Kafka v4.0 where it becomes default.
This way, the users can choose to take the incremental hit of message
conversion.
2. About changes to data layout: Since we are changing the version number,
let's bundle all pending changes together. I have another change proposed
that can be clubbed into this KIP, see below.
3. Note that protocol buffer encoded varlong can take upto 10 bytes (and 5
bytes for varInt). Mentioning it here so that it could be added as an FYI
in the doc since we are proposing moving from int64 to varLog. The worst
case of varLong uses more space. Not that it impacts the proposal, but
worth keeping an eye on.
4. Since we are changing the message version, we should revisit usage of
protobuf zigzag encoding for varInt. There are alternatives developed in
the last few years which might be more suited to our use case such as
prefixVarint. Developers of Protocol buffers recommend looking at
alternatives suited for a use case [1]. More discussions here [2].
[Additional proposal] Changing the batch format proposal
Problem: Currently, we perform validation of data schema (such as verifying
monotonically increasing offsets etc.). To do that, we have to read the
headers of each message in a batch. For a compressed batch, this means that
we have to decompress the entire batch including the payload (key/value)
even though we don't really require any information out of them.
Solution: If we change the ordering of messages in a batch to prefix with
headers of all messages first and followed by the payload of all messages,
it will lead to an optimization where we have to "partially" decompress the
batch (i.e. the prefix).
[1] https://github.com/WebAssembly/design/issues/601#issuecomment-196045891
[2] https://news.ycombinator.com/item?id=11263378
--
Divij Vaidya
On Sat, May 13, 2023 at 1:14 AM Luke Chen <sh...@gmail.com> wrote:
> Hi Kirk,
>
> Yes, the pressure in broker comes from the message format down conversion.
>
> Luke
>
>
> Kirk True <ki...@kirktrue.pro> 於 2023年5月13日 週六 上午1:30 寫道：
>
> > Hi David,
> >
> > For my own edification, when you refer to this change possibly putting
> > "more pressure on the brokers," is that from the downconversion of the
> > message format, specifically, or something else?
> >
> > Thanks,
> > Kirk
> >
> > On Fri, May 12, 2023, at 1:59 AM, Luke Chen wrote:
> > > Hi David,
> > >
> > > I know what you mean.
> > > Let's hear what others' thoughts about it. :)
> > >
> > > Luke
> > >
> > > On Fri, May 12, 2023 at 4:53 PM David Jacot
> <djacot@confluent.io.invalid
> > >
> > > wrote:
> > >
> > > > Thanks, Luke.
> > > >
> > > > > But if the producers and consumers all existed in the same
> > organization,
> > > > which means upgrading producers/consumers for the org's cost saving,
> > should
> > > > be a reasonable motivation.
> > > >
> > > > Yeah, that works in this case. However, Kafka is often used as a
> > service
> > > > (on premise or in cloud) nowadays and in this case the
> > producers/consumers
> > > > versions are completely out of control thus my concern.
> > > >
> > > > BR,
> > > > David
> > > >
> > > > On Fri, May 12, 2023 at 10:47 AM Luke Chen <sh...@gmail.com>
> wrote:
> > > >
> > > > > Hi David,
> > > > >
> > > > > Yes, you're right. I've bumped the version of record batch, and
> > describe
> > > > > the down-conversion will happen like what we do for message format
> > v1 now
> > > > > when old consumers consuming records.
> > > > >
> > > > > > Overall, I wonder if the bandwidth saving is worth this change
> > given
> > > > that
> > > > > it will put more pressure on the brokers.
> > > > > Actually, I'm not 100% sure. So I'd also like to hear what the
> > community
> > > > > thought about it.
> > > > > But if the producers and consumers all existed in the same
> > organization,
> > > > > which means upgrading producers/consumers for the org's cost
> saving,
> > > > should
> > > > > be a reasonable motivation.
> > > > >
> > > > > Thanks.
> > > > > Luke
> > > > >
> > > > >
> > > > > On Fri, May 12, 2023 at 3:43 PM David Jacot
> > <djacot@confluent.io.invalid
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Luke,
> > > > > >
> > > > > > Thanks for the KIP.
> > > > > >
> > > > > > What do we do in the case where a batch is written with
> > > > > > `ignoreMessageAttributes` set to 1, which means that messages
> won't
> > > > have
> > > > > > the `attributes`, and is consumed by a consumer which does not
> > > > understand
> > > > > > this new format? I suppose that we would need to introduce a new
> > > > version
> > > > > > for the message format (v3) and that we will have to downconvert
> > > > records
> > > > > > from the new format version to v2 in this case. This is not clear
> > in
> > > > the
> > > > > > KIP. Could you elaborate a bit more on this? Overall, I wonder if
> > the
> > > > > > bandwidth saving is worth this change given that it will put more
> > > > > pressure
> > > > > > on the brokers.
> > > > > >
> > > > > > Best,
> > > > > > David
> > > > > >
> > > > > > On Fri, May 12, 2023 at 9:04 AM Luke Chen <sh...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'd like to start a discussion for the KIP-931: Flag to ignore
> > unused
> > > > > > > message attribute field. This KIP is to add a flag in the batch
> > > > header
> > > > > to
> > > > > > > indicate if messages inside the batch have attribute field or
> > not, to
> > > > > > > reduce the message size, thus, save network traffic and storage
> > size
> > > > > (and
> > > > > > > money, of course).
> > > > > > >
> > > > > > > Please check the link for more detail:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field
> > > > > > >
> > > > > > > Any feedback is welcome.
> > > > > > >
> > > > > > > Thank you.
> > > > > > > Luke
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field
Posted by Luke Chen <sh...@gmail.com>.
Hi Kirk,
Yes, the pressure in broker comes from the message format down conversion.
Luke
Kirk True <ki...@kirktrue.pro> 於 2023年5月13日 週六 上午1:30 寫道：
> Hi David,
>
> For my own edification, when you refer to this change possibly putting
> "more pressure on the brokers," is that from the downconversion of the
> message format, specifically, or something else?
>
> Thanks,
> Kirk
>
> On Fri, May 12, 2023, at 1:59 AM, Luke Chen wrote:
> > Hi David,
> >
> > I know what you mean.
> > Let's hear what others' thoughts about it. :)
> >
> > Luke
> >
> > On Fri, May 12, 2023 at 4:53 PM David Jacot <djacot@confluent.io.invalid
> >
> > wrote:
> >
> > > Thanks, Luke.
> > >
> > > > But if the producers and consumers all existed in the same
> organization,
> > > which means upgrading producers/consumers for the org's cost saving,
> should
> > > be a reasonable motivation.
> > >
> > > Yeah, that works in this case. However, Kafka is often used as a
> service
> > > (on premise or in cloud) nowadays and in this case the
> producers/consumers
> > > versions are completely out of control thus my concern.
> > >
> > > BR,
> > > David
> > >
> > > On Fri, May 12, 2023 at 10:47 AM Luke Chen <sh...@gmail.com> wrote:
> > >
> > > > Hi David,
> > > >
> > > > Yes, you're right. I've bumped the version of record batch, and
> describe
> > > > the down-conversion will happen like what we do for message format
> v1 now
> > > > when old consumers consuming records.
> > > >
> > > > > Overall, I wonder if the bandwidth saving is worth this change
> given
> > > that
> > > > it will put more pressure on the brokers.
> > > > Actually, I'm not 100% sure. So I'd also like to hear what the
> community
> > > > thought about it.
> > > > But if the producers and consumers all existed in the same
> organization,
> > > > which means upgrading producers/consumers for the org's cost saving,
> > > should
> > > > be a reasonable motivation.
> > > >
> > > > Thanks.
> > > > Luke
> > > >
> > > >
> > > > On Fri, May 12, 2023 at 3:43 PM David Jacot
> <djacot@confluent.io.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Hi Luke,
> > > > >
> > > > > Thanks for the KIP.
> > > > >
> > > > > What do we do in the case where a batch is written with
> > > > > `ignoreMessageAttributes` set to 1, which means that messages won't
> > > have
> > > > > the `attributes`, and is consumed by a consumer which does not
> > > understand
> > > > > this new format? I suppose that we would need to introduce a new
> > > version
> > > > > for the message format (v3) and that we will have to downconvert
> > > records
> > > > > from the new format version to v2 in this case. This is not clear
> in
> > > the
> > > > > KIP. Could you elaborate a bit more on this? Overall, I wonder if
> the
> > > > > bandwidth saving is worth this change given that it will put more
> > > > pressure
> > > > > on the brokers.
> > > > >
> > > > > Best,
> > > > > David
> > > > >
> > > > > On Fri, May 12, 2023 at 9:04 AM Luke Chen <sh...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to start a discussion for the KIP-931: Flag to ignore
> unused
> > > > > > message attribute field. This KIP is to add a flag in the batch
> > > header
> > > > to
> > > > > > indicate if messages inside the batch have attribute field or
> not, to
> > > > > > reduce the message size, thus, save network traffic and storage
> size
> > > > (and
> > > > > > money, of course).
> > > > > >
> > > > > > Please check the link for more detail:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field
> > > > > >
> > > > > > Any feedback is welcome.
> > > > > >
> > > > > > Thank you.
> > > > > > Luke
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field
Posted by Kirk True <ki...@kirktrue.pro>.
Hi David,
For my own edification, when you refer to this change possibly putting "more pressure on the brokers," is that from the downconversion of the message format, specifically, or something else?
Thanks,
Kirk
On Fri, May 12, 2023, at 1:59 AM, Luke Chen wrote:
> Hi David,
>
> I know what you mean.
> Let's hear what others' thoughts about it. :)
>
> Luke
>
> On Fri, May 12, 2023 at 4:53 PM David Jacot <dj...@confluent.io.invalid>
> wrote:
>
> > Thanks, Luke.
> >
> > > But if the producers and consumers all existed in the same organization,
> > which means upgrading producers/consumers for the org's cost saving, should
> > be a reasonable motivation.
> >
> > Yeah, that works in this case. However, Kafka is often used as a service
> > (on premise or in cloud) nowadays and in this case the producers/consumers
> > versions are completely out of control thus my concern.
> >
> > BR,
> > David
> >
> > On Fri, May 12, 2023 at 10:47 AM Luke Chen <sh...@gmail.com> wrote:
> >
> > > Hi David,
> > >
> > > Yes, you're right. I've bumped the version of record batch, and describe
> > > the down-conversion will happen like what we do for message format v1 now
> > > when old consumers consuming records.
> > >
> > > > Overall, I wonder if the bandwidth saving is worth this change given
> > that
> > > it will put more pressure on the brokers.
> > > Actually, I'm not 100% sure. So I'd also like to hear what the community
> > > thought about it.
> > > But if the producers and consumers all existed in the same organization,
> > > which means upgrading producers/consumers for the org's cost saving,
> > should
> > > be a reasonable motivation.
> > >
> > > Thanks.
> > > Luke
> > >
> > >
> > > On Fri, May 12, 2023 at 3:43 PM David Jacot <djacot@confluent.io.invalid
> > >
> > > wrote:
> > >
> > > > Hi Luke,
> > > >
> > > > Thanks for the KIP.
> > > >
> > > > What do we do in the case where a batch is written with
> > > > `ignoreMessageAttributes` set to 1, which means that messages won't
> > have
> > > > the `attributes`, and is consumed by a consumer which does not
> > understand
> > > > this new format? I suppose that we would need to introduce a new
> > version
> > > > for the message format (v3) and that we will have to downconvert
> > records
> > > > from the new format version to v2 in this case. This is not clear in
> > the
> > > > KIP. Could you elaborate a bit more on this? Overall, I wonder if the
> > > > bandwidth saving is worth this change given that it will put more
> > > pressure
> > > > on the brokers.
> > > >
> > > > Best,
> > > > David
> > > >
> > > > On Fri, May 12, 2023 at 9:04 AM Luke Chen <sh...@gmail.com> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to start a discussion for the KIP-931: Flag to ignore unused
> > > > > message attribute field. This KIP is to add a flag in the batch
> > header
> > > to
> > > > > indicate if messages inside the batch have attribute field or not, to
> > > > > reduce the message size, thus, save network traffic and storage size
> > > (and
> > > > > money, of course).
> > > > >
> > > > > Please check the link for more detail:
> > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field
> > > > >
> > > > > Any feedback is welcome.
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field
Posted by Luke Chen <sh...@gmail.com>.
Hi David,
I know what you mean.
Let's hear what others' thoughts about it. :)
Luke
On Fri, May 12, 2023 at 4:53 PM David Jacot <dj...@confluent.io.invalid>
wrote:
> Thanks, Luke.
>
> > But if the producers and consumers all existed in the same organization,
> which means upgrading producers/consumers for the org's cost saving, should
> be a reasonable motivation.
>
> Yeah, that works in this case. However, Kafka is often used as a service
> (on premise or in cloud) nowadays and in this case the producers/consumers
> versions are completely out of control thus my concern.
>
> BR,
> David
>
> On Fri, May 12, 2023 at 10:47 AM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi David,
> >
> > Yes, you're right. I've bumped the version of record batch, and describe
> > the down-conversion will happen like what we do for message format v1 now
> > when old consumers consuming records.
> >
> > > Overall, I wonder if the bandwidth saving is worth this change given
> that
> > it will put more pressure on the brokers.
> > Actually, I'm not 100% sure. So I'd also like to hear what the community
> > thought about it.
> > But if the producers and consumers all existed in the same organization,
> > which means upgrading producers/consumers for the org's cost saving,
> should
> > be a reasonable motivation.
> >
> > Thanks.
> > Luke
> >
> >
> > On Fri, May 12, 2023 at 3:43 PM David Jacot <djacot@confluent.io.invalid
> >
> > wrote:
> >
> > > Hi Luke,
> > >
> > > Thanks for the KIP.
> > >
> > > What do we do in the case where a batch is written with
> > > `ignoreMessageAttributes` set to 1, which means that messages won't
> have
> > > the `attributes`, and is consumed by a consumer which does not
> understand
> > > this new format? I suppose that we would need to introduce a new
> version
> > > for the message format (v3) and that we will have to downconvert
> records
> > > from the new format version to v2 in this case. This is not clear in
> the
> > > KIP. Could you elaborate a bit more on this? Overall, I wonder if the
> > > bandwidth saving is worth this change given that it will put more
> > pressure
> > > on the brokers.
> > >
> > > Best,
> > > David
> > >
> > > On Fri, May 12, 2023 at 9:04 AM Luke Chen <sh...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to start a discussion for the KIP-931: Flag to ignore unused
> > > > message attribute field. This KIP is to add a flag in the batch
> header
> > to
> > > > indicate if messages inside the batch have attribute field or not, to
> > > > reduce the message size, thus, save network traffic and storage size
> > (and
> > > > money, of course).
> > > >
> > > > Please check the link for more detail:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field
> > > >
> > > > Any feedback is welcome.
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field
Posted by David Jacot <dj...@confluent.io.INVALID>.
Thanks, Luke.
> But if the producers and consumers all existed in the same organization,
which means upgrading producers/consumers for the org's cost saving, should
be a reasonable motivation.
Yeah, that works in this case. However, Kafka is often used as a service
(on premise or in cloud) nowadays and in this case the producers/consumers
versions are completely out of control thus my concern.
BR,
David
On Fri, May 12, 2023 at 10:47 AM Luke Chen <sh...@gmail.com> wrote:
> Hi David,
>
> Yes, you're right. I've bumped the version of record batch, and describe
> the down-conversion will happen like what we do for message format v1 now
> when old consumers consuming records.
>
> > Overall, I wonder if the bandwidth saving is worth this change given that
> it will put more pressure on the brokers.
> Actually, I'm not 100% sure. So I'd also like to hear what the community
> thought about it.
> But if the producers and consumers all existed in the same organization,
> which means upgrading producers/consumers for the org's cost saving, should
> be a reasonable motivation.
>
> Thanks.
> Luke
>
>
> On Fri, May 12, 2023 at 3:43 PM David Jacot <dj...@confluent.io.invalid>
> wrote:
>
> > Hi Luke,
> >
> > Thanks for the KIP.
> >
> > What do we do in the case where a batch is written with
> > `ignoreMessageAttributes` set to 1, which means that messages won't have
> > the `attributes`, and is consumed by a consumer which does not understand
> > this new format? I suppose that we would need to introduce a new version
> > for the message format (v3) and that we will have to downconvert records
> > from the new format version to v2 in this case. This is not clear in the
> > KIP. Could you elaborate a bit more on this? Overall, I wonder if the
> > bandwidth saving is worth this change given that it will put more
> pressure
> > on the brokers.
> >
> > Best,
> > David
> >
> > On Fri, May 12, 2023 at 9:04 AM Luke Chen <sh...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I'd like to start a discussion for the KIP-931: Flag to ignore unused
> > > message attribute field. This KIP is to add a flag in the batch header
> to
> > > indicate if messages inside the batch have attribute field or not, to
> > > reduce the message size, thus, save network traffic and storage size
> (and
> > > money, of course).
> > >
> > > Please check the link for more detail:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field
> > >
> > > Any feedback is welcome.
> > >
> > > Thank you.
> > > Luke
> > >
> >
>
Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field
Posted by Luke Chen <sh...@gmail.com>.
Hi David,
Yes, you're right. I've bumped the version of record batch, and describe
the down-conversion will happen like what we do for message format v1 now
when old consumers consuming records.
> Overall, I wonder if the bandwidth saving is worth this change given that
it will put more pressure on the brokers.
Actually, I'm not 100% sure. So I'd also like to hear what the community
thought about it.
But if the producers and consumers all existed in the same organization,
which means upgrading producers/consumers for the org's cost saving, should
be a reasonable motivation.
Thanks.
Luke
On Fri, May 12, 2023 at 3:43 PM David Jacot <dj...@confluent.io.invalid>
wrote:
> Hi Luke,
>
> Thanks for the KIP.
>
> What do we do in the case where a batch is written with
> `ignoreMessageAttributes` set to 1, which means that messages won't have
> the `attributes`, and is consumed by a consumer which does not understand
> this new format? I suppose that we would need to introduce a new version
> for the message format (v3) and that we will have to downconvert records
> from the new format version to v2 in this case. This is not clear in the
> KIP. Could you elaborate a bit more on this? Overall, I wonder if the
> bandwidth saving is worth this change given that it will put more pressure
> on the brokers.
>
> Best,
> David
>
> On Fri, May 12, 2023 at 9:04 AM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi all,
> >
> > I'd like to start a discussion for the KIP-931: Flag to ignore unused
> > message attribute field. This KIP is to add a flag in the batch header to
> > indicate if messages inside the batch have attribute field or not, to
> > reduce the message size, thus, save network traffic and storage size (and
> > money, of course).
> >
> > Please check the link for more detail:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field
> >
> > Any feedback is welcome.
> >
> > Thank you.
> > Luke
> >
>
Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field
Posted by David Jacot <dj...@confluent.io.INVALID>.
Hi Luke,
Thanks for the KIP.
What do we do in the case where a batch is written with
`ignoreMessageAttributes` set to 1, which means that messages won't have
the `attributes`, and is consumed by a consumer which does not understand
this new format? I suppose that we would need to introduce a new version
for the message format (v3) and that we will have to downconvert records
from the new format version to v2 in this case. This is not clear in the
KIP. Could you elaborate a bit more on this? Overall, I wonder if the
bandwidth saving is worth this change given that it will put more pressure
on the brokers.
Best,
David
On Fri, May 12, 2023 at 9:04 AM Luke Chen <sh...@gmail.com> wrote:
> Hi all,
>
> I'd like to start a discussion for the KIP-931: Flag to ignore unused
> message attribute field. This KIP is to add a flag in the batch header to
> indicate if messages inside the batch have attribute field or not, to
> reduce the message size, thus, save network traffic and storage size (and
> money, of course).
>
> Please check the link for more detail:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field
>
> Any feedback is welcome.
>
> Thank you.
> Luke
>
Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field
Posted by Luke Chen <sh...@gmail.com>.
Good idea!
I've created a wiki for the ideas for message format v.3, and added the
link in this KIP.
https://cwiki.apache.org/confluence/display/KAFKA/ideas+for+kafka+message+format+v.3
Thanks.
Luke
On Tue, May 16, 2023 at 4:30 PM Ismael Juma <is...@juma.me.uk> wrote:
> Perhaps we can start a wiki page where we collect these ideas as a
> precursor to a KIP for record format v3?
>
> Ismael
>
> On Mon, May 15, 2023, 8:19 PM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi Divij and Ismael,
> >
> > Thanks for your great comments.
> > Yes, I know record format changes are _extremely expensive_ for the
> > ecosystem.
> > But on the other hand, it's not clear "what kind of change" is worth
> > changing it.
> > That's why I posted the KIP for discussion.
> >
> > It looks like the benefit of this KIP is still not strong enough, and we
> > have more further changes planned for message format v3.
> > I'll move this KIP into "discarded" state and add some reasons there.
> > Please remember to take this KIP (and Divij's proposal) into
> consideration
> > when we plan to propose a new message format.
> >
> > Thank you.
> > Luke
> >
> >
> > On Mon, May 15, 2023 at 10:55 PM Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Luke,
> > >
> > > Thanks for the KIP. A few things:
> > >
> > > 1. Record format changes are _extremely expensive_ for the ecosystem,
> so
> > we
> > > need to have very strong motivation for them. There is a reason why we
> > have
> > > had so few of them and the last one was in 0.11.
> > > 2. It was a conscious decision to make the record header fixed size -
> it
> > > would be a lot more complicated to set some of the fields after writing
> > the
> > > actual records otherwise. If we want the record header to be variable
> > size,
> > > then we would probably want to move some fields to a "trailer".
> > > 3. v3 of the record format should make it cheaper to make changes in
> the
> > > future (perhaps it could support tagged fields or similar)
> > > 4. We'd want to fix other known issues at the same time (eg log append
> > time
> > > should always be available, there may be others)
> > > 5. We should consider whether we would want to introduce a user header
> > that
> > > is at the batch level vs record level for efficiency reasons
> > >
> > > Ismael
> > >
> > > On Fri, May 12, 2023 at 12:04 AM Luke Chen <sh...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to start a discussion for the KIP-931: Flag to ignore unused
> > > > message attribute field. This KIP is to add a flag in the batch
> header
> > to
> > > > indicate if messages inside the batch have attribute field or not, to
> > > > reduce the message size, thus, save network traffic and storage size
> > (and
> > > > money, of course).
> > > >
> > > > Please check the link for more detail:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field
> > > >
> > > > Any feedback is welcome.
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field
Posted by Ismael Juma <is...@juma.me.uk>.
Perhaps we can start a wiki page where we collect these ideas as a
precursor to a KIP for record format v3?
Ismael
On Mon, May 15, 2023, 8:19 PM Luke Chen <sh...@gmail.com> wrote:
> Hi Divij and Ismael,
>
> Thanks for your great comments.
> Yes, I know record format changes are _extremely expensive_ for the
> ecosystem.
> But on the other hand, it's not clear "what kind of change" is worth
> changing it.
> That's why I posted the KIP for discussion.
>
> It looks like the benefit of this KIP is still not strong enough, and we
> have more further changes planned for message format v3.
> I'll move this KIP into "discarded" state and add some reasons there.
> Please remember to take this KIP (and Divij's proposal) into consideration
> when we plan to propose a new message format.
>
> Thank you.
> Luke
>
>
> On Mon, May 15, 2023 at 10:55 PM Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Luke,
> >
> > Thanks for the KIP. A few things:
> >
> > 1. Record format changes are _extremely expensive_ for the ecosystem, so
> we
> > need to have very strong motivation for them. There is a reason why we
> have
> > had so few of them and the last one was in 0.11.
> > 2. It was a conscious decision to make the record header fixed size - it
> > would be a lot more complicated to set some of the fields after writing
> the
> > actual records otherwise. If we want the record header to be variable
> size,
> > then we would probably want to move some fields to a "trailer".
> > 3. v3 of the record format should make it cheaper to make changes in the
> > future (perhaps it could support tagged fields or similar)
> > 4. We'd want to fix other known issues at the same time (eg log append
> time
> > should always be available, there may be others)
> > 5. We should consider whether we would want to introduce a user header
> that
> > is at the batch level vs record level for efficiency reasons
> >
> > Ismael
> >
> > On Fri, May 12, 2023 at 12:04 AM Luke Chen <sh...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I'd like to start a discussion for the KIP-931: Flag to ignore unused
> > > message attribute field. This KIP is to add a flag in the batch header
> to
> > > indicate if messages inside the batch have attribute field or not, to
> > > reduce the message size, thus, save network traffic and storage size
> (and
> > > money, of course).
> > >
> > > Please check the link for more detail:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field
> > >
> > > Any feedback is welcome.
> > >
> > > Thank you.
> > > Luke
> > >
> >
>
Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field
Posted by Luke Chen <sh...@gmail.com>.
Hi Divij and Ismael,
Thanks for your great comments.
Yes, I know record format changes are _extremely expensive_ for the
ecosystem.
But on the other hand, it's not clear "what kind of change" is worth
changing it.
That's why I posted the KIP for discussion.
It looks like the benefit of this KIP is still not strong enough, and we
have more further changes planned for message format v3.
I'll move this KIP into "discarded" state and add some reasons there.
Please remember to take this KIP (and Divij's proposal) into consideration
when we plan to propose a new message format.
Thank you.
Luke
On Mon, May 15, 2023 at 10:55 PM Ismael Juma <is...@juma.me.uk> wrote:
> Hi Luke,
>
> Thanks for the KIP. A few things:
>
> 1. Record format changes are _extremely expensive_ for the ecosystem, so we
> need to have very strong motivation for them. There is a reason why we have
> had so few of them and the last one was in 0.11.
> 2. It was a conscious decision to make the record header fixed size - it
> would be a lot more complicated to set some of the fields after writing the
> actual records otherwise. If we want the record header to be variable size,
> then we would probably want to move some fields to a "trailer".
> 3. v3 of the record format should make it cheaper to make changes in the
> future (perhaps it could support tagged fields or similar)
> 4. We'd want to fix other known issues at the same time (eg log append time
> should always be available, there may be others)
> 5. We should consider whether we would want to introduce a user header that
> is at the batch level vs record level for efficiency reasons
>
> Ismael
>
> On Fri, May 12, 2023 at 12:04 AM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi all,
> >
> > I'd like to start a discussion for the KIP-931: Flag to ignore unused
> > message attribute field. This KIP is to add a flag in the batch header to
> > indicate if messages inside the batch have attribute field or not, to
> > reduce the message size, thus, save network traffic and storage size (and
> > money, of course).
> >
> > Please check the link for more detail:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field
> >
> > Any feedback is welcome.
> >
> > Thank you.
> > Luke
> >
>
Re: [DISCUSS] KIP-931: Flag to ignore unused message attribute field
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Luke,
Thanks for the KIP. A few things:
1. Record format changes are _extremely expensive_ for the ecosystem, so we
need to have very strong motivation for them. There is a reason why we have
had so few of them and the last one was in 0.11.
2. It was a conscious decision to make the record header fixed size - it
would be a lot more complicated to set some of the fields after writing the
actual records otherwise. If we want the record header to be variable size,
then we would probably want to move some fields to a "trailer".
3. v3 of the record format should make it cheaper to make changes in the
future (perhaps it could support tagged fields or similar)
4. We'd want to fix other known issues at the same time (eg log append time
should always be available, there may be others)
5. We should consider whether we would want to introduce a user header that
is at the batch level vs record level for efficiency reasons
Ismael
On Fri, May 12, 2023 at 12:04 AM Luke Chen <sh...@gmail.com> wrote:
> Hi all,
>
> I'd like to start a discussion for the KIP-931: Flag to ignore unused
> message attribute field. This KIP is to add a flag in the batch header to
> indicate if messages inside the batch have attribute field or not, to
> reduce the message size, thus, save network traffic and storage size (and
> money, of course).
>
> Please check the link for more detail:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field
>
> Any feedback is welcome.
>
> Thank you.
> Luke
>