You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Tom Bentley <tb...@redhat.com> on 2020/07/01 16:21:39 UTC
[DISCUSS] KIP-363
Hi all,
Following a suggestion from Colin in the KIP-625 discussion thread, I'd
like to start discussion on a much smaller KIP which proposes to make error
codes and messages tagged fields in all RPCs.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-363%3A+Make+RPC+error+codes+and+messages+tagged+fields
I'd be grateful for any feedback you may have.
Kind regards,
Tom
Re: [DISCUSS] KIP-363
Posted by Tom Bentley <tb...@redhat.com>.
Hi,
I'd like to pick up this conversation again, now that all the RPCs are
using the generated classes.
Does anyone have any more questions or comments about this?
Cheers,
Tom
On Mon, Jul 13, 2020 at 2:48 PM Tom Bentley <tb...@redhat.com> wrote:
> Hi Magnus,
>
> Sorry for the late reply. In the specific case you mentioned of an
> OffsetCommitResponse with 100 partitions across 10 topics we end up with
> -200bytes (no error) and +200bytes (any error) in absolute terms. Precise
> numbers depend on topic names too, of course but I did a little test using
> a hypothetical version 9 response and got these numbers:
>
> Version=8 no error size=806
> Version=9 no error size=606
> Version=8 any error size=806
> Version=9 any error size=1006
>
> So about 25% less in the no-error case and 25% more in the error case,
> compared with version 8.
>
> Kind regards,
>
> Tom
>
>
>
>
>
> On Wed, Jul 8, 2020 at 8:47 PM Magnus Edenhill <ma...@edenhill.se> wrote:
>
>> Hi Tom,
>>
>> I think it would be useful with some real world (or made up!) numbers on
>> how much relative/% space is saved for
>> the most error-dense protocol requests.
>> E.g., an OffsetCommitResponse with 10 topics and 100 failing partitions
>> would reduce the overall size by % bytes.
>>
>> Thanks,
>> Magnus
>>
>>
>> Den tis 7 juli 2020 kl 17:01 skrev Colin McCabe <cm...@apache.org>:
>>
>> > Hi Tom,
>> >
>> > Thanks for this. I think the tough part is probably the few messages
>> that
>> > are still using manual serialization, which can't be easily converted to
>> > using this. So we will probably have to upgrade them to using automatic
>> > generation, or accept a little inconsistency for a while until they are
>> > upgraded.
>> >
>> > best,
>> > Colin
>> >
>> >
>> > On Wed, Jul 1, 2020, at 09:21, Tom Bentley wrote:
>> > > Hi all,
>> > >
>> > > Following a suggestion from Colin in the KIP-625 discussion thread,
>> I'd
>> > > like to start discussion on a much smaller KIP which proposes to make
>> > error
>> > > codes and messages tagged fields in all RPCs.
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-363%3A+Make+RPC+error+codes+and+messages+tagged+fields
>> > >
>> > > I'd be grateful for any feedback you may have.
>> > >
>> > > Kind regards,
>> > >
>> > > Tom
>> > >
>> >
>>
>
Re: [DISCUSS] KIP-363
Posted by Tom Bentley <tb...@redhat.com>.
Hi Magnus,
Sorry for the late reply. In the specific case you mentioned of an
OffsetCommitResponse with 100 partitions across 10 topics we end up with
-200bytes (no error) and +200bytes (any error) in absolute terms. Precise
numbers depend on topic names too, of course but I did a little test using
a hypothetical version 9 response and got these numbers:
Version=8 no error size=806
Version=9 no error size=606
Version=8 any error size=806
Version=9 any error size=1006
So about 25% less in the no-error case and 25% more in the error case,
compared with version 8.
Kind regards,
Tom
On Wed, Jul 8, 2020 at 8:47 PM Magnus Edenhill <ma...@edenhill.se> wrote:
> Hi Tom,
>
> I think it would be useful with some real world (or made up!) numbers on
> how much relative/% space is saved for
> the most error-dense protocol requests.
> E.g., an OffsetCommitResponse with 10 topics and 100 failing partitions
> would reduce the overall size by % bytes.
>
> Thanks,
> Magnus
>
>
> Den tis 7 juli 2020 kl 17:01 skrev Colin McCabe <cm...@apache.org>:
>
> > Hi Tom,
> >
> > Thanks for this. I think the tough part is probably the few messages
> that
> > are still using manual serialization, which can't be easily converted to
> > using this. So we will probably have to upgrade them to using automatic
> > generation, or accept a little inconsistency for a while until they are
> > upgraded.
> >
> > best,
> > Colin
> >
> >
> > On Wed, Jul 1, 2020, at 09:21, Tom Bentley wrote:
> > > Hi all,
> > >
> > > Following a suggestion from Colin in the KIP-625 discussion thread, I'd
> > > like to start discussion on a much smaller KIP which proposes to make
> > error
> > > codes and messages tagged fields in all RPCs.
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-363%3A+Make+RPC+error+codes+and+messages+tagged+fields
> > >
> > > I'd be grateful for any feedback you may have.
> > >
> > > Kind regards,
> > >
> > > Tom
> > >
> >
>
Re: [DISCUSS] KIP-363
Posted by Magnus Edenhill <ma...@edenhill.se>.
Hi Tom,
I think it would be useful with some real world (or made up!) numbers on
how much relative/% space is saved for
the most error-dense protocol requests.
E.g., an OffsetCommitResponse with 10 topics and 100 failing partitions
would reduce the overall size by % bytes.
Thanks,
Magnus
Den tis 7 juli 2020 kl 17:01 skrev Colin McCabe <cm...@apache.org>:
> Hi Tom,
>
> Thanks for this. I think the tough part is probably the few messages that
> are still using manual serialization, which can't be easily converted to
> using this. So we will probably have to upgrade them to using automatic
> generation, or accept a little inconsistency for a while until they are
> upgraded.
>
> best,
> Colin
>
>
> On Wed, Jul 1, 2020, at 09:21, Tom Bentley wrote:
> > Hi all,
> >
> > Following a suggestion from Colin in the KIP-625 discussion thread, I'd
> > like to start discussion on a much smaller KIP which proposes to make
> error
> > codes and messages tagged fields in all RPCs.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-363%3A+Make+RPC+error+codes+and+messages+tagged+fields
> >
> > I'd be grateful for any feedback you may have.
> >
> > Kind regards,
> >
> > Tom
> >
>
Re: [DISCUSS] KIP-363
Posted by Colin McCabe <cm...@apache.org>.
Hi Tom,
Thanks for this. I think the tough part is probably the few messages that are still using manual serialization, which can't be easily converted to using this. So we will probably have to upgrade them to using automatic generation, or accept a little inconsistency for a while until they are upgraded.
best,
Colin
On Wed, Jul 1, 2020, at 09:21, Tom Bentley wrote:
> Hi all,
>
> Following a suggestion from Colin in the KIP-625 discussion thread, I'd
> like to start discussion on a much smaller KIP which proposes to make error
> codes and messages tagged fields in all RPCs.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-363%3A+Make+RPC+error+codes+and+messages+tagged+fields
>
> I'd be grateful for any feedback you may have.
>
> Kind regards,
>
> Tom
>