You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Theo Hultberg <th...@iconara.net> on 2015/02/28 20:35:25 UTC
Problems decoding fetch responses, getting messages that are bigger
than their message sets
Hi,
Mostly out of curiosity I'm writing a Kafka client, and I'm getting stuck
at decoding fetch responses. Most of the time everything goes fine, but
quite often I get frames back that seem to be wrong. I'm sure I've
misunderstood something about the spec, so maybe someone can get me on the
right track.
I get frames where the MessageSetSize is 2048, but it contains messages
whose MessageSize is larger than this. The MessageSetSize is always 2048,
which makes me suspicious.
I've tried feeding these message sets to the Java client (into
MemoryRecords.readableRecords), but when I iterate over it, it is empty.
Further inspecting what it does with the bytes I see that it gets an
EOFException and stops iterating, and I don't really understand what's
going on.
I'm doing these fetch requests against a cluster running 0.8.2.1.
yours,
Theo
Re: Problems decoding fetch responses, getting messages that are
bigger than their message sets
Posted by Theo Hultberg <th...@iconara.net>.
Thank you for confirming my suspicion.
T#
On Sun, Mar 1, 2015 at 9:48 AM, Magnus Edenhill <ma...@edenhill.se> wrote:
> Hi Theo,
>
> seems like you found the answer yourself, the server may return partial
> messages.
> While parsing a MessageSet you simply ignore any message whose length >
> remainingBufferLength.
>
> Regards,
> Magnus
>
>
> 2015-03-01 8:55 GMT+01:00 Theo Hultberg <th...@iconara.net>:
>
> > That the message set size was always the same when this happened made me
> > start looking for that number, and it turned out that was what I had set
> as
> > the MaxBytes field of the requests. This make me think that what happens
> is
> > that the next message to fetch is larger than this, but it's sent anyway.
> > Is this correct? Is this how it is supposed to work? I'm looking through
> > other clients to see how they deal with this, and I think I might have
> > found the same thing in the Go client (
> > https://github.com/Shopify/sarama/blob/master/message_set.go#L72-L76),
> and
> > the corresponding comment in the protocol guide. Is what I have found the
> > thing described as "As an optimization the server is allowed to return a
> > partial message at the end of the message set. Clients should handle this
> > case."?
> >
> > yours,
> > Theo
> >
> > On Sat, Feb 28, 2015 at 10:19 PM, Theo Hultberg <th...@iconara.net>
> wrote:
> >
> > > Here's an example of a frame that looks malformed, I've split the bytes
> > > apart and annotated the pieces with the names from the Kafka protocol
> > guide
> > > (
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchResponse
> > > ).
> > >
> > > Notice that the MessageSetSize is 0x00000800 (2048), but the
> MessageSize
> > > is 0x000010B3 (4275). The size prefix of the Value field agrees with
> > > MessageSize, but the size of the frame agrees with MessageSetSize,
> > > because there are only 2022 bytes left in the frame. The bytes after
> the
> > > crc field to the end of the frame don't checksum to the value of the
> crc
> > > field either.
> > >
> > > (topics) 00 00 00 01
> > > TopicName 00 1F (+ 31 character topic name)
> > > (partitions) 00 00 00 01
> > > Partition 00 00 00 02
> > > ErrorCode 00 00
> > > HighwaterMarkOffset 00 00 00 00 00 3F 5B 17
> > > MessageSetSize 00 00 08 00
> > > Offset 00 00 00 00 00 3F 5B 16
> > > MessageSize 00 00 10 B3
> > > Crc B0 13 F4 E3
> > > MagicByte 00
> > > Attributes 02
> > > Key FF FF FF FF
> > > Value 00 00 10 A5 (+ 2022 bytes left in frame)
> > >
> > > What's going on here, what have I misunderstood?
> > >
> > > yours,
> > > Theo
> > >
> > > On Sat, Feb 28, 2015 at 8:35 PM, Theo Hultberg <th...@iconara.net>
> wrote:
> > >
> > >> Hi,
> > >>
> > >> Mostly out of curiosity I'm writing a Kafka client, and I'm getting
> > stuck
> > >> at decoding fetch responses. Most of the time everything goes fine,
> but
> > >> quite often I get frames back that seem to be wrong. I'm sure I've
> > >> misunderstood something about the spec, so maybe someone can get me on
> > the
> > >> right track.
> > >>
> > >> I get frames where the MessageSetSize is 2048, but it contains
> messages
> > >> whose MessageSize is larger than this. The MessageSetSize is always
> > 2048,
> > >> which makes me suspicious.
> > >>
> > >> I've tried feeding these message sets to the Java client (into
> > >> MemoryRecords.readableRecords), but when I iterate over it, it is
> empty.
> > >> Further inspecting what it does with the bytes I see that it gets an
> > >> EOFException and stops iterating, and I don't really understand what's
> > >> going on.
> > >>
> > >> I'm doing these fetch requests against a cluster running 0.8.2.1.
> > >>
> > >> yours,
> > >> Theo
> > >>
> > >
> > >
> >
>
Re: Problems decoding fetch responses, getting messages that are
bigger than their message sets
Posted by Magnus Edenhill <ma...@edenhill.se>.
Hi Theo,
seems like you found the answer yourself, the server may return partial
messages.
While parsing a MessageSet you simply ignore any message whose length >
remainingBufferLength.
Regards,
Magnus
2015-03-01 8:55 GMT+01:00 Theo Hultberg <th...@iconara.net>:
> That the message set size was always the same when this happened made me
> start looking for that number, and it turned out that was what I had set as
> the MaxBytes field of the requests. This make me think that what happens is
> that the next message to fetch is larger than this, but it's sent anyway.
> Is this correct? Is this how it is supposed to work? I'm looking through
> other clients to see how they deal with this, and I think I might have
> found the same thing in the Go client (
> https://github.com/Shopify/sarama/blob/master/message_set.go#L72-L76), and
> the corresponding comment in the protocol guide. Is what I have found the
> thing described as "As an optimization the server is allowed to return a
> partial message at the end of the message set. Clients should handle this
> case."?
>
> yours,
> Theo
>
> On Sat, Feb 28, 2015 at 10:19 PM, Theo Hultberg <th...@iconara.net> wrote:
>
> > Here's an example of a frame that looks malformed, I've split the bytes
> > apart and annotated the pieces with the names from the Kafka protocol
> guide
> > (
> >
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchResponse
> > ).
> >
> > Notice that the MessageSetSize is 0x00000800 (2048), but the MessageSize
> > is 0x000010B3 (4275). The size prefix of the Value field agrees with
> > MessageSize, but the size of the frame agrees with MessageSetSize,
> > because there are only 2022 bytes left in the frame. The bytes after the
> > crc field to the end of the frame don't checksum to the value of the crc
> > field either.
> >
> > (topics) 00 00 00 01
> > TopicName 00 1F (+ 31 character topic name)
> > (partitions) 00 00 00 01
> > Partition 00 00 00 02
> > ErrorCode 00 00
> > HighwaterMarkOffset 00 00 00 00 00 3F 5B 17
> > MessageSetSize 00 00 08 00
> > Offset 00 00 00 00 00 3F 5B 16
> > MessageSize 00 00 10 B3
> > Crc B0 13 F4 E3
> > MagicByte 00
> > Attributes 02
> > Key FF FF FF FF
> > Value 00 00 10 A5 (+ 2022 bytes left in frame)
> >
> > What's going on here, what have I misunderstood?
> >
> > yours,
> > Theo
> >
> > On Sat, Feb 28, 2015 at 8:35 PM, Theo Hultberg <th...@iconara.net> wrote:
> >
> >> Hi,
> >>
> >> Mostly out of curiosity I'm writing a Kafka client, and I'm getting
> stuck
> >> at decoding fetch responses. Most of the time everything goes fine, but
> >> quite often I get frames back that seem to be wrong. I'm sure I've
> >> misunderstood something about the spec, so maybe someone can get me on
> the
> >> right track.
> >>
> >> I get frames where the MessageSetSize is 2048, but it contains messages
> >> whose MessageSize is larger than this. The MessageSetSize is always
> 2048,
> >> which makes me suspicious.
> >>
> >> I've tried feeding these message sets to the Java client (into
> >> MemoryRecords.readableRecords), but when I iterate over it, it is empty.
> >> Further inspecting what it does with the bytes I see that it gets an
> >> EOFException and stops iterating, and I don't really understand what's
> >> going on.
> >>
> >> I'm doing these fetch requests against a cluster running 0.8.2.1.
> >>
> >> yours,
> >> Theo
> >>
> >
> >
>
Re: Problems decoding fetch responses, getting messages that are
bigger than their message sets
Posted by Theo Hultberg <th...@iconara.net>.
That the message set size was always the same when this happened made me
start looking for that number, and it turned out that was what I had set as
the MaxBytes field of the requests. This make me think that what happens is
that the next message to fetch is larger than this, but it's sent anyway.
Is this correct? Is this how it is supposed to work? I'm looking through
other clients to see how they deal with this, and I think I might have
found the same thing in the Go client (
https://github.com/Shopify/sarama/blob/master/message_set.go#L72-L76), and
the corresponding comment in the protocol guide. Is what I have found the
thing described as "As an optimization the server is allowed to return a
partial message at the end of the message set. Clients should handle this
case."?
yours,
Theo
On Sat, Feb 28, 2015 at 10:19 PM, Theo Hultberg <th...@iconara.net> wrote:
> Here's an example of a frame that looks malformed, I've split the bytes
> apart and annotated the pieces with the names from the Kafka protocol guide
> (
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchResponse
> ).
>
> Notice that the MessageSetSize is 0x00000800 (2048), but the MessageSize
> is 0x000010B3 (4275). The size prefix of the Value field agrees with
> MessageSize, but the size of the frame agrees with MessageSetSize,
> because there are only 2022 bytes left in the frame. The bytes after the
> crc field to the end of the frame don't checksum to the value of the crc
> field either.
>
> (topics) 00 00 00 01
> TopicName 00 1F (+ 31 character topic name)
> (partitions) 00 00 00 01
> Partition 00 00 00 02
> ErrorCode 00 00
> HighwaterMarkOffset 00 00 00 00 00 3F 5B 17
> MessageSetSize 00 00 08 00
> Offset 00 00 00 00 00 3F 5B 16
> MessageSize 00 00 10 B3
> Crc B0 13 F4 E3
> MagicByte 00
> Attributes 02
> Key FF FF FF FF
> Value 00 00 10 A5 (+ 2022 bytes left in frame)
>
> What's going on here, what have I misunderstood?
>
> yours,
> Theo
>
> On Sat, Feb 28, 2015 at 8:35 PM, Theo Hultberg <th...@iconara.net> wrote:
>
>> Hi,
>>
>> Mostly out of curiosity I'm writing a Kafka client, and I'm getting stuck
>> at decoding fetch responses. Most of the time everything goes fine, but
>> quite often I get frames back that seem to be wrong. I'm sure I've
>> misunderstood something about the spec, so maybe someone can get me on the
>> right track.
>>
>> I get frames where the MessageSetSize is 2048, but it contains messages
>> whose MessageSize is larger than this. The MessageSetSize is always 2048,
>> which makes me suspicious.
>>
>> I've tried feeding these message sets to the Java client (into
>> MemoryRecords.readableRecords), but when I iterate over it, it is empty.
>> Further inspecting what it does with the bytes I see that it gets an
>> EOFException and stops iterating, and I don't really understand what's
>> going on.
>>
>> I'm doing these fetch requests against a cluster running 0.8.2.1.
>>
>> yours,
>> Theo
>>
>
>
Re: Problems decoding fetch responses, getting messages that are
bigger than their message sets
Posted by Theo Hultberg <th...@iconara.net>.
Here's an example of a frame that looks malformed, I've split the bytes
apart and annotated the pieces with the names from the Kafka protocol guide
(
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchResponse
).
Notice that the MessageSetSize is 0x00000800 (2048), but the MessageSize is
0x000010B3 (4275). The size prefix of the Value field agrees with
MessageSize, but the size of the frame agrees with MessageSetSize, because
there are only 2022 bytes left in the frame. The bytes after the crc field
to the end of the frame don't checksum to the value of the crc field either.
(topics) 00 00 00 01
TopicName 00 1F (+ 31 character topic name)
(partitions) 00 00 00 01
Partition 00 00 00 02
ErrorCode 00 00
HighwaterMarkOffset 00 00 00 00 00 3F 5B 17
MessageSetSize 00 00 08 00
Offset 00 00 00 00 00 3F 5B 16
MessageSize 00 00 10 B3
Crc B0 13 F4 E3
MagicByte 00
Attributes 02
Key FF FF FF FF
Value 00 00 10 A5 (+ 2022 bytes left in frame)
What's going on here, what have I misunderstood?
yours,
Theo
On Sat, Feb 28, 2015 at 8:35 PM, Theo Hultberg <th...@iconara.net> wrote:
> Hi,
>
> Mostly out of curiosity I'm writing a Kafka client, and I'm getting stuck
> at decoding fetch responses. Most of the time everything goes fine, but
> quite often I get frames back that seem to be wrong. I'm sure I've
> misunderstood something about the spec, so maybe someone can get me on the
> right track.
>
> I get frames where the MessageSetSize is 2048, but it contains messages
> whose MessageSize is larger than this. The MessageSetSize is always 2048,
> which makes me suspicious.
>
> I've tried feeding these message sets to the Java client (into
> MemoryRecords.readableRecords), but when I iterate over it, it is empty.
> Further inspecting what it does with the bytes I see that it gets an
> EOFException and stops iterating, and I don't really understand what's
> going on.
>
> I'm doing these fetch requests against a cluster running 0.8.2.1.
>
> yours,
> Theo
>