You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Aaron Raddon <ar...@gmail.com> on 2012/01/03 23:08:28 UTC

Partial message when packet size = max size?

Hello;

I am using the golang Consumer and ran into some errors when there was
a large backlog (greater than max message size) of messages on broker
and had a question about the format.

In the case above, the behavior i was seeing was the total packet
bytes size == max message (containing many messages), even if this
meant a partial/incomplete last message.   Is this correct?

I wrote a fix for it in the driver, but had second thoughts if this was right.

My Kafka broker was built from source ~ 1 week ago.

Thanks

Aaron

Re: Partial message when packet size = max size?

Posted by Jun Rao <ju...@gmail.com>.
Aaron,

The consumer has to use a fetch size larger than the largest single message
(+ overhead, about 10 bytes). Otherwise, the consumer won't be able to
decode a single message from a fetched chunk and will get stuck.

Thanks,

Jun

On Tue, Jan 3, 2012 at 2:08 PM, Aaron Raddon <ar...@gmail.com> wrote:

> Hello;
>
> I am using the golang Consumer and ran into some errors when there was
> a large backlog (greater than max message size) of messages on broker
> and had a question about the format.
>
> In the case above, the behavior i was seeing was the total packet
> bytes size == max message (containing many messages), even if this
> meant a partial/incomplete last message.   Is this correct?
>
> I wrote a fix for it in the driver, but had second thoughts if this was
> right.
>
> My Kafka broker was built from source ~ 1 week ago.
>
> Thanks
>
> Aaron
>