You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jason Rosenberg <jb...@squareup.com> on 2013/09/27 05:11:24 UTC
understanding MessageSizeTooLarge and batches
I've been doing some testing, trying to understand how the
max.message.bytes works, with respect to sending batches of messages. In a
previous discussion, there appeared to be a suggestion that one work around
when triggering a MessageSizeTooLargeException is to reduce the batch size
and resubmit the batch in smaller sub-batches.
However, so far in my testing, I'm not seeing a way to trigger a
MessageSizeTooLargeException by having a large batch of messages, whose
cumulative size is greater than max.message.bytes, but without any
individual message exceeding the max. And in fact, I'm able to send
through some very large batches. E.g. 200 messages of 500000 bytes each in
a single batch (where the max.message.bytes is 1000000. However, if
anyone of the messages in the batch exceeds that limit, then it will reject
the whole batch.
Is this the expected behavior? What was being referred to in the previous
discussions around retrying smaller batch sizes as a work-around for
MessageSizeTooLargeException.
I haven't been using compression with any of these tests, does that make a
difference?
Thanks,
Jason
Re: understanding MessageSizeTooLarge and batches
Posted by Jun Rao <ju...@gmail.com>.
Jason,
Glad that you asked. This is only an issue if compression is turned on for
a batch of messages. The problem is that the compressed batch is treated as
a single message, whose size has to be smaller than max.message.bytes. If
you are sending data uncompressed, only each uncompressed message needs to
be less than max.message.bytes. The batch size doesn't really matter in
this case.
Thanks,
Jun
On Thu, Sep 26, 2013 at 8:11 PM, Jason Rosenberg <jb...@squareup.com> wrote:
> I've been doing some testing, trying to understand how the
> max.message.bytes works, with respect to sending batches of messages. In a
> previous discussion, there appeared to be a suggestion that one work around
> when triggering a MessageSizeTooLargeException is to reduce the batch size
> and resubmit the batch in smaller sub-batches.
>
> However, so far in my testing, I'm not seeing a way to trigger a
> MessageSizeTooLargeException by having a large batch of messages, whose
> cumulative size is greater than max.message.bytes, but without any
> individual message exceeding the max. And in fact, I'm able to send
> through some very large batches. E.g. 200 messages of 500000 bytes each in
> a single batch (where the max.message.bytes is 1000000. However, if
> anyone of the messages in the batch exceeds that limit, then it will reject
> the whole batch.
>
> Is this the expected behavior? What was being referred to in the previous
> discussions around retrying smaller batch sizes as a work-around for
> MessageSizeTooLargeException.
>
> I haven't been using compression with any of these tests, does that make a
> difference?
>
> Thanks,
>
> Jason
>