You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by je...@tutanota.com on 2018/07/03 00:08:40 UTC
RecordTooLargeException
Hi all,
I get this error even when my records are smaller than the 1000012 byte limit:
org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept.
How do I ensure that my producer doesn't send records that are too large?
Thanks in advance for any suggestions and help.
Re: RecordTooLargeException
Posted by Jerry Richardson <je...@tutanota.com>.
Setting max.request.size does not prevent RecordTooLargeException from being thrown.
Thanks for the header info.
9. Jul 2018 09:20 by jiangtao.liu@zuora.com <ma...@zuora.com>.INVALID:
> you can configure max.request.size
> <> https://kafka.apache.org/documentation/#configuration <https://kafka.apache.org/documentation/#configuration>> > when you create
> Kafka producer, about the second questions, Where is the key size + value
> size + header size calculated? you can check `
>
> org.apache.kafka.common.record.AbstractRecords.estimateSizeInBytesUpperBound()
> {...}
> `
>
> Ty
>
>
> On Fri, Jul 6, 2018 at 12:42 AM, Jerry Richardson <
> jerryrichardson@tutanota.com <ma...@tutanota.com>> > wrote:
>
>> My administrator will not allow messages larger than 1MB to be stored in
>> Kafka.
>> How can I limit the size of my messages to 1MB? If I have a message larger
>> than 1MB, I want to truncate or throw away the message to avoid the
>> RecordTooLargeException. What is the max size of the headers? Where is
>> the key size + value size + header size calculated?
>>
>> 5. Jul 2018 14:15 by >> jiangtao.liu@zuora.com <ma...@zuora.com>>> <>> mailto:jiangtao.liu@zuora.com <ma...@zuora.com>
>> >.INVALID:
>>
>>
>> > sorry, just saw your email a little late.
>> >
>> > I am confused with why you said `I cannot increase the max size of
>> messages
>> > stored in Kafka` ? can we add some explanation ?
>> >
>> > On Thu, Jul 5, 2018 at 10:09 AM, Jerry Richardson <
>> > >> jerryrichardson@tutanota.com <ma...@tutanota.com>>> <>> mailto:jerryrichardson@tutanota.com <ma...@tutanota.com>>> >> >
>> wrote:
>> >
>> >>
>> >> What class is this in? What's the maximum header size? Is there
>> >> documentation on this?
>> >>
>> >> 2. Jul 2018 18:29 by >> >> jiangtao.liu@zuora.com <ma...@zuora.com>>> <mailto:
>> jiangtao.liu@zuora.com <ma...@zuora.com>>> >>> <>> >> mailto:jiangtao.liu@zuora.com <ma...@zuora.com>>> <mailto:
>> mailto:jiangtao.liu@zuora.com <ma...@zuora.com>>> >
>> >> >.INVALID:
>> >>
>> >>
>> >> > You can consider increase `max.request.size` a little big (the
>> default
>> >> > value is `1048576`.), after checking Kafka client source code, they
>> count
>> >> > [`key size` + `value size` + `header size` + others] together, so it's
>> >> > possible the calculated size is a little bigger than the default
>> value.
>> >> >
>> >> > please check with > >> >> https://kafka.apache.org <https://kafka.apache.org>>> <
>> https://kafka.apache.org <https://kafka.apache.org/>>> >
>> >> documentation/#configuration <>> >> https://kafka.apache.org <https://kafka.apache.org>>> <
>> https://kafka.apache.org <https://kafka.apache.org/>>> >
>> >> documentation/#configuration>> .
>> >> >
>> >> > On Mon, Jul 2, 2018 at 5:08 PM, <> >> >> jerryrichardson@tutanota.com <ma...@tutanota.com>
>> <>> mailto:jerryrichardson@tutanota.com <ma...@tutanota.com>>> >>> <mailto:
>> >> >> jerryrichardson@tutanota.com <ma...@tutanota.com>>> <>> mailto:jerryrichardson@tutanota.com <ma...@tutanota.com>>> >>>
>> >> > wrote:
>> >> >
>> >> >> Hi all,
>> >> >>
>> >> >> I get this error even when my records are smaller than the 1000012
>> byte
>> >> >> limit:
>> >> >>
>> >> >> org.apache.kafka.common.errors.RecordTooLargeException: The request
>> >> >> included a message larger than the max message size the server will
>> >> accept.
>> >> >>
>> >> >> How do I ensure that my producer doesn't send records that are too
>> >> large?
>> >> >>
>> >> >> Thanks in advance for any suggestions and help.
>> >> >>
>> >>
>>
Re: RecordTooLargeException
Posted by Tony Liu <ji...@zuora.com.INVALID>.
you can configure max.request.size
<https://kafka.apache.org/documentation/#configuration> when you create
Kafka producer, about the second questions, Where is the key size + value
size + header size calculated? you can check `
org.apache.kafka.common.record.AbstractRecords.estimateSizeInBytesUpperBound()
{...}
`
Ty
On Fri, Jul 6, 2018 at 12:42 AM, Jerry Richardson <
jerryrichardson@tutanota.com> wrote:
> My administrator will not allow messages larger than 1MB to be stored in
> Kafka.
> How can I limit the size of my messages to 1MB? If I have a message larger
> than 1MB, I want to truncate or throw away the message to avoid the
> RecordTooLargeException. What is the max size of the headers? Where is
> the key size + value size + header size calculated?
>
> 5. Jul 2018 14:15 by jiangtao.liu@zuora.com <mailto:jiangtao.liu@zuora.com
> >.INVALID:
>
>
> > sorry, just saw your email a little late.
> >
> > I am confused with why you said `I cannot increase the max size of
> messages
> > stored in Kafka` ? can we add some explanation ?
> >
> > On Thu, Jul 5, 2018 at 10:09 AM, Jerry Richardson <
> > jerryrichardson@tutanota.com <ma...@tutanota.com>> >
> wrote:
> >
> >>
> >> What class is this in? What's the maximum header size? Is there
> >> documentation on this?
> >>
> >> 2. Jul 2018 18:29 by >> jiangtao.liu@zuora.com <mailto:
> jiangtao.liu@zuora.com>>> <>> mailto:jiangtao.liu@zuora.com <mailto:
> mailto:jiangtao.liu@zuora.com>
> >> >.INVALID:
> >>
> >>
> >> > You can consider increase `max.request.size` a little big (the
> default
> >> > value is `1048576`.), after checking Kafka client source code, they
> count
> >> > [`key size` + `value size` + `header size` + others] together, so it's
> >> > possible the calculated size is a little bigger than the default
> value.
> >> >
> >> > please check with > >> https://kafka.apache.org <
> https://kafka.apache.org/>
> >> documentation/#configuration <>> https://kafka.apache.org <
> https://kafka.apache.org/>
> >> documentation/#configuration>> .
> >> >
> >> > On Mon, Jul 2, 2018 at 5:08 PM, <> >> jerryrichardson@tutanota.com
> <ma...@tutanota.com>>> <mailto:
> >> jerryrichardson@tutanota.com <ma...@tutanota.com>>>
> >> > wrote:
> >> >
> >> >> Hi all,
> >> >>
> >> >> I get this error even when my records are smaller than the 1000012
> byte
> >> >> limit:
> >> >>
> >> >> org.apache.kafka.common.errors.RecordTooLargeException: The request
> >> >> included a message larger than the max message size the server will
> >> accept.
> >> >>
> >> >> How do I ensure that my producer doesn't send records that are too
> >> large?
> >> >>
> >> >> Thanks in advance for any suggestions and help.
> >> >>
> >>
>
Re: RecordTooLargeException
Posted by Jerry Richardson <je...@tutanota.com>.
My administrator will not allow messages larger than 1MB to be stored in Kafka.
How can I limit the size of my messages to 1MB? If I have a message larger than 1MB, I want to truncate or throw away the message to avoid the RecordTooLargeException. What is the max size of the headers? Where is the key size + value size + header size calculated?
5. Jul 2018 14:15 by jiangtao.liu@zuora.com <ma...@zuora.com>.INVALID:
> sorry, just saw your email a little late.
>
> I am confused with why you said `I cannot increase the max size of messages
> stored in Kafka` ? can we add some explanation ?
>
> On Thu, Jul 5, 2018 at 10:09 AM, Jerry Richardson <
> jerryrichardson@tutanota.com <ma...@tutanota.com>> > wrote:
>
>>
>> What class is this in? What's the maximum header size? Is there
>> documentation on this?
>>
>> 2. Jul 2018 18:29 by >> jiangtao.liu@zuora.com <ma...@zuora.com>>> <>> mailto:jiangtao.liu@zuora.com <ma...@zuora.com>
>> >.INVALID:
>>
>>
>> > You can consider increase `max.request.size` a little big (the default
>> > value is `1048576`.), after checking Kafka client source code, they count
>> > [`key size` + `value size` + `header size` + others] together, so it's
>> > possible the calculated size is a little bigger than the default value.
>> >
>> > please check with > >> https://kafka.apache.org <https://kafka.apache.org/>
>> documentation/#configuration <>> https://kafka.apache.org <https://kafka.apache.org/>
>> documentation/#configuration>> .
>> >
>> > On Mon, Jul 2, 2018 at 5:08 PM, <> >> jerryrichardson@tutanota.com <ma...@tutanota.com>>> <mailto:
>> jerryrichardson@tutanota.com <ma...@tutanota.com>>> >> > wrote:
>> >
>> >> Hi all,
>> >>
>> >> I get this error even when my records are smaller than the 1000012 byte
>> >> limit:
>> >>
>> >> org.apache.kafka.common.errors.RecordTooLargeException: The request
>> >> included a message larger than the max message size the server will
>> accept.
>> >>
>> >> How do I ensure that my producer doesn't send records that are too
>> large?
>> >>
>> >> Thanks in advance for any suggestions and help.
>> >>
>>
Re: RecordTooLargeException
Posted by Tony Liu <ji...@zuora.com.INVALID>.
sorry, just saw your email a little late.
I am confused with why you said `I cannot increase the max size of messages
stored in Kafka` ? can we add some explanation ?
On Thu, Jul 5, 2018 at 10:09 AM, Jerry Richardson <
jerryrichardson@tutanota.com> wrote:
>
> What class is this in? What's the maximum header size? Is there
> documentation on this?
>
> 2. Jul 2018 18:29 by jiangtao.liu@zuora.com <mailto:jiangtao.liu@zuora.com
> >.INVALID:
>
>
> > You can consider increase `max.request.size` a little big (the default
> > value is `1048576`.), after checking Kafka client source code, they count
> > [`key size` + `value size` + `header size` + others] together, so it's
> > possible the calculated size is a little bigger than the default value.
> >
> > please check with > https://kafka.apache.org/
> documentation/#configuration <https://kafka.apache.org/
> documentation/#configuration>> .
> >
> > On Mon, Jul 2, 2018 at 5:08 PM, <> jerryrichardson@tutanota.com <mailto:
> jerryrichardson@tutanota.com>> > wrote:
> >
> >> Hi all,
> >>
> >> I get this error even when my records are smaller than the 1000012 byte
> >> limit:
> >>
> >> org.apache.kafka.common.errors.RecordTooLargeException: The request
> >> included a message larger than the max message size the server will
> accept.
> >>
> >> How do I ensure that my producer doesn't send records that are too
> large?
> >>
> >> Thanks in advance for any suggestions and help.
> >>
>
Re: RecordTooLargeException
Posted by Jerry Richardson <je...@tutanota.com>.
What class is this in? What's the maximum header size? Is there documentation on this?
2. Jul 2018 18:29 by jiangtao.liu@zuora.com <ma...@zuora.com>.INVALID:
> You can consider increase `max.request.size` a little big (the default
> value is `1048576`.), after checking Kafka client source code, they count
> [`key size` + `value size` + `header size` + others] together, so it's
> possible the calculated size is a little bigger than the default value.
>
> please check with > https://kafka.apache.org/documentation/#configuration <https://kafka.apache.org/documentation/#configuration>> .
>
> On Mon, Jul 2, 2018 at 5:08 PM, <> jerryrichardson@tutanota.com <ma...@tutanota.com>> > wrote:
>
>> Hi all,
>>
>> I get this error even when my records are smaller than the 1000012 byte
>> limit:
>>
>> org.apache.kafka.common.errors.RecordTooLargeException: The request
>> included a message larger than the max message size the server will accept.
>>
>> How do I ensure that my producer doesn't send records that are too large?
>>
>> Thanks in advance for any suggestions and help.
>>
Re: RecordTooLargeException
Posted by je...@tutanota.com.
I cannot increase the max size of messages stored in Kafka. How do I limit them to avoid the RecordTooLargeException?
2. Jul 2018 18:29 by jiangtao.liu@zuora.com <ma...@zuora.com>.INVALID:
> You can consider increase `max.request.size` a little big (the default
> value is `1048576`.), after checking Kafka client source code, they count
> [`key size` + `value size` + `header size` + others] together, so it's
> possible the calculated size is a little bigger than the default value.
>
> please check with > https://kafka.apache.org/documentation/#configuration <https://kafka.apache.org/documentation/#configuration>> .
>
> On Mon, Jul 2, 2018 at 5:08 PM, <> jerryrichardson@tutanota.com <ma...@tutanota.com>> > wrote:
>
>> Hi all,
>>
>> I get this error even when my records are smaller than the 1000012 byte
>> limit:
>>
>> org.apache.kafka.common.errors.RecordTooLargeException: The request
>> included a message larger than the max message size the server will accept.
>>
>> How do I ensure that my producer doesn't send records that are too large?
>>
>> Thanks in advance for any suggestions and help.
>>
Re: RecordTooLargeException
Posted by Tony Liu <ji...@zuora.com.INVALID>.
You can consider increase `max.request.size` a little big (the default
value is `1048576`.), after checking Kafka client source code, they count
[`key size` + `value size` + `header size` + others] together, so it's
possible the calculated size is a little bigger than the default value.
please check with https://kafka.apache.org/documentation/#configuration.
On Mon, Jul 2, 2018 at 5:08 PM, <je...@tutanota.com> wrote:
> Hi all,
>
> I get this error even when my records are smaller than the 1000012 byte
> limit:
>
> org.apache.kafka.common.errors.RecordTooLargeException: The request
> included a message larger than the max message size the server will accept.
>
> How do I ensure that my producer doesn't send records that are too large?
>
> Thanks in advance for any suggestions and help.
>