You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by l vic <lv...@gmail.com> on 2019/08/15 01:23:20 UTC
RecordTooLargeException on 16M messages in Kafka?
My kafka (1.0.0) producer errors out on large (16M) messages.
ERROR Error when sending message to topic test with key: null, value:
16777239 bytes with error: (org.apache.kafka.clients.producer.internals.
ErrorLoggingCallback)
org.apache.kafka.common.errors.RecordTooLargeException: The message is
16777327 bytes when serialized which is larger than the maximum request
size you have configured with the max.request.size configuration.
I found couple of links describing the solution:
*https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
<https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb>*
in my server.properties on brokers I set:
socket.request.max.bytes=104857600
message.max.bytes=18874368
max.request.size=18874368
replica.fetch.max.bytes=18874368
fetch.message.max.bytes=18874368
Then in my producer.properties i tried to set
max.request.size=18874368
But no matter how i large i try to set max.request.size -
i still have the same problem...Are there other settings i am missing?
Can it be solved in configuration alone, or do i need to make code changes?
Thank you,
Re: RecordTooLargeException on 16M messages in Kafka?
Posted by l vic <lv...@gmail.com>.
Yes, it's still there
On Thu, Aug 15, 2019 at 4:49 AM Jonathan Santilli <
jonathansantilli@gmail.com> wrote:
> Hello, try to send and flush just one message of 16777239 bytes, to verify
> the error still shows up.
>
> Cheers!
> --
> Jonathan
>
>
>
> On Thu, Aug 15, 2019 at 2:23 AM l vic <lv...@gmail.com> wrote:
>
> > My kafka (1.0.0) producer errors out on large (16M) messages.
> > ERROR Error when sending message to topic test with key: null, value:
> > 16777239 bytes with error: (org.apache.kafka.clients.producer.internals.
> > ErrorLoggingCallback)
> >
> > org.apache.kafka.common.errors.RecordTooLargeException: The message is
> > 16777327 bytes when serialized which is larger than the maximum request
> > size you have configured with the max.request.size configuration.
> > I found couple of links describing the solution:
> > *
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > <
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > >*
> >
> > in my server.properties on brokers I set:
> > socket.request.max.bytes=104857600
> > message.max.bytes=18874368
> > max.request.size=18874368
> > replica.fetch.max.bytes=18874368
> > fetch.message.max.bytes=18874368
> >
> > Then in my producer.properties i tried to set
> > max.request.size=18874368
> >
> > But no matter how i large i try to set max.request.size -
> > i still have the same problem...Are there other settings i am missing?
> > Can it be solved in configuration alone, or do i need to make code
> changes?
> > Thank you,
> >
>
>
> --
> Santilli Jonathan
>
Re: RecordTooLargeException on 16M messages in Kafka?
Posted by "claude.warren@wipro.com.INVALID" <cl...@wipro.com.INVALID>.
I had a similar problem before. I could find no way for the producer to determine the smallest maximum buffer size of the servers.
We had an issue where we had very large items that we wanted to send through kafka, basically to tunnel through a firewall.
We utilized an open source project that I have to allow a logical buffer to span multiple smaller buffers[1] [2]
We were then able to split the large file into multiple messages and reassemble them on the remote side only reading from Kafka when the data in the specific buffer was needed.
We still do not have an answer for figuring out, from the producer, what the smalled maximum buffer size of the servers is, so we just picked a suitably small number to make it work.
Claude
[1] https://github.com/Claudenw/spanbuffer
[2] Kafka specific code is not included in the spanbuffer project but it is fairly simple to implement. I have plans to do so in the future.
________________________________
From: l vic <lv...@gmail.com>
Sent: Thursday, August 15, 2019 13:48
To: users@kafka.apache.org <us...@kafka.apache.org>
Subject: Re: RecordTooLargeException on 16M messages in Kafka?
** This mail has been sent from an external source. Treat hyperlinks and attachments in this email with caution**
I tested it with kafka-console-consumer and kafka-console-producer reading
from 16M text file (no newlines):
kafka-console-producer.sh --broker-list <broker-ip>:6667 --topic test <
./large-file
The error comes out on producer side:
org.apache.kafka.common.errors.RecordTooLargeException: The message is
16777239 bytes when serialized which is larger than the maximum request
size you have configured with the max.request.size configuration.
On Thu, Aug 15, 2019 at 4:49 AM Jonathan Santilli <
jonathansantilli@gmail.com> wrote:
> Hello, try to send and flush just one message of 16777239 bytes, to verify
> the error still shows up.
>
> Cheers!
> --
> Jonathan
>
>
>
> On Thu, Aug 15, 2019 at 2:23 AM l vic <lv...@gmail.com> wrote:
>
> > My kafka (1.0.0) producer errors out on large (16M) messages.
> > ERROR Error when sending message to topic test with key: null, value:
> > 16777239 bytes with error: (org.apache.kafka.clients.producer.internals.
> > ErrorLoggingCallback)
> >
> > org.apache.kafka.common.errors.RecordTooLargeException: The message is
> > 16777327 bytes when serialized which is larger than the maximum request
> > size you have configured with the max.request.size configuration.
> > I found couple of links describing the solution:
> > *
> >
> https://clicktime.symantec.com/3RuErYUTQJAnPJRNn8RUTiW7Vc?u=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F21020347%2Fhow-can-i-send-large-messages-with-kafka-over-15mb
> > <
> >
> https://clicktime.symantec.com/3RuErYUTQJAnPJRNn8RUTiW7Vc?u=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F21020347%2Fhow-can-i-send-large-messages-with-kafka-over-15mb
> > >*
> >
> > in my server.properties on brokers I set:
> > socket.request.max.bytes=104857600
> > message.max.bytes=18874368
> > max.request.size=18874368
> > replica.fetch.max.bytes=18874368
> > fetch.message.max.bytes=18874368
> >
> > Then in my producer.properties i tried to set
> > max.request.size=18874368
> >
> > But no matter how i large i try to set max.request.size -
> > i still have the same problem...Are there other settings i am missing?
> > Can it be solved in configuration alone, or do i need to make code
> changes?
> > Thank you,
> >
>
>
> --
> Santilli Jonathan
>
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
Re: RecordTooLargeException on 16M messages in Kafka?
Posted by l vic <lv...@gmail.com>.
I tested it with kafka-console-consumer and kafka-console-producer reading
from 16M text file (no newlines):
kafka-console-producer.sh --broker-list <broker-ip>:6667 --topic test <
./large-file
The error comes out on producer side:
org.apache.kafka.common.errors.RecordTooLargeException: The message is
16777239 bytes when serialized which is larger than the maximum request
size you have configured with the max.request.size configuration.
On Thu, Aug 15, 2019 at 4:49 AM Jonathan Santilli <
jonathansantilli@gmail.com> wrote:
> Hello, try to send and flush just one message of 16777239 bytes, to verify
> the error still shows up.
>
> Cheers!
> --
> Jonathan
>
>
>
> On Thu, Aug 15, 2019 at 2:23 AM l vic <lv...@gmail.com> wrote:
>
> > My kafka (1.0.0) producer errors out on large (16M) messages.
> > ERROR Error when sending message to topic test with key: null, value:
> > 16777239 bytes with error: (org.apache.kafka.clients.producer.internals.
> > ErrorLoggingCallback)
> >
> > org.apache.kafka.common.errors.RecordTooLargeException: The message is
> > 16777327 bytes when serialized which is larger than the maximum request
> > size you have configured with the max.request.size configuration.
> > I found couple of links describing the solution:
> > *
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > <
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > >*
> >
> > in my server.properties on brokers I set:
> > socket.request.max.bytes=104857600
> > message.max.bytes=18874368
> > max.request.size=18874368
> > replica.fetch.max.bytes=18874368
> > fetch.message.max.bytes=18874368
> >
> > Then in my producer.properties i tried to set
> > max.request.size=18874368
> >
> > But no matter how i large i try to set max.request.size -
> > i still have the same problem...Are there other settings i am missing?
> > Can it be solved in configuration alone, or do i need to make code
> changes?
> > Thank you,
> >
>
>
> --
> Santilli Jonathan
>
Re: RecordTooLargeException on 16M messages in Kafka?
Posted by Jonathan Santilli <jo...@gmail.com>.
Ok, it means that, using the KafkaProducer class setting the config
max.request.size worked correctly, right?
Now you are wondering about the kafka-console-producer.sh utility, right?
if so, have you tried using the parameter:
--producer.config /path/to/the/config/producer.properties
Just let me know, cheers!
--
Jonathan
On Thu, Aug 15, 2019 at 8:09 PM l vic <lv...@gmail.com> wrote:
> setting max.request.size worked when i did it in producer code; don't
> understand what producer.properties is for, seems like it's not used.
>
> On Thu, Aug 15, 2019 at 2:25 PM Jonathan Santilli <
> jonathansantilli@gmail.com> wrote:
>
> > I have asked because I did not see that in your previous email when you
> > tried the console producer.
> >
> > Jonathan.
> >
> > On Thu, Aug 15, 2019, 3:07 PM l vic <lv...@gmail.com> wrote:
> >
> > > yes, in producer.properties
> > >
> > > On Thu, Aug 15, 2019 at 9:59 AM Jonathan Santilli <
> > > jonathansantilli@gmail.com> wrote:
> > >
> > > > Just to be sure, please confirm the configuration parameter is well
> > > > set/configure at producer level:
> > > >
> > > > max.request.size = 177772390 (for instance)
> > > >
> > > > Cheers!
> > > > --
> > > > Jonathan
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Aug 15, 2019 at 1:44 PM l vic <lv...@gmail.com> wrote:
> > > >
> > > > > I tested it with kafka-console-consumer and kafka-console-producer
> > > > reading
> > > > > from large text file (no newlines):
> > > > >
> > > > > kafka-console-producer.sh --broker-list 10.10.105.24:6667
> --topic
> > > > test <
> > > > > ./large-file
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Aug 15, 2019 at 4:49 AM Jonathan Santilli <
> > > > > jonathansantilli@gmail.com> wrote:
> > > > >
> > > > > > Hello, try to send and flush just one message of 16777239 bytes,
> to
> > > > > verify
> > > > > > the error still shows up.
> > > > > >
> > > > > > Cheers!
> > > > > > --
> > > > > > Jonathan
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Aug 15, 2019 at 2:23 AM l vic <lv...@gmail.com>
> wrote:
> > > > > >
> > > > > > > My kafka (1.0.0) producer errors out on large (16M) messages.
> > > > > > > ERROR Error when sending message to topic test with key: null,
> > > value:
> > > > > > > 16777239 bytes with error:
> > > > > (org.apache.kafka.clients.producer.internals.
> > > > > > > ErrorLoggingCallback)
> > > > > > >
> > > > > > > org.apache.kafka.common.errors.RecordTooLargeException: The
> > message
> > > > is
> > > > > > > 16777327 bytes when serialized which is larger than the maximum
> > > > request
> > > > > > > size you have configured with the max.request.size
> configuration.
> > > > > > > I found couple of links describing the solution:
> > > > > > > *
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > > > > > > >*
> > > > > > >
> > > > > > > in my server.properties on brokers I set:
> > > > > > > socket.request.max.bytes=104857600
> > > > > > > message.max.bytes=18874368
> > > > > > > max.request.size=18874368
> > > > > > > replica.fetch.max.bytes=18874368
> > > > > > > fetch.message.max.bytes=18874368
> > > > > > >
> > > > > > > Then in my producer.properties i tried to set
> > > > > > > max.request.size=18874368
> > > > > > >
> > > > > > > But no matter how i large i try to set max.request.size -
> > > > > > > i still have the same problem...Are there other settings i am
> > > > missing?
> > > > > > > Can it be solved in configuration alone, or do i need to make
> > code
> > > > > > changes?
> > > > > > > Thank you,
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Santilli Jonathan
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Santilli Jonathan
> > > >
> > >
> >
>
--
Santilli Jonathan
Re: RecordTooLargeException on 16M messages in Kafka?
Posted by l vic <lv...@gmail.com>.
setting max.request.size worked when i did it in producer code; don't
understand what producer.properties is for, seems like it's not used.
On Thu, Aug 15, 2019 at 2:25 PM Jonathan Santilli <
jonathansantilli@gmail.com> wrote:
> I have asked because I did not see that in your previous email when you
> tried the console producer.
>
> Jonathan.
>
> On Thu, Aug 15, 2019, 3:07 PM l vic <lv...@gmail.com> wrote:
>
> > yes, in producer.properties
> >
> > On Thu, Aug 15, 2019 at 9:59 AM Jonathan Santilli <
> > jonathansantilli@gmail.com> wrote:
> >
> > > Just to be sure, please confirm the configuration parameter is well
> > > set/configure at producer level:
> > >
> > > max.request.size = 177772390 (for instance)
> > >
> > > Cheers!
> > > --
> > > Jonathan
> > >
> > >
> > >
> > >
> > > On Thu, Aug 15, 2019 at 1:44 PM l vic <lv...@gmail.com> wrote:
> > >
> > > > I tested it with kafka-console-consumer and kafka-console-producer
> > > reading
> > > > from large text file (no newlines):
> > > >
> > > > kafka-console-producer.sh --broker-list 10.10.105.24:6667 --topic
> > > test <
> > > > ./large-file
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Aug 15, 2019 at 4:49 AM Jonathan Santilli <
> > > > jonathansantilli@gmail.com> wrote:
> > > >
> > > > > Hello, try to send and flush just one message of 16777239 bytes, to
> > > > verify
> > > > > the error still shows up.
> > > > >
> > > > > Cheers!
> > > > > --
> > > > > Jonathan
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Aug 15, 2019 at 2:23 AM l vic <lv...@gmail.com> wrote:
> > > > >
> > > > > > My kafka (1.0.0) producer errors out on large (16M) messages.
> > > > > > ERROR Error when sending message to topic test with key: null,
> > value:
> > > > > > 16777239 bytes with error:
> > > > (org.apache.kafka.clients.producer.internals.
> > > > > > ErrorLoggingCallback)
> > > > > >
> > > > > > org.apache.kafka.common.errors.RecordTooLargeException: The
> message
> > > is
> > > > > > 16777327 bytes when serialized which is larger than the maximum
> > > request
> > > > > > size you have configured with the max.request.size configuration.
> > > > > > I found couple of links describing the solution:
> > > > > > *
> > > > > >
> > > > >
> > > >
> > >
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > > > > > >*
> > > > > >
> > > > > > in my server.properties on brokers I set:
> > > > > > socket.request.max.bytes=104857600
> > > > > > message.max.bytes=18874368
> > > > > > max.request.size=18874368
> > > > > > replica.fetch.max.bytes=18874368
> > > > > > fetch.message.max.bytes=18874368
> > > > > >
> > > > > > Then in my producer.properties i tried to set
> > > > > > max.request.size=18874368
> > > > > >
> > > > > > But no matter how i large i try to set max.request.size -
> > > > > > i still have the same problem...Are there other settings i am
> > > missing?
> > > > > > Can it be solved in configuration alone, or do i need to make
> code
> > > > > changes?
> > > > > > Thank you,
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Santilli Jonathan
> > > > >
> > > >
> > >
> > >
> > > --
> > > Santilli Jonathan
> > >
> >
>
Re: RecordTooLargeException on 16M messages in Kafka?
Posted by Jonathan Santilli <jo...@gmail.com>.
I have asked because I did not see that in your previous email when you
tried the console producer.
Jonathan.
On Thu, Aug 15, 2019, 3:07 PM l vic <lv...@gmail.com> wrote:
> yes, in producer.properties
>
> On Thu, Aug 15, 2019 at 9:59 AM Jonathan Santilli <
> jonathansantilli@gmail.com> wrote:
>
> > Just to be sure, please confirm the configuration parameter is well
> > set/configure at producer level:
> >
> > max.request.size = 177772390 (for instance)
> >
> > Cheers!
> > --
> > Jonathan
> >
> >
> >
> >
> > On Thu, Aug 15, 2019 at 1:44 PM l vic <lv...@gmail.com> wrote:
> >
> > > I tested it with kafka-console-consumer and kafka-console-producer
> > reading
> > > from large text file (no newlines):
> > >
> > > kafka-console-producer.sh --broker-list 10.10.105.24:6667 --topic
> > test <
> > > ./large-file
> > >
> > >
> > >
> > >
> > > On Thu, Aug 15, 2019 at 4:49 AM Jonathan Santilli <
> > > jonathansantilli@gmail.com> wrote:
> > >
> > > > Hello, try to send and flush just one message of 16777239 bytes, to
> > > verify
> > > > the error still shows up.
> > > >
> > > > Cheers!
> > > > --
> > > > Jonathan
> > > >
> > > >
> > > >
> > > > On Thu, Aug 15, 2019 at 2:23 AM l vic <lv...@gmail.com> wrote:
> > > >
> > > > > My kafka (1.0.0) producer errors out on large (16M) messages.
> > > > > ERROR Error when sending message to topic test with key: null,
> value:
> > > > > 16777239 bytes with error:
> > > (org.apache.kafka.clients.producer.internals.
> > > > > ErrorLoggingCallback)
> > > > >
> > > > > org.apache.kafka.common.errors.RecordTooLargeException: The message
> > is
> > > > > 16777327 bytes when serialized which is larger than the maximum
> > request
> > > > > size you have configured with the max.request.size configuration.
> > > > > I found couple of links describing the solution:
> > > > > *
> > > > >
> > > >
> > >
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > > > > <
> > > > >
> > > >
> > >
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > > > > >*
> > > > >
> > > > > in my server.properties on brokers I set:
> > > > > socket.request.max.bytes=104857600
> > > > > message.max.bytes=18874368
> > > > > max.request.size=18874368
> > > > > replica.fetch.max.bytes=18874368
> > > > > fetch.message.max.bytes=18874368
> > > > >
> > > > > Then in my producer.properties i tried to set
> > > > > max.request.size=18874368
> > > > >
> > > > > But no matter how i large i try to set max.request.size -
> > > > > i still have the same problem...Are there other settings i am
> > missing?
> > > > > Can it be solved in configuration alone, or do i need to make code
> > > > changes?
> > > > > Thank you,
> > > > >
> > > >
> > > >
> > > > --
> > > > Santilli Jonathan
> > > >
> > >
> >
> >
> > --
> > Santilli Jonathan
> >
>
Re: RecordTooLargeException on 16M messages in Kafka?
Posted by l vic <lv...@gmail.com>.
yes, in producer.properties
On Thu, Aug 15, 2019 at 9:59 AM Jonathan Santilli <
jonathansantilli@gmail.com> wrote:
> Just to be sure, please confirm the configuration parameter is well
> set/configure at producer level:
>
> max.request.size = 177772390 (for instance)
>
> Cheers!
> --
> Jonathan
>
>
>
>
> On Thu, Aug 15, 2019 at 1:44 PM l vic <lv...@gmail.com> wrote:
>
> > I tested it with kafka-console-consumer and kafka-console-producer
> reading
> > from large text file (no newlines):
> >
> > kafka-console-producer.sh --broker-list 10.10.105.24:6667 --topic
> test <
> > ./large-file
> >
> >
> >
> >
> > On Thu, Aug 15, 2019 at 4:49 AM Jonathan Santilli <
> > jonathansantilli@gmail.com> wrote:
> >
> > > Hello, try to send and flush just one message of 16777239 bytes, to
> > verify
> > > the error still shows up.
> > >
> > > Cheers!
> > > --
> > > Jonathan
> > >
> > >
> > >
> > > On Thu, Aug 15, 2019 at 2:23 AM l vic <lv...@gmail.com> wrote:
> > >
> > > > My kafka (1.0.0) producer errors out on large (16M) messages.
> > > > ERROR Error when sending message to topic test with key: null, value:
> > > > 16777239 bytes with error:
> > (org.apache.kafka.clients.producer.internals.
> > > > ErrorLoggingCallback)
> > > >
> > > > org.apache.kafka.common.errors.RecordTooLargeException: The message
> is
> > > > 16777327 bytes when serialized which is larger than the maximum
> request
> > > > size you have configured with the max.request.size configuration.
> > > > I found couple of links describing the solution:
> > > > *
> > > >
> > >
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > > > <
> > > >
> > >
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > > > >*
> > > >
> > > > in my server.properties on brokers I set:
> > > > socket.request.max.bytes=104857600
> > > > message.max.bytes=18874368
> > > > max.request.size=18874368
> > > > replica.fetch.max.bytes=18874368
> > > > fetch.message.max.bytes=18874368
> > > >
> > > > Then in my producer.properties i tried to set
> > > > max.request.size=18874368
> > > >
> > > > But no matter how i large i try to set max.request.size -
> > > > i still have the same problem...Are there other settings i am
> missing?
> > > > Can it be solved in configuration alone, or do i need to make code
> > > changes?
> > > > Thank you,
> > > >
> > >
> > >
> > > --
> > > Santilli Jonathan
> > >
> >
>
>
> --
> Santilli Jonathan
>
Re: RecordTooLargeException on 16M messages in Kafka?
Posted by Jonathan Santilli <jo...@gmail.com>.
Just to be sure, please confirm the configuration parameter is well
set/configure at producer level:
max.request.size = 177772390 (for instance)
Cheers!
--
Jonathan
On Thu, Aug 15, 2019 at 1:44 PM l vic <lv...@gmail.com> wrote:
> I tested it with kafka-console-consumer and kafka-console-producer reading
> from large text file (no newlines):
>
> kafka-console-producer.sh --broker-list 10.10.105.24:6667 --topic test <
> ./large-file
>
>
>
>
> On Thu, Aug 15, 2019 at 4:49 AM Jonathan Santilli <
> jonathansantilli@gmail.com> wrote:
>
> > Hello, try to send and flush just one message of 16777239 bytes, to
> verify
> > the error still shows up.
> >
> > Cheers!
> > --
> > Jonathan
> >
> >
> >
> > On Thu, Aug 15, 2019 at 2:23 AM l vic <lv...@gmail.com> wrote:
> >
> > > My kafka (1.0.0) producer errors out on large (16M) messages.
> > > ERROR Error when sending message to topic test with key: null, value:
> > > 16777239 bytes with error:
> (org.apache.kafka.clients.producer.internals.
> > > ErrorLoggingCallback)
> > >
> > > org.apache.kafka.common.errors.RecordTooLargeException: The message is
> > > 16777327 bytes when serialized which is larger than the maximum request
> > > size you have configured with the max.request.size configuration.
> > > I found couple of links describing the solution:
> > > *
> > >
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > > <
> > >
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > > >*
> > >
> > > in my server.properties on brokers I set:
> > > socket.request.max.bytes=104857600
> > > message.max.bytes=18874368
> > > max.request.size=18874368
> > > replica.fetch.max.bytes=18874368
> > > fetch.message.max.bytes=18874368
> > >
> > > Then in my producer.properties i tried to set
> > > max.request.size=18874368
> > >
> > > But no matter how i large i try to set max.request.size -
> > > i still have the same problem...Are there other settings i am missing?
> > > Can it be solved in configuration alone, or do i need to make code
> > changes?
> > > Thank you,
> > >
> >
> >
> > --
> > Santilli Jonathan
> >
>
--
Santilli Jonathan
Re: RecordTooLargeException on 16M messages in Kafka?
Posted by l vic <lv...@gmail.com>.
I tested it with kafka-console-consumer and kafka-console-producer reading
from large text file (no newlines):
kafka-console-producer.sh --broker-list 10.10.105.24:6667 --topic test <
./large-file
On Thu, Aug 15, 2019 at 4:49 AM Jonathan Santilli <
jonathansantilli@gmail.com> wrote:
> Hello, try to send and flush just one message of 16777239 bytes, to verify
> the error still shows up.
>
> Cheers!
> --
> Jonathan
>
>
>
> On Thu, Aug 15, 2019 at 2:23 AM l vic <lv...@gmail.com> wrote:
>
> > My kafka (1.0.0) producer errors out on large (16M) messages.
> > ERROR Error when sending message to topic test with key: null, value:
> > 16777239 bytes with error: (org.apache.kafka.clients.producer.internals.
> > ErrorLoggingCallback)
> >
> > org.apache.kafka.common.errors.RecordTooLargeException: The message is
> > 16777327 bytes when serialized which is larger than the maximum request
> > size you have configured with the max.request.size configuration.
> > I found couple of links describing the solution:
> > *
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > <
> >
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> > >*
> >
> > in my server.properties on brokers I set:
> > socket.request.max.bytes=104857600
> > message.max.bytes=18874368
> > max.request.size=18874368
> > replica.fetch.max.bytes=18874368
> > fetch.message.max.bytes=18874368
> >
> > Then in my producer.properties i tried to set
> > max.request.size=18874368
> >
> > But no matter how i large i try to set max.request.size -
> > i still have the same problem...Are there other settings i am missing?
> > Can it be solved in configuration alone, or do i need to make code
> changes?
> > Thank you,
> >
>
>
> --
> Santilli Jonathan
>
Re: RecordTooLargeException on 16M messages in Kafka?
Posted by Jonathan Santilli <jo...@gmail.com>.
Hello, try to send and flush just one message of 16777239 bytes, to verify
the error still shows up.
Cheers!
--
Jonathan
On Thu, Aug 15, 2019 at 2:23 AM l vic <lv...@gmail.com> wrote:
> My kafka (1.0.0) producer errors out on large (16M) messages.
> ERROR Error when sending message to topic test with key: null, value:
> 16777239 bytes with error: (org.apache.kafka.clients.producer.internals.
> ErrorLoggingCallback)
>
> org.apache.kafka.common.errors.RecordTooLargeException: The message is
> 16777327 bytes when serialized which is larger than the maximum request
> size you have configured with the max.request.size configuration.
> I found couple of links describing the solution:
> *
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> <
> https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb
> >*
>
> in my server.properties on brokers I set:
> socket.request.max.bytes=104857600
> message.max.bytes=18874368
> max.request.size=18874368
> replica.fetch.max.bytes=18874368
> fetch.message.max.bytes=18874368
>
> Then in my producer.properties i tried to set
> max.request.size=18874368
>
> But no matter how i large i try to set max.request.size -
> i still have the same problem...Are there other settings i am missing?
> Can it be solved in configuration alone, or do i need to make code changes?
> Thank you,
>
--
Santilli Jonathan