You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by sarath reddy <pu...@gmail.com> on 2018/09/20 10:50:54 UTC
Kafka_2.12-1.1.0 config
Hi Team,
We are trying to configure Kafka to produce larger messages,
Below are the configs:-
Server.properties
message.max.bytes=100000000
replica.fetch.max.bytes=100000001
Producer.properties
max.request.size=100000000
compression.type=gzip
Consumer.properties
fetch.message.max.bytes=100000000
When trying to execute a larger file with below command
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic
metadata_upload < /export/home/Userid/metadata/TopUBP-1_metadata.json
Getting below error:-
>[2018-09-19 11:20:03,307] ERROR Error when sending message to topic
metadata_upload with key: null, value: 2170060 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.RecordTooLargeException: The message is
2170148 bytes when serialized which is larger than the maximum request size
you have configured with the max.request.size configuration.
Thanks,
Sarath Reddy.
Re: Kafka_2.12-1.1.0 config
Posted by hacker win7 <ha...@gmail.com>.
bin/kafka-console-producer.sh add --producer.config
your_producer.properties file.
you don't add --producer.config , you still use the default value of
max.request.size.
sarath reddy <pu...@gmail.com> 于2018年9月21日周五 上午12:27写道：
> ++pushkar
>
> On Thu 20 Sep, 2018, 16:20 sarath reddy, <pu...@gmail.com> wrote:
>
> > Hi Team,
> >
> > We are trying to configure Kafka to produce larger messages,
> >
> > Below are the configs:-
> >
> > Server.properties
> >
> > message.max.bytes=100000000
> > replica.fetch.max.bytes=100000001
> >
> > Producer.properties
> >
> > max.request.size=100000000
> > compression.type=gzip
> >
> > Consumer.properties
> >
> > fetch.message.max.bytes=100000000
> >
> > When trying to execute a larger file with below command
> > bin/kafka-console-producer.sh --broker-list localhost:9092 --topic
> > metadata_upload < /export/home/Userid/metadata/TopUBP-1_metadata.json
> >
> > Getting below error:-
> >
> > >[2018-09-19 11:20:03,307] ERROR Error when sending message to topic
> > metadata_upload with key: null, value: 2170060 bytes with error:
> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> > org.apache.kafka.common.errors.RecordTooLargeException: The message is
> > 2170148 bytes when serialized which is larger than the maximum request
> size
> > you have configured with the max.request.size configuration.
> >
> > Thanks,
> > Sarath Reddy.
> >
> >
> >
>
Re: Kafka_2.12-1.1.0 config
Posted by Subash Konar <su...@gmail.com>.
You can refer this closed issue in github
<https://github.com/wurstmeister/kafka-docker/issues/100>.
On Thu, Sep 20, 2018 at 9:57 PM sarath reddy <pu...@gmail.com> wrote:
> ++pushkar
>
> On Thu 20 Sep, 2018, 16:20 sarath reddy, <pu...@gmail.com> wrote:
>
> > Hi Team,
> >
> > We are trying to configure Kafka to produce larger messages,
> >
> > Below are the configs:-
> >
> > Server.properties
> >
> > message.max.bytes=100000000
> > replica.fetch.max.bytes=100000001
> >
> > Producer.properties
> >
> > max.request.size=100000000
> > compression.type=gzip
> >
> > Consumer.properties
> >
> > fetch.message.max.bytes=100000000
> >
> > When trying to execute a larger file with below command
> > bin/kafka-console-producer.sh --broker-list localhost:9092 --topic
> > metadata_upload < /export/home/Userid/metadata/TopUBP-1_metadata.json
> >
> > Getting below error:-
> >
> > >[2018-09-19 11:20:03,307] ERROR Error when sending message to topic
> > metadata_upload with key: null, value: 2170060 bytes with error:
> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> > org.apache.kafka.common.errors.RecordTooLargeException: The message is
> > 2170148 bytes when serialized which is larger than the maximum request
> size
> > you have configured with the max.request.size configuration.
> >
> > Thanks,
> > Sarath Reddy.
> >
> >
> >
>
--
Thanks and Regards,
Subash Konar
Re: Kafka_2.12-1.1.0 config
Posted by sarath reddy <pu...@gmail.com>.
++pushkar
On Thu 20 Sep, 2018, 16:20 sarath reddy, <pu...@gmail.com> wrote:
> Hi Team,
>
> We are trying to configure Kafka to produce larger messages,
>
> Below are the configs:-
>
> Server.properties
>
> message.max.bytes=100000000
> replica.fetch.max.bytes=100000001
>
> Producer.properties
>
> max.request.size=100000000
> compression.type=gzip
>
> Consumer.properties
>
> fetch.message.max.bytes=100000000
>
> When trying to execute a larger file with below command
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic
> metadata_upload < /export/home/Userid/metadata/TopUBP-1_metadata.json
>
> Getting below error:-
>
> >[2018-09-19 11:20:03,307] ERROR Error when sending message to topic
> metadata_upload with key: null, value: 2170060 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.RecordTooLargeException: The message is
> 2170148 bytes when serialized which is larger than the maximum request size
> you have configured with the max.request.size configuration.
>
> Thanks,
> Sarath Reddy.
>
>
>