You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Something Something <ma...@gmail.com> on 2020/05/25 21:42:34 UTC

RecordTooLargeException in Spark *Structured* Streaming

I keep getting this error message:


*The message is 1169350 bytes when serialized which is larger than the
maximum request size you have configured with the max.request.size
configuration.*



As indicated in other posts, I am trying to set the “max.request.size”
configuration in the Producer as follows:


---------------------

.writeStream

.format(*"kafka"*)

.option(

  *"kafka.bootstrap.servers"*,

  conig.outputBootstrapServer

)

.option(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, *"10000000"*)

---------------------



But this is not working. Am I setting this correctly? Is there a different
way to set this property under Spark Structured Streaming?


Please help. Thanks.

Re: RecordTooLargeException in Spark *Structured* Streaming

Posted by Something Something <ma...@gmail.com>.
Thanks. Missed that part of documentation. Appreciate your help. Regards.

On Mon, May 25, 2020 at 10:42 PM Jungtaek Lim <ka...@gmail.com>
wrote:

> Hi,
>
> You need to add the prefix "kafka." for the configurations which should be
> propagated to the Kafka. Others will be used in Spark data source
> itself. (Kafka connector in this case)
>
>
> https://spark.apache.org/docs/2.4.5/structured-streaming-kafka-integration.html#kafka-specific-configurations
>
> Hope this helps.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
>
> On Tue, May 26, 2020 at 6:42 AM Something Something <
> mailinglists19@gmail.com> wrote:
>
>> I keep getting this error message:
>>
>>
>> *The message is 1169350 bytes when serialized which is larger than the
>> maximum request size you have configured with the max.request.size
>> configuration.*
>>
>>
>>
>> As indicated in other posts, I am trying to set the “max.request.size”
>> configuration in the Producer as follows:
>>
>>
>> ---------------------
>>
>> .writeStream
>>
>> .format(*"kafka"*)
>>
>> .option(
>>
>>   *"kafka.bootstrap.servers"*,
>>
>>   conig.outputBootstrapServer
>>
>> )
>>
>> .option(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, *"10000000"*)
>>
>> ---------------------
>>
>>
>>
>> But this is not working. Am I setting this correctly? Is there a
>> different way to set this property under Spark Structured Streaming?
>>
>>
>> Please help. Thanks.
>>
>>
>>

Re: RecordTooLargeException in Spark *Structured* Streaming

Posted by Jungtaek Lim <ka...@gmail.com>.
Hi,

You need to add the prefix "kafka." for the configurations which should be
propagated to the Kafka. Others will be used in Spark data source
itself. (Kafka connector in this case)

https://spark.apache.org/docs/2.4.5/structured-streaming-kafka-integration.html#kafka-specific-configurations

Hope this helps.

Thanks,
Jungtaek Lim (HeartSaVioR)


On Tue, May 26, 2020 at 6:42 AM Something Something <
mailinglists19@gmail.com> wrote:

> I keep getting this error message:
>
>
> *The message is 1169350 bytes when serialized which is larger than the
> maximum request size you have configured with the max.request.size
> configuration.*
>
>
>
> As indicated in other posts, I am trying to set the “max.request.size”
> configuration in the Producer as follows:
>
>
> ---------------------
>
> .writeStream
>
> .format(*"kafka"*)
>
> .option(
>
>   *"kafka.bootstrap.servers"*,
>
>   conig.outputBootstrapServer
>
> )
>
> .option(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, *"10000000"*)
>
> ---------------------
>
>
>
> But this is not working. Am I setting this correctly? Is there a different
> way to set this property under Spark Structured Streaming?
>
>
> Please help. Thanks.
>
>
>