You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Arthur Li <li...@126.com> on 2023/03/26 14:27:47 UTC

How to calculate the spark.kryoserializer.buffer.max?

Hello all,

The data is generated by the vendors, while some days, the data size will be very huge, and it will overflow the default  value of spark.kryoserializer.buffer.max,  
So how to calculate the spark.kryoserializer.buffer.max when the data size is changed ahead of raising the exception during the runtime?

Appreciate your any suggestions. 

BR.
Arthur Li

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org