[GitHub] [hudi] codope commented on a diff in pull request #5197: [HUDI-3768] Array Index Overflow Error Is Reported When Spark-Beeline…

[GitBox <gi...@apache.org>]

codope commented on code in PR #5197:
URL: https://github.com/apache/hudi/pull/5197#discussion_r853695986


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/SerDeUtils.scala:
##########
@@ -30,7 +30,10 @@ object SerDeUtils {
   private val kryoLocal = new ThreadLocal[Kryo] {
 
     override protected def initialValue: Kryo = {
-      val serializer = new KryoSerializer(new SparkConf(true))
+      val conf = new SparkConf(true)
+      conf.set("spark.kryo.referenceTracking", "true")

Review Comment:
   Couple of concerns about setting this true for all:
   1. Won't tracking references affect performance?
   2. What about thrift server? In that case, default value is false. Setting to true will affect default behaviour.
   
   Would it be better to guard this by a config, or let users pass it as spark config in the launch command? Users can enable/disable based on their deployment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org