[GitHub] [spark] liuzqt commented on a diff in pull request #38064: [SPARK-40622][SQL][CORE]Result of a single task in collect() must fit in 2GB

[GitBox <gi...@apache.org>]

liuzqt commented on code in PR #38064:
URL: https://github.com/apache/spark/pull/38064#discussion_r1016232819


##########
core/src/test/scala/org/apache/spark/serializer/KryoSerializerResizableOutputSuite.scala:
##########
@@ -34,18 +32,20 @@ class KryoSerializerResizableOutputSuite extends SparkFunSuite {
     conf.set(SERIALIZER, "org.apache.spark.serializer.KryoSerializer")
     conf.set(KRYO_SERIALIZER_BUFFER_SIZE.key, "1m")
     conf.set(KRYO_SERIALIZER_MAX_BUFFER_SIZE.key, "1m")
-    withSpark(new SparkContext("local", "test", conf)) { sc =>

Review Comment:
   After code change, this test(`sc.parallelize(x).collect()`) doesn't go through `Serializer.serialize` code path anymore, it goes through `Serializer.serializeStream` instead. So I directly construct a serializer to test against that logic.
   
   The reason why `Serializer.serializeStream` is not affected by buffer size is that the way `KryoSerializeStream` constructs `KryoOutput` is different from how `KyroSerializeInstance` does it:
   - https://github.com/apache/spark/blob/2298cebcf83f7cad21fa291a718505032830474a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala#L264
   - https://github.com/apache/spark/blob/2298cebcf83f7cad21fa291a718505032830474a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala#L380
   
   I'm not sure this is by intention or not, but to minimize the change I leave it as it's. The test suite should also preserve the original testing semantics.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org