[GitHub] [spark] liuzqt commented on a diff in pull request #37713: [SPARK-40261][CORE]Exclude DirectTaskResult metadata when calculating result size

[GitBox <gi...@apache.org>]

liuzqt commented on code in PR #37713:
URL: https://github.com/apache/spark/pull/37713#discussion_r958809543


##########
core/src/test/scala/org/apache/spark/scheduler/TaskResultGetterSuite.scala:
##########
@@ -297,6 +298,18 @@ class TaskResultGetterSuite extends SparkFunSuite with BeforeAndAfter with Local
     assert(unknownFailure.findFirstMatchIn(message).isDefined)
   }
 
+  test("SPARK-40261 task result metadata should not be counted into result size") {
+    val conf = new SparkConf().set(MAX_RESULT_SIZE.key, "1M")
+    sc = new SparkContext("local", "test", conf)
+    val rdd = sc.parallelize(1 to 10000, 10000)
+    // This will trigger 10k task but return empty result. The total serialized return tasks
+    // size(including accumUpdates metadata) would be ~10M in total in this example, but the result
+    // value itself is pretty small(empty arrays)
+    // Even setting MAX_RESULT_SIZE to a small value(1M here), it should not throw exception
+    // because the actual result is small
+    rdd.filter(_ < 0).collect()

Review Comment:
   Before the fix, it will throw exception defined [here](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala#L780) .  This rdd operation is just to contrive a example reproducing that exception (trigger large amounts of data but collecting empty result)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org