[spark] branch master updated: [SPARK-45447][MLLIB][TESTS] Reduce the memory required to start the `LocalClusterSparkContext` in the `mllib` module test cases

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 2bca4426e6dd [SPARK-45447][MLLIB][TESTS] Reduce the memory required to start the `LocalClusterSparkContext` in the `mllib` module test cases
2bca4426e6dd is described below

commit 2bca4426e6dd830161ca518e22ca3ffb955fa24c
Author: YangJie <ya...@baidu.com>
AuthorDate: Sat Oct 7 15:37:37 2023 -0700

    [SPARK-45447][MLLIB][TESTS] Reduce the memory required to start the `LocalClusterSparkContext` in the `mllib` module test cases
    
    ### What changes were proposed in this pull request?
    This pr aims to reduce the chance of GA test tasks being killed by decreasing the memory required to start `LocalClusterSparkContext`.
    
    ### Why are the changes needed?
    The GA test task for mllib is sometimes killed, and the test cases to be tested that often get killed are those that inherit from `LocalClusterSparkContext`, such as
    
    - LBFGSClusterSuite: https://github.com/apache/spark/actions/runs/6421930285/job/17437383223
    - NaiveBayesClusterSuite: https://github.com/apache/spark/actions/runs/6436807001/job/17480899578
    - LassoClusterSuite: https://github.com/apache/spark/actions/runs/6362618008/job/17289887813
    - LBFGSClusterSuite: https://github.com/apache/spark/actions/runs/6346972272/job/17289897396
    
    So this pr circumvents this issue by reducing the startup memory required for `LocalClusterSparkContext`.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    Should monitor GA
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No
    
    Closes #43242 from LuciferYang/mllib-mem.
    
    Lead-authored-by: YangJie <ya...@baidu.com>
    Co-authored-by: yangjie01 <ya...@baidu.com>
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 .../scala/org/apache/spark/mllib/util/LocalClusterSparkContext.scala  | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/mllib/src/test/scala/org/apache/spark/mllib/util/LocalClusterSparkContext.scala b/mllib/src/test/scala/org/apache/spark/mllib/util/LocalClusterSparkContext.scala
index 79d4785fd6fa..892d2f627425 100644
--- a/mllib/src/test/scala/org/apache/spark/mllib/util/LocalClusterSparkContext.scala
+++ b/mllib/src/test/scala/org/apache/spark/mllib/util/LocalClusterSparkContext.scala
@@ -20,6 +20,7 @@ package org.apache.spark.mllib.util
 import org.scalatest.{BeforeAndAfterAll, Suite}
 
 import org.apache.spark.{SparkConf, SparkContext}
+import org.apache.spark.internal.config.EXECUTOR_MEMORY
 import org.apache.spark.internal.config.Network.RPC_MESSAGE_MAX_SIZE
 
 trait LocalClusterSparkContext extends BeforeAndAfterAll { self: Suite =>
@@ -28,8 +29,9 @@ trait LocalClusterSparkContext extends BeforeAndAfterAll { self: Suite =>
   override def beforeAll(): Unit = {
     super.beforeAll()
     val conf = new SparkConf()
-      .setMaster("local-cluster[2, 1, 1024]")
+      .setMaster("local-cluster[2, 1, 512]")
       .setAppName("test-cluster")
+      .set(EXECUTOR_MEMORY.key, "512m")
       .set(RPC_MESSAGE_MAX_SIZE, 1) // set to 1MB to detect direct serialization of data
     sc = new SparkContext(conf)
   }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org