[GitHub] [spark] Ngone51 commented on a change in pull request #28917: [SPARK-31847][CORE][TESTS] DAGSchedulerSuite: Rewrite the test framework to support apply specified spark configurations.

[GitBox <gi...@apache.org>]

Ngone51 commented on a change in pull request #28917:
URL: https://github.com/apache/spark/pull/28917#discussion_r446730477



##########
File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -278,7 +280,26 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with TimeLi
 
   override def beforeEach(): Unit = {
     super.beforeEach()
-    init(new SparkConf())
+  }
+
+  override protected def test(testName: String, testTags: Tag*)(testFun: => Any)
+      (implicit pos: Position): Unit = {
+    testWithSparkConf(testName, testTags: _*)()(testFun)(pos)
+  }
+
+  private def testWithSparkConf(testName: String, testTags: Tag*)

Review comment:
       Shall we simulate the usage of `withSQLConf` instead of integrating the confs with `test()`?

##########
File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -278,7 +280,26 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with TimeLi
 
   override def beforeEach(): Unit = {
     super.beforeEach()
-    init(new SparkConf())
+  }
+
+  override protected def test(testName: String, testTags: Tag*)(testFun: => Any)
+      (implicit pos: Position): Unit = {
+    testWithSparkConf(testName, testTags: _*)()(testFun)(pos)
+  }
+
+  private def testWithSparkConf(testName: String, testTags: Tag*)
+      (pairs: (String, String)*)(testFun: => Any)(implicit pos: Position): Unit = {
+    super.test(testName, testTags: _*) {
+      withSparkConf(pairs: _*)(testFun)
+    }
+  }
+
+  /** Sets all configurations specified in `pairs`, calls `init`, and then calls `testFun` */
+  private def withSparkConf(pairs: (String, String)*)(testFun: => Any): Unit = {
+    val conf = new SparkConf()
+    pairs.foreach(kv => conf.set(kv._1, kv._2))
+    init(conf)

Review comment:
       `init()` is specific to `DAGSchedulerSuite`, we should separate it from the test framework.

##########
File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -278,7 +280,26 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with TimeLi
 
   override def beforeEach(): Unit = {
     super.beforeEach()
-    init(new SparkConf())
+  }
+
+  override protected def test(testName: String, testTags: Tag*)(testFun: => Any)
+      (implicit pos: Position): Unit = {
+    testWithSparkConf(testName, testTags: _*)()(testFun)(pos)
+  }
+
+  private def testWithSparkConf(testName: String, testTags: Tag*)
+      (pairs: (String, String)*)(testFun: => Any)(implicit pos: Position): Unit = {
+    super.test(testName, testTags: _*) {
+      withSparkConf(pairs: _*)(testFun)
+    }
+  }
+
+  /** Sets all configurations specified in `pairs`, calls `init`, and then calls `testFun` */
+  private def withSparkConf(pairs: (String, String)*)(testFun: => Any): Unit = {

Review comment:
       For `test()`, `withSparkConf()`, shall we extract them into a base class? I guess they could be used by other test suites as well? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org