[GitHub] [spark] martin-g commented on a diff in pull request #38789: [SPARK-41253][K8S][TESTS] Make Spark K8S volcano IT work in Github Action

[GitBox <gi...@apache.org>]

martin-g commented on code in PR #38789:
URL: https://github.com/apache/spark/pull/38789#discussion_r1036763619


##########
resource-managers/kubernetes/integration-tests/README.md:
##########
@@ -283,6 +283,14 @@ to the wrapper scripts and using the wrapper scripts will simply set these appro
     </td>
     <td></td>
   </tr>
+  <tr>
+    <td><code>spark.kubernetes.test.volcano.maxConcurrencyJobNum</code></td>
+    <td>
+      Set maximum number for concurrency job, it helps developers setting suitable resources according to test env in

Review Comment:
   ```suggestion
         Set maximum number for concurrency jobs. It helps developers setting suitable resources according to test env in
   ```



##########
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/VolcanoTestsSuite.scala:
##########
@@ -435,10 +464,11 @@ private[spark] object VolcanoTestsSuite extends SparkFunSuite {
   val GROUP_PREFIX = "volcano-test" + UUID.randomUUID().toString.replaceAll("-", "") + "-"
   val VOLCANO_PRIORITY_YAML
     = new File(getClass.getResource("/volcano/priorityClasses.yml").getFile).getAbsolutePath
-  val DRIVER_PG_TEMPLATE_CPU_2U = new File(
-    getClass.getResource("/volcano/driver-podgroup-template-cpu-2u.yml").getFile
-  ).getAbsolutePath
   val DRIVER_PG_TEMPLATE_MEMORY_3G = new File(
     getClass.getResource("/volcano/driver-podgroup-template-memory-3g.yml").getFile
   ).getAbsolutePath
+  val DRIVER_REQUEST_CORES = sys.props.get(CONFIG_DRIVER_REQUEST_CORES).getOrElse("1")
+  val EXECUTOR_REQUEST_CORES = sys.props.get(CONFIG_EXECUTOR_REQUEST_CORES).getOrElse("1")
+  val VOLCANO_MAX_JOB_NUM = sys.props.get(CONFIG_KEY_VOLCANO_MAX_JOB_NUM).getOrElse("2")
+  val HOST_PATH = "/tmp/"

Review Comment:
   Better use `System.getProperty("java.io.tmpdir")` 



##########
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/VolcanoTestsSuite.scala:
##########
@@ -303,16 +312,30 @@ private[spark] trait VolcanoTestsSuite extends BeforeAndAfterEach { k8sSuite: Ku
 
   test("SPARK-38187: Run SparkPi Jobs with minCPU", k8sTestTag, volcanoTag) {
     val groupName = generateGroupName("min-cpu")
-    // Create a queue with 2 CPU capacity
-    createOrReplaceQueue(name = "queue-2u", cpu = Some("2"))
+    // Create a queue with drvier + executor CPU capacity

Review Comment:
   ```suggestion
       // Create a queue with driver + executor CPU capacity
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org