(spark) 01/02: [SPARK-57710][YARN][TEST] Reduce YarnClusterSuite flakiness from CI runner contention

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git

commit fec32fcd13b1009e26b12aa850bc79fa4defc888
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Sat Jun 27 11:09:09 2026 +0900

    [SPARK-57710][YARN][TEST] Reduce YarnClusterSuite flakiness from CI runner contention
    
    ### What changes were proposed in this pull request?
    
    Follow-up to [SPARK-57650](https://issues.apache.org/jira/browse/SPARK-57650), which fixed the deterministic "AM stuck in ACCEPTED" hang in `BaseYarnClusterSuite`. Two further **test-only** changes to reduce the remaining flakiness of `YarnClusterSuite`:
    
    1. Give the single mini `NodeManager` 8GB (`yarn.nodemanager.resource.memory-mb` + `yarn.scheduler.maximum-allocation-mb`) so executor allocation is never starved once the ~1.4GB AM is running.
    2. Raise the executor→driver connection retry budget for the launched apps (`spark.rpc.io.maxRetries=10`, `spark.rpc.io.retryWait=2s`) so a transient RPC-accept stall does not permanently fail an executor. These are defaults that individual tests can still override via `extraConf`.
    
    ### Why are the changes needed?
    
    Even after SPARK-57650, the scheduled `Build / Java21` and `Build / Java25` master lanes fail in the `yarn` module roughly **50% of runs** (e.g. fork run `28151220075` PASS vs `28151247521` FAIL — same commit, 40s apart). All failures are the same six `YarnClusterSuite` tests timing out after 3 minutes (`The code passed to eventually never returned normally ... handle.getState().isFinal() was false`).
    
    From the `yarn-app-log` / `unit-tests-log` artifacts, the AM/driver comes up, but the executor (and sometimes the AM) intermittently fail to connect back to the driver's RPC server on `localhost` (`java.io.IOException: Failed to connect to localhost/127.0.0.1:<port>`, connection refused). The in-JVM mini RM+NM, the driver subprocess and the AM/executor JVMs all contend for CPU on a single CI runner, so the driver's accept loop occasionally stalls; an executor that loses this race exit [...]
    
    ### Does this PR introduce any user-facing change?
    
    No. Test-only.
    
    ### How was this patch tested?
    
    `YarnClusterSuite` was previously failing ~50% of the time. With this change the `yarn` module job was run **6 times** on the fork; all 6 passed, with `YarnClusterSuite` reporting `tests=30, failures=0, skipped=0` (the 6 formerly-failing tests now pass):
    
    - Before (master, failing): apache/spark run `28148781009` (`Build / Java21`) — 6 `YarnClusterSuite` timeouts.
    - After (this branch): HyukjinKwon/spark runs `28162182834`, `28162247111`, `28162249819`, `28175759262`, `28175762257`, `28175765871` — `yarn` job green in all six.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Yes, Generated-by: Claude Code.
    
    Closes #56785 from HyukjinKwon/ci-fix/agent8-yarn-cluster-flaky.
    
    Authored-by: Hyukjin Kwon <gu...@apache.org>
    Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
    (cherry picked from commit 41f3e67e3ebe9ca654af98c6464d607c92ae97f3)
---
 .../spark/deploy/yarn/BaseYarnClusterSuite.scala     | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
index 69c515d6a381..5112e62d838c 100644
--- a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
+++ b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
@@ -111,6 +111,14 @@ abstract class BaseYarnClusterSuite extends SparkFunSuite with Matchers {
     yarnConf.setFloat("yarn.scheduler.capacity.maximum-am-resource-percent", 1.0f)
     yarnConf.setFloat("yarn.scheduler.capacity.root.default.maximum-am-resource-percent", 1.0f)
 
+    // Give the single mini NodeManager generous memory. By default the mini cluster advertises
+    // only a small amount of memory, so once the AM (~1.4GB) is running there is barely enough
+    // headroom left for the executors these tests request. On a busy CI runner that makes
+    // executor allocation slow/racy and the YarnClusterSuite apps time out waiting to finish.
+    // The CI hosts have plenty of RAM, so let the NM offer enough for the AM plus a few executors.
+    yarnConf.setInt("yarn.nodemanager.resource.memory-mb", 8192)
+    yarnConf.setInt("yarn.scheduler.maximum-allocation-mb", 8192)
+
     // Support both IPv4 and IPv6
     yarnConf.set("yarn.resourcemanager.hostname", Utils.localHostNameForURI())
 
@@ -292,6 +300,18 @@ abstract class BaseYarnClusterSuite extends SparkFunSuite with Matchers {
         props.setProperty(k, v)
       }
     }
+
+    // On a busy CI runner the in-JVM mini cluster, the driver and the container JVMs all compete
+    // for CPU, and the driver's RPC server occasionally cannot accept a connection in time. With
+    // the default of 3 retries an executor that loses this race gives up and exits, which leaves
+    // the application unable to finish and the suite times out. Give the executor->driver
+    // connection a larger retry budget so a transient stall does not permanently fail the app.
+    // Set after the spark.* JVM properties are copied above so these values are not silently
+    // overridden by an inherited -Dspark.rpc.io.* flag; individual tests can still override them
+    // via extraConf below.
+    props.setProperty("spark.rpc.io.maxRetries", "10")
+    props.setProperty("spark.rpc.io.retryWait", "2s")
+
     extraConf.foreach { case (k, v) => props.setProperty(k, v) }
 
     val propsFile = File.createTempFile("spark", ".properties", tempDir)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org