You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2026/07/06 03:28:26 UTC
(spark) branch branch-4.0 updated (e5fdf777b290 -> d97aed432e24)
This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a change to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
from e5fdf777b290 [SPARK-57882][SQL] Enforce AuthV2 metadata authorization on GetPrimaryKeys and GetCrossReference operations
new 2eba1fb1d906 [SPARK-57650][YARN][TEST] Allow AMs to use the whole queue in BaseYarnClusterSuite to fix ACCEPTED-state timeouts
new 9f797600b16a [SPARK-57710][YARN][TEST] Reduce YarnClusterSuite flakiness from CI runner contention
new d97aed432e24 [SPARK-57710][YARN][TEST][FOLLOWUP] Size YarnClusterSuite mini NodeManager via the yarn.minicluster.* key
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../spark/deploy/yarn/BaseYarnClusterSuite.scala | 34 ++++++++++++++++++++++
1 file changed, 34 insertions(+)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
(spark) 03/03: [SPARK-57710][YARN][TEST][FOLLOWUP] Size YarnClusterSuite mini NodeManager via the yarn.minicluster.* key
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit d97aed432e243f0f132bdca553280819399cd734
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Mon Jul 6 12:14:49 2026 +0900
[SPARK-57710][YARN][TEST][FOLLOWUP] Size YarnClusterSuite mini NodeManager via the yarn.minicluster.* key
### What changes were proposed in this pull request?
Follow-up to [SPARK-57710](https://issues.apache.org/jira/browse/SPARK-57710) / [SPARK-57650](https://issues.apache.org/jira/browse/SPARK-57650). Those changes tried to stop the recurring `YarnClusterSuite` timeouts by giving the test mini `NodeManager` more memory:
```scala
yarnConf.setInt("yarn.nodemanager.resource.memory-mb", 8192)
yarnConf.setInt("yarn.scheduler.maximum-allocation-mb", 8192)
```
The `yarn.scheduler.maximum-allocation-mb` part takes effect, but **`yarn.nodemanager.resource.memory-mb` is silently ignored by `MiniYARNCluster`**. Its `NodeManagerWrapper.serviceInit` (hadoop-yarn-server-tests) does:
```java
config.setInt(NM_PMEM_MB, config.getInt(
YarnConfiguration.YARN_MINICLUSTER_NM_PMEM_MB, // "yarn.minicluster.yarn.nodemanager.resource.memory-mb"
YarnConfiguration.DEFAULT_YARN_MINICLUSTER_NM_PMEM_MB)); // 4 * 1024 = 4096
```
i.e. it unconditionally overwrites `yarn.nodemanager.resource.memory-mb` with the value of the **minicluster-prefixed** key, which defaults to 4096. So the mini NM only ever advertised ~4GB, not 8GB.
This PR sets the key that `MiniYARNCluster` actually reads:
```scala
yarnConf.setInt("yarn.minicluster.yarn.nodemanager.resource.memory-mb", 8192)
```
(The pre-existing `yarn.nodemanager.resource.memory-mb` line is kept — harmless, and keeps the RM's view consistent.) Test-only change.
### Why are the changes needed?
The scheduled `Build / Java21`, `Build / Java25` and `Build / Maven (JDK 17/21)` master lanes still fail in the `yarn` module ~30-40% of runs, always the same six `YarnClusterSuite` cluster-mode tests timing out after 3 minutes:
```
The code passed to eventually never returned normally. Attempted 190 times over 3.0 minutes.
Last failure message: handle.getState().isFinal() was false. (BaseYarnClusterSuite.scala:228)
```
The `yarn-app-log` artifact of a failing run (`28752150710`) is conclusive:
- `__spark_conf__.properties` contains `yarn.minicluster.yarn.nodemanager.resource.memory-mb=4096`.
- AM container `stderr` reports `Cluster resources: <memory:1024, vCores:6>` (also `<memory:0>` / `<memory:2048>`) — never the intended 8192.
- With only ~4GB, once the ~1.4GB AM plus a prior test's containers are running, the next app's executors (2 × 1408MB) cannot be scheduled; the app never reaches a final state and the suite times out.
Sizing the mini NM via the correct key gives the AM plus a few executors real headroom, removing the starvation race.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
The `yarn` module was run repeatedly on a fork via GitHub Actions. Because the failure is intermittent (the six cluster-mode tests timed out in ~30-40% of unpatched runs), multiple green runs are collected rather than one.
**Failed (before this change — apache/spark `master`, unpatched):**
- `Build / Java25` -> `yarn` module FAILED: https://github.com/apache/spark/actions/runs/28752150710/job/85254025298
- `Build / Maven (JDK 17)` -> `yarn` module FAILED: https://github.com/apache/spark/actions/runs/28742560271/job/85231134021
**Passed (with this change — fork verification):**
- `yarn` module PASSED, sample 1: https://github.com/apache/spark/actions/runs/28757193236 — `YarnClusterSuite` `tests=30, failures=0, errors=0, skipped=0`; full module `tests=243, failures=0`. Ran in ~17 min vs. the ~40 min drag of the timing-out failures.
- `yarn` module PASSED, sample 2: https://github.com/apache/spark/actions/runs/28758958288 — `YarnClusterSuite` again `tests=30, failures=0, errors=0`.
- Repeat validation (independent verification that converged on the same fix): 6 consecutive full `YarnClusterSuite` runs on JDK 17, all green ([run 28757813841](https://github.com/HyukjinKwon/spark/actions/runs/28757813841)) — 6/6 ✅.
Given the original failure reproduced ~30-40% of the time, these consecutive green runs are strong evidence the mini-NodeManager memory starvation race is gone. The six formerly-timing-out cluster-mode tests pass in every run.
Closes #57017 from HyukjinKwon/ci-fix/tmp5-yarn-minicluster-mem.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
(cherry picked from commit f4fc59d906782870c1e8f67cdb5472f9725b2e87)
---
.../scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
index 835b1547befd..1aaa0756df68 100644
--- a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
+++ b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
@@ -116,6 +116,13 @@ abstract class BaseYarnClusterSuite extends SparkFunSuite with Matchers {
// headroom left for the executors these tests request. On a busy CI runner that makes
// executor allocation slow/racy and the YarnClusterSuite apps time out waiting to finish.
// The CI hosts have plenty of RAM, so let the NM offer enough for the AM plus a few executors.
+ //
+ // NOTE: MiniYARNCluster ignores a plain `yarn.nodemanager.resource.memory-mb`. Its
+ // NodeManager.serviceInit unconditionally overwrites that key with
+ // yarn.minicluster.yarn.nodemanager.resource.memory-mb (default 4096)
+ // so the minicluster-prefixed key is the one that actually sizes the mini NM. Set both: the
+ // prefixed key is what takes effect, and the plain key keeps the RM's view consistent.
+ yarnConf.setInt("yarn.minicluster.yarn.nodemanager.resource.memory-mb", 8192)
yarnConf.setInt("yarn.nodemanager.resource.memory-mb", 8192)
yarnConf.setInt("yarn.scheduler.maximum-allocation-mb", 8192)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
(spark) 01/03: [SPARK-57650][YARN][TEST] Allow AMs to use the whole queue in BaseYarnClusterSuite to fix ACCEPTED-state timeouts
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 2eba1fb1d90696356d8fdc30c0947a3eda4fa0e1
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Thu Jun 25 05:09:33 2026 +0900
[SPARK-57650][YARN][TEST] Allow AMs to use the whole queue in BaseYarnClusterSuite to fix ACCEPTED-state timeouts
`BaseYarnClusterSuite` configures a mini `CapacityScheduler` but never sets `yarn.scheduler.capacity.maximum-am-resource-percent`, so it defaults to `0.1`. On memory-constrained CI runners the queue's total AM resource budget becomes ~1GB, which is smaller than the 1–2GB AM/driver memory these tests request. Applications then wedge in the `ACCEPTED` state (never activated) and the suite times out after 3 minutes with `handle.getState().isFinal() was false`.
This sets `maximum-am-resource-percent` to `1.0` (global + `root.default`) so AMs can use the whole test queue and applications are always activated.
`YarnClusterSuite` fails 6 tests with a 3-minute `eventually` timeout on the scheduled Maven builds (`resource-managers#yarn` module):
- run Spark in yarn-client/cluster mode with different configurations, ensuring redaction
- yarn-cluster should respect conf overrides in SparkHadoopUtil (SPARK-16414, SPARK-23630)
- SPARK-35672: additional jar using URI scheme 'local' (client, cluster, client + gateway-replacement)
The YARN diagnostics show `Queue's AM resource limit exceeded. AM Resource Request = <memory:2048>; Queue Resource Limit for AM = <memory:1024>` repeated >1000 times.
**Failing job (before):** https://github.com/apache/spark/actions/runs/28045133937/job/83029837948 — `Build / Maven (branch-4.2, Scala 2.13, JDK 21)`, `resource-managers#yarn` (6 failures).
**Passing job (with this fix):** https://github.com/HyukjinKwon/spark/actions/runs/28066027338/job/83090387029 — `resource-managers/yarn` tests: `YarnClusterSuite` 30/30 pass, the 6 formerly-failing tests now complete in ~11s each (was 180s timeout).
No. Test-only.
Ran the `resource-managers/yarn` module tests on a fork (link above); `YarnClusterSuite` passes 30/30.
This pull request and its description were written by Isaac.
Closes #56715 from HyukjinKwon/ci-fix/yarn-cluster-am-resource-percent.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
(cherry picked from commit 10427c3b2ca704dc01b4a246cb8253986ff2e68f)
Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
(cherry picked from commit 0ed01fd2124640a27d65d3f8e826a24cf05eb680)
---
.../scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
index 3bf6a6e84a88..a22d609370bb 100644
--- a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
+++ b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
@@ -103,6 +103,13 @@ abstract class BaseYarnClusterSuite extends SparkFunSuite with Matchers {
yarnConf.set("yarn.scheduler.capacity.root.default.acl_submit_applications", "*")
yarnConf.set("yarn.scheduler.capacity.root.default.acl_administer_queue", "*")
yarnConf.setInt("yarn.scheduler.capacity.node-locality-delay", -1)
+ // `maximum-am-resource-percent` defaults to 0.1, which caps the queue's total AM resource
+ // usage to 10% of its capacity. On memory-constrained CI runners this becomes ~1GB, smaller
+ // than the AM/driver memory these tests request (1-2GB), so applications get stuck in the
+ // ACCEPTED state (never activated) and the suite times out waiting for a final state. Let
+ // AMs use the whole queue in tests so applications are always activated.
+ yarnConf.setFloat("yarn.scheduler.capacity.maximum-am-resource-percent", 1.0f)
+ yarnConf.setFloat("yarn.scheduler.capacity.root.default.maximum-am-resource-percent", 1.0f)
// Support both IPv4 and IPv6
yarnConf.set("yarn.resourcemanager.hostname", Utils.localHostNameForURI())
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
(spark) 02/03: [SPARK-57710][YARN][TEST] Reduce YarnClusterSuite flakiness from CI runner contention
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 9f797600b16aa09aa5afc3c238924f9801fb3d4c
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Sat Jun 27 11:09:09 2026 +0900
[SPARK-57710][YARN][TEST] Reduce YarnClusterSuite flakiness from CI runner contention
### What changes were proposed in this pull request?
Follow-up to [SPARK-57650](https://issues.apache.org/jira/browse/SPARK-57650), which fixed the deterministic "AM stuck in ACCEPTED" hang in `BaseYarnClusterSuite`. Two further **test-only** changes to reduce the remaining flakiness of `YarnClusterSuite`:
1. Give the single mini `NodeManager` 8GB (`yarn.nodemanager.resource.memory-mb` + `yarn.scheduler.maximum-allocation-mb`) so executor allocation is never starved once the ~1.4GB AM is running.
2. Raise the executor→driver connection retry budget for the launched apps (`spark.rpc.io.maxRetries=10`, `spark.rpc.io.retryWait=2s`) so a transient RPC-accept stall does not permanently fail an executor. These are defaults that individual tests can still override via `extraConf`.
### Why are the changes needed?
Even after SPARK-57650, the scheduled `Build / Java21` and `Build / Java25` master lanes fail in the `yarn` module roughly **50% of runs** (e.g. fork run `28151220075` PASS vs `28151247521` FAIL — same commit, 40s apart). All failures are the same six `YarnClusterSuite` tests timing out after 3 minutes (`The code passed to eventually never returned normally ... handle.getState().isFinal() was false`).
From the `yarn-app-log` / `unit-tests-log` artifacts, the AM/driver comes up, but the executor (and sometimes the AM) intermittently fail to connect back to the driver's RPC server on `localhost` (`java.io.IOException: Failed to connect to localhost/127.0.0.1:<port>`, connection refused). The in-JVM mini RM+NM, the driver subprocess and the AM/executor JVMs all contend for CPU on a single CI runner, so the driver's accept loop occasionally stalls; an executor that loses this race exit [...]
### Does this PR introduce any user-facing change?
No. Test-only.
### How was this patch tested?
`YarnClusterSuite` was previously failing ~50% of the time. With this change the `yarn` module job was run **6 times** on the fork; all 6 passed, with `YarnClusterSuite` reporting `tests=30, failures=0, skipped=0` (the 6 formerly-failing tests now pass):
- Before (master, failing): apache/spark run `28148781009` (`Build / Java21`) — 6 `YarnClusterSuite` timeouts.
- After (this branch): HyukjinKwon/spark runs `28162182834`, `28162247111`, `28162249819`, `28175759262`, `28175762257`, `28175765871` — `yarn` job green in all six.
### Was this patch authored or co-authored using generative AI tooling?
Yes, Generated-by: Claude Code.
Closes #56785 from HyukjinKwon/ci-fix/agent8-yarn-cluster-flaky.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
(cherry picked from commit 41f3e67e3ebe9ca654af98c6464d607c92ae97f3)
---
.../spark/deploy/yarn/BaseYarnClusterSuite.scala | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
index a22d609370bb..835b1547befd 100644
--- a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
+++ b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
@@ -111,6 +111,14 @@ abstract class BaseYarnClusterSuite extends SparkFunSuite with Matchers {
yarnConf.setFloat("yarn.scheduler.capacity.maximum-am-resource-percent", 1.0f)
yarnConf.setFloat("yarn.scheduler.capacity.root.default.maximum-am-resource-percent", 1.0f)
+ // Give the single mini NodeManager generous memory. By default the mini cluster advertises
+ // only a small amount of memory, so once the AM (~1.4GB) is running there is barely enough
+ // headroom left for the executors these tests request. On a busy CI runner that makes
+ // executor allocation slow/racy and the YarnClusterSuite apps time out waiting to finish.
+ // The CI hosts have plenty of RAM, so let the NM offer enough for the AM plus a few executors.
+ yarnConf.setInt("yarn.nodemanager.resource.memory-mb", 8192)
+ yarnConf.setInt("yarn.scheduler.maximum-allocation-mb", 8192)
+
// Support both IPv4 and IPv6
yarnConf.set("yarn.resourcemanager.hostname", Utils.localHostNameForURI())
@@ -292,6 +300,18 @@ abstract class BaseYarnClusterSuite extends SparkFunSuite with Matchers {
props.setProperty(k, v)
}
}
+
+ // On a busy CI runner the in-JVM mini cluster, the driver and the container JVMs all compete
+ // for CPU, and the driver's RPC server occasionally cannot accept a connection in time. With
+ // the default of 3 retries an executor that loses this race gives up and exits, which leaves
+ // the application unable to finish and the suite times out. Give the executor->driver
+ // connection a larger retry budget so a transient stall does not permanently fail the app.
+ // Set after the spark.* JVM properties are copied above so these values are not silently
+ // overridden by an inherited -Dspark.rpc.io.* flag; individual tests can still override them
+ // via extraConf below.
+ props.setProperty("spark.rpc.io.maxRetries", "10")
+ props.setProperty("spark.rpc.io.retryWait", "2s")
+
extraConf.foreach { case (k, v) => props.setProperty(k, v) }
val propsFile = File.createTempFile("spark", ".properties", tempDir)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org