You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2026/06/26 02:53:50 UTC

[PR] [SPARK-57650][YARN][TESTS][FOLLOWUP] Reduce YarnClusterSuite flakiness from runner contention [spark]

HyukjinKwon opened a new pull request, #56800:
URL: https://github.com/apache/spark/pull/56800

   ### What changes were proposed in this pull request?
   
   Two test-only mitigations in `BaseYarnClusterSuite` to reduce `YarnClusterSuite` flakiness on busy CI runners:
   
   - Give the mini `NodeManager` 8GB (`yarn.nodemanager.resource.memory-mb`) and a matching `yarn.scheduler.maximum-allocation-mb`, so executor allocation is never starved once the ~1.4GB AM is running.
   - Raise the executor→driver connection retry budget (`spark.rpc.io.maxRetries=10`, `spark.rpc.io.retryWait=2s`) so a transient accept stall does not permanently fail the executor. These are defaults; individual tests can still override them.
   
   ### Why are the changes needed?
   
   This is a follow-up to SPARK-57650, which fixed the deterministic ACCEPTED-state hang (`maximum-am-resource-percent`). After that fix, the master `Build / Maven` `yarn` lanes still go red intermittently: `YarnClusterSuite` tests time out (`handle.getState().isFinal() was false`) because the AM/executor containers fail to connect to the driver's RPC server on localhost (Connection refused).
   
   The in-JVM mini RM+NM, the driver subprocess, and the AM/executor JVMs all contend for CPU on a single CI runner, so the driver's accept loop occasionally stalls; an executor that loses the race exits after the default 3 connection retries, and the application can then never finish.
   
   This is one of the two remaining unmerged fixes keeping the apache/spark master matrix builds red.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No. Test-only.
   
   ### How was this patch tested?
   
   `resource-managers/yarn` module tests on a fork, repeated multiple times to confirm the flakiness is gone (the formerly-failing `YarnClusterSuite` tests pass consistently).
   
   - **Before (still flaky on apache/spark master, after the SPARK-57650 base fix merged):** `Build / Maven (Scala 2.13, JDK 17)` — `resource-managers#yarn`, `YarnClusterSuite` times out (`handle.getState().isFinal() was false`, `BaseYarnClusterSuite.scala:220`): https://github.com/apache/spark/actions/runs/28174751831
   - **After (passing on this branch):** `resource-managers/yarn` module green on the fork — https://github.com/HyukjinKwon/spark/actions/runs/28204995015/job/83556094951
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Yes.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57650][YARN][TESTS][FOLLOW-UP] Reduce YarnClusterSuite flakiness from runner contention [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #56800:
URL: https://github.com/apache/spark/pull/56800#issuecomment-4806722818

   Closing as a duplicate of #56785, which carries the same test-only YarnClusterSuite mitigations as a focused draft and is CI-validated (full yarn-module Maven on JDK17 + JDK21). This branch also picked up unrelated commits. Let's consolidate on #56785.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57650][YARN][TESTS][FOLLOW-UP] Reduce YarnClusterSuite flakiness from runner contention [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #56800: [SPARK-57650][YARN][TESTS][FOLLOW-UP] Reduce YarnClusterSuite flakiness from runner contention
URL: https://github.com/apache/spark/pull/56800


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org