[jira] [Commented] (YARN-12004) Fix Router-vs-RM read races in TestRouterWebServicesREST and TestFederationWebApp

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-12004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18122202#comment-18122202 ] 

ASF GitHub Bot commented on YARN-12004:
---------------------------------------

joseluisll opened a new pull request, #8786:
URL: https://github.com/apache/hadoop/pull/8786

   ### Description of PR
   
   JIRA: https://issues.apache.org/jira/browse/YARN-12004
   
   `TestRouterWebServicesREST`, and `TestFederationWebApp` which extends it, fail intermittently on CI because several tests don't wait for the RM's asynchronous application lifecycle. The change is test-only, in `TestRouterWebServicesREST.java`.
   
   - **`getAppAttempt`** read `getAttempts().get(0)` right after the submission returned, before the RM had created the first attempt, and threw `IndexOutOfBoundsException`. It now waits for the attempt, up to 10s. `testGetAppAttemptXML` and `testGetContainersXML` use it, and `testGetAppsMultiThread` calls `testGetContainersXML`.
   - **`testAppXML`, `testAppStateXML`, `testAppAttemptXML`, `testGetContainersXML`** each compared one Router read with one RM read of a value the RM was still changing (AM host, state, attempts, containers). Failures look like `expected: <ACCEPTED> but was: <SUBMITTED>` and `expected: <0> but was: <1>`.
   - A new **`assertRouterMatchesRM`** helper reads the RM, then the Router, then the RM again:
     - If the RM changed between its two reads, it retries.
     - If the RM was stable, the Router must match it. A Router that disagrees with a stable RM twice fails at once, so a broken Router can't pass just because the RM's value eventually reaches whatever the Router reports.
   - The checks compare **values that can't repeat**. `testGetContainersXML` compares container IDs instead of counts, which go 1 → 0 → 1 as one attempt fails and the next starts. `testAppXML` also compares the AM container log URL, which identifies the attempt.
   - The tests that now wait get a 30s `@Timeout` instead of 2s, so the 10s wait can't be cut short. `performGetCalls` now closes the two clients it creates.
   
   Tests that compare values which don't change after submission (priority, queue, timeouts, app counts) are unchanged.
   
   
   ### How was this patch tested?
   
   - **Before the fix:** looping `testAppXML`, `testAppStateXML` and `testAppAttemptXML` 3000 times each failed 8, 14 and 38 times. Upstream CI on #8681 failed in `TestFederationWebApp` and `TestRouterWebServicesREST`, and the same two classes are the failing router tests in the CI reports on HADOOP-19970, HADOOP-19972 and HADOOP-19982.
   - **After the fix:** looping each of the four changed methods 3200 times, in three fresh JVMs, failed none.
   - **Negative check:** with a Router changed to always report no attempts, `testAppAttemptXML` fails 100 out of 100 runs, in about 0.5s each.
   - **Full classes:** Linux, JDK 21: `TestRouterWebServicesREST` 42/42 and `TestFederationWebApp` 52/52 pass.
   
   ### For code changes:
   
   - [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [ ] Object storage: Have the integration tests been executed and the endpoint
         declared according to the connector-specific documentation? *Note: Automated CI
         testing doesn't cover all cases so manual testing with cloud storage is still
         required.*
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
   
   ### AI Tooling
   
   If an AI tool was used:
   
   - [x] The PR includes the phrase "Contains content generated by <tool>"
         where <tool> is the name of the AI tool used.
   - [x] My use of AI contributions follows the ASF legal policy
         https://www.apache.org/legal/generative-tooling.html
   
   Contains content generated by Claude Code.




> Fix Router-vs-RM read races in TestRouterWebServicesREST and TestFederationWebApp
> ---------------------------------------------------------------------------------
>
>                 Key: YARN-12004
>                 URL: https://issues.apache.org/jira/browse/YARN-12004
>             Project: Hadoop YARN
>          Issue Type: Test
>          Components: router, test
>    Affects Versions: 3.6.0
>            Reporter: Jose Luis López
>            Priority: Minor
>
> {{{}TestRouterWebServicesREST{}}}, and {{TestFederationWebApp}} which extends it, fail intermittently on CI because several tests do not wait for the RM's asynchronous application lifecycle.
> h3. Symptoms
>  * {{getAppAttempt}} reads {{getAttempts().get(0)}} right after the submission returns, before the RM has created the first attempt, and throws {{{}IndexOutOfBoundsException{}}}. This affects {{testGetAppAttemptXML}} and {{{}testGetContainersXML{}}}, which {{testGetAppsMultiThread}} also calls.
>  * {{{}testAppXML{}}}, {{{}testAppStateXML{}}}, {{testAppAttemptXML}} and {{testGetContainersXML}} each compare one Router read with one RM read, taken one after the other, of a value the RM is still changing on a newly submitted application: the AM host, the application state, the number of attempts, the attempt's containers. Typical failures:
> {noformat}
> expected: <ACCEPTED> but was: <SUBMITTED>
> expected: <0> but was: <1>
> {noformat}
> Looping {{{}testAppXML{}}}, {{testAppStateXML}} and {{testAppAttemptXML}} 3000 times each against unchanged trunk failed 8, 14 and 38 times respectively. Upstream CI on [https://github.com/apache/hadoop/pull/8681] (YARN-11984) failed in both {{TestFederationWebApp}} and {{TestRouterWebServicesREST}} for these reasons.
> h3. Proposed fix (test code only)
>  * {{getAppAttempt}} waits, up to 10s, for the first attempt to appear.
>  * A new {{assertRouterMatchesRM}} helper reads the RM, then the Router, then the RM again.
>  ** If the RM changed between its two reads, it retries.
>  ** If the RM was stable, the Router must match it. A Router that disagrees with a stable RM twice fails at once, so a broken Router cannot pass just because the RM's value eventually reaches whatever the Router reports.
>  * The checks compare values that cannot repeat. {{testGetContainersXML}} compares container IDs instead of counts, which go 1, 0, 1 as one attempt fails and the next starts. {{testAppXML}} also compares the AM container log URL, which identifies the attempt.
>  * The tests that now wait get a 30s timeout instead of 2s. {{performGetCalls}} now closes the two clients it creates.
> Tests that compare values which do not change after submission (priority, queue, timeouts, application counts) are unchanged. With the fix, looping each of the four methods 3200 times in three fresh JVMs failed none.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org