You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "ConfX (Jira)" <ji...@apache.org> on 2025/12/04 04:37:00 UTC

[jira] [Created] (YARN-11904) TestNMClient has race condition causing intermittent IndexOutOfBoundsException

ConfX created YARN-11904:
----------------------------

             Summary: TestNMClient has race condition causing intermittent IndexOutOfBoundsException
                 Key: YARN-11904
                 URL: https://issues.apache.org/jira/browse/YARN-11904
             Project: Hadoop YARN
          Issue Type: Bug
          Components: test, yarn-client
    Affects Versions: 3.3.5
         Environment: - OS: macOS Darwin 25.0.0 (also reproducible on other platforms)
- Java: OpenJDK 1.8
- Maven: 3.6+
- Test: org.apache.hadoop.yarn.client.api.impl.TestNMClient
            Reporter: ConfX


## DESCRIPTION:

The TestNMClient test class has a race condition in its @Before setup() method
that causes intermittent test failures with IndexOutOfBoundsException.

The issue occurs because the test fetches NodeManager reports immediately after
starting the YARN cluster, without waiting for NodeManagers to fully register
and transition to RUNNING state. This results in an empty nodeReports list,
which later causes an IndexOutOfBoundsException when the test tries to access
nodeReports.get(0) in the allocateContainers() method.

This is a timing-dependent bug that may pass on fast hardware (where NMs register
quickly) but fails on slower systems or under load.

 

## ROOT CAUSE:

File: hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestNMClient.java
Method: setup() (lines 160-182)

Problematic code sequence:

 
{code:java}
@Before
public void setup() throws YarnException, IOException {
    // start minicluster
    yarnCluster = new MiniYARNCluster(TestAMRMClient.class.getName(), nodeCount, 1, 1);
    yarnCluster.init(conf);
    yarnCluster.start();              // ← NodeManagers start asynchronously    // start rm client
    yarnClient = (YarnClientImpl) YarnClient.createYarnClient();
    yarnClient.init(conf);
    yarnClient.start();    // get node info
    nodeReports = yarnClient.getNodeReports(NodeState.RUNNING);  // ← RACE CONDITION!
    // At this point, NodeManagers may not have registered yet
    // Result: nodeReports is EMPTY    // ... rest of setup ...
} {code}
Later in the test, allocateContainers() method tries to access:
{code:java}
String node = nodeReports.get(0).getNodeId().getHost();  // ← IndexOutOfBoundsException! {code}
## STEPS TO REPRODUCE:
 # Check out Apache Hadoop 3.3.5 source code
 # Navigate to hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
 # Run the test:  
{code:java}
mvn test -Dtest=TestNMClient#testNMClientNoCleanupOnStop   {code}

 # The test may fail intermittently depending on system performanceNote: The failure is timing-dependent. It may pass on fast systems but fail on slower systems or when system is under load.

## EXPECTED RESULT:

Test should wait for NodeManagers to register and be in RUNNING state before
fetching node reports. The test should pass consistently regardless of system
performance.

 

ACTUAL RESULT:

Test fails with IndexOutOfBoundsException:
{code:java}
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    at java.util.ArrayList.rangeCheck(ArrayList.java:659)
    at java.util.ArrayList.get(ArrayList.java:435)
    at org.apache.hadoop.yarn.client.api.impl.TestNMClient.allocateContainers(TestNMClient.java:324)
    at org.apache.hadoop.yarn.client.api.impl.TestNMClient.testNMClientNoCleanupOnStop(TestNMClient.java:290){code}
## PROPOSED FIX:

Add proper synchronization in the setup() method to wait for NodeManagers
before fetching node reports:
{code:java}
@Before
public void setup() throws YarnException, IOException {
    // start minicluster
    conf = new YarnConfiguration();
    conf.set(YarnConfiguration.NM_CONTAINER_STATE_TRANSITION_LISTENERS,
        DebugSumContainerStateListener.class.getName());
    yarnCluster =
        new MiniYARNCluster(TestAMRMClient.class.getName(), nodeCount, 1, 1);
    yarnCluster.init(conf);
    yarnCluster.start();
    assertNotNull(yarnCluster);
    assertEquals(STATE.STARTED, yarnCluster.getServiceState());    // Wait for NodeManagers to connect
    try {
        if (!yarnCluster.waitForNodeManagersToConnect(30000)) {
            fail("NodeManagers failed to connect within 30 seconds");
        }
    } catch (InterruptedException e) {
        fail("Interrupted while waiting for NodeManagers: " + e.getMessage());
    }    // start rm client
    yarnClient = (YarnClientImpl) YarnClient.createYarnClient();
    yarnClient.init(conf);
    yarnClient.start();
    assertNotNull(yarnClient);
    assertEquals(STATE.STARTED, yarnClient.getServiceState());    // get node info - wait for nodes to be in RUNNING state
    int retries = 10;
    while (retries > 0) {
        nodeReports = yarnClient.getNodeReports(NodeState.RUNNING);
        if (nodeReports != null && !nodeReports.isEmpty()) {
            break;
        }
        sleep(1000);
        retries--;
    }    if (nodeReports == null || nodeReports.isEmpty()) {
        fail("No NodeManagers in RUNNING state after waiting");
    }    // ... rest of setup ...
} {code}
I'm happy to submit a patch for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org