[jira] [Created] (YARN-12042) [Federation] Router cannot find applications whose home SubCluster is no longer in the StateStore

["QI, Jiale (Jira)" <ji...@apache.org>]

QI, Jiale created YARN-12042:
--------------------------------

             Summary: [Federation] Router cannot find applications whose home SubCluster is no longer in the StateStore
                 Key: YARN-12042
                 URL: https://issues.apache.org/jira/browse/YARN-12042
             Project: Hadoop YARN
          Issue Type: Improvement
            Reporter: QI, Jiale


The RM removes a completed application once it keeps more than yarn.resourcemanager.max-completed-applications, and the home SubCluster of the application is deleted from the Federation StateStore with it, by the RM (FederationStateStoreService) or by the GPG application cleaner. From then on, the Router does not behave like an RM for that application:

1. FederationClientInterceptor#getApplicationReport and #forceKillApplication only look at the StateStore and fail with a YarnException "Application ... does not exist in FederationStateStore". When talking to the RM directly, YarnClient gets an ApplicationNotFoundException and falls back to its history server; through the Router it does not, so `yarn application -status` fails for every application that the RM removed.
2. The other calls that go through FederationClientInterceptor#getApplicationHomeSubCluster ask the RM of each active SubCluster in turn through the regular RM proxies. An RM that is down but not yet marked lost in the StateStore holds the request for yarn.resourcemanager.connect.max-wait.ms before the next one is asked, and an RM that could not be asked is taken as not knowing the application.
3. The REST getApp, getAppState and updateAppState, and the Router web proxy (FedAppReportFetcher), only look at the StateStore.

Proposal:
- Add FederationApplicationRMProbe, which asks the RMs of all the active SubClusters at once, through proxies that try each RM once with short timeouts, and waits at most yarn.router.application-probe.timeout (default 10s) for their answers.
- Use it in the client RPC and REST interceptors and in the web proxy when the StateStore has no home SubCluster. The interceptors record the SubCluster found in the StateStore again, as getApplicationHomeSubCluster already does. They ask the RMs as the client, like their other calls; the web proxy as its login user, like its other calls.
- Answer ApplicationNotFoundException, so that clients fall back to their history server, only when every RM answered that it does not know the application, and pass on the ApplicationNotFoundException of the home SubCluster's RM. If some RM could not be asked, failed or did not answer in time, answer a YarnException so that the client retries, as the application may be running there. The web proxy asks its history server in both cases.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org