[jira] [Updated] (YARN-11202) Optimize ClientRMService.getApplications

["Shilun Fan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-11202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shilun Fan updated YARN-11202:
------------------------------
          Component/s: yarn
     Target Version/s: 3.4.0
    Affects Version/s: 3.4.0

> Optimize ClientRMService.getApplications
> ----------------------------------------
>
>                 Key: YARN-11202
>                 URL: https://issues.apache.org/jira/browse/YARN-11202
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: yarn
>    Affects Versions: 3.4.0
>            Reporter: Tamas Domok
>            Assignee: Tamas Domok
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.4.0
>
>          Time Spent: 50m
>  Remaining Estimate: 0h
>
> YARN-11114 introduced an extra check in ClientRMService.getApplications() where it calculates a map (queue to a list of rmapps) inside a while loop which could have been calculated once before the while loop (which iterates over the apps).
> Code walk-through:
> {code}
> queues:
>  [q1, q2, q3]
> apps:                 (apps running/non-running)
>  [a1, a2, a3, a4, a5, a6, a7, a8, a9]
> inside the while cycle:
>   application:         (current iterator)
>    [a5]
>   foundApps:           (this is re-calculated every time, running apps)
>     q1: [a1, a2]
>     q2: [a4, a5, a6]
>     q3: [a7, a9]
>   runningAppsByQueues: (foundApps is already filtered by queues ... basically this concatenates the lists with extra steps)
>     [a1, a2, a4, a5, a6, a7, a9]
>   runningAppsById:
>     [a5]
> {code}
> I think this could be simplified significantly, basically we just need a set of running app-ids (filtered by queues), which is calculated before the while loop, once / request. Then inside the while loop:
> {code}
> - if (!queues.contains(application.getQueue())) {
> + if (!runningApps.contains(application.getApplicationId()) && !queues.contains(application.getQueue())) {
> {code}
> The tests pass with this fix.
> There is a potential bug inside the queryApplicationsByQueues, between the following time range: a new app is already in a queue, but not in the apps that was queried before:
>   
>   apps = rmContext.getRMApps();
>    ...
>   scheduler.getAppsInQueue(queue);



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org