[jira] [Assigned] (YUNIKORN-1724) Improve the performance of shim side scheduling cycle

["Peter Bacsko (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YUNIKORN-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Peter Bacsko reassigned YUNIKORN-1724:
--------------------------------------

    Assignee: Peter Bacsko

> Improve the performance of shim side scheduling cycle
> -----------------------------------------------------
>
>                 Key: YUNIKORN-1724
>                 URL: https://issues.apache.org/jira/browse/YUNIKORN-1724
>             Project: Apache YuniKorn
>          Issue Type: Sub-task
>          Components: shim - kubernetes
>            Reporter: Peter Bacsko
>            Assignee: Peter Bacsko
>            Priority: Major
>
> Performance testing of Yunikorn uncovered that a lot of time is spent in {{Application.Schedule()}} in the shim. The problem is related to the fact that we collect task objects based on their state which is maintained by {{{}fsm.FSM{}}}. Even though we run {{Application.Schedule()}} once per second, it's still an issue due to the large number of {{RWMutex.RLock()}} calls. With a lot of pods, this consumes significant amount of CPU time.
> Also, different code paths are affected:
> The first is inside the switch-case part in {{{}Schedule(){}}}. We want to know the number of tasks in "New" state and we end up scanning all task objects for their status. 
> The second is retrieving the "New" tasks from {{taskMap}} structure. This is done by {{GetNewTasks()}} / {{{}getTasks(){}}}, copying tasks based on their respective state to a new slice.
> To speed things up, we have to track the "New" tasks in a new map which is dynamically maintained when a new task added and when it leaves the New state (or the task gets removed). This also helps knowing how many of them we have without counting.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org