You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Rainie Li (Jira)" <ji...@apache.org> on 2022/08/24 22:07:00 UTC
[jira] [Comment Edited] (YUNIKORN-790) Implement MaxApplications enforcement
[ https://issues.apache.org/jira/browse/YUNIKORN-790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584485#comment-17584485 ]
Rainie Li edited comment on YUNIKORN-790 at 8/24/22 10:06 PM:
--------------------------------------------------------------
hi [~wilfreds] I addressed your comments for [https://github.com/apache/yunikorn-core/pull/429] && updated change in [https://github.com/apache/yunikorn-core/pull/435 |https://github.com/apache/yunikorn-core/pull/435]
Can you please review #435? I will abandon #429.
*Summary:*
The reason I added a new method *incAllocatingAcceptedAppsIfCanRun()* is we increase runningApps when application enters _Starting_ state, this logic is executed in a separate thread from where we check canRun.
During large scale load testing, we saw concurrency issue: several apps can be accepted and passed canRun check at the same time. After they got allocated resource, they will be added to runningApps. It can exceed MaxApplications value.
*Potential issue & proposed solution:*
With my change, this concurrency issue will still happen if I ran large scale testing in extreme case: I set 5 Yunikorn queues, each queue maxApplication value is set to 12. Then I launched 1000 spark jobs concurrently , each queue got launched 200 jobs. I saw runningApps exceeded maxApplication after an hour. First hour runningApps was limited.
To address this issue, I added an additional check [https://github.com/apache/yunikorn-core/pull/435/files#diff-eeab7f6e1845e8d0ca68c02daa7151e8d68a5e7fc6cf4247b74632956d54f359R1201 |https://github.com/apache/yunikorn-core/pull/435/files#diff-eeab7f6e1845e8d0ca68c02daa7151e8d68a5e7fc6cf4247b74632956d54f359R1201]
When we check canRun again in tryNode, we can guarantee runningApps in each queue always be limited to maxApplication value. I have ran several large scale load tests to validate it's working. However, with this additional check, it will slow down the test. That's why I commented this change in PR #435.
*Questions:*
1.Is concurrency issue in large scale load extreme case is accepted? (We will not have same extreme case in our production environment, not sure others)
2.If 1 is not accepted, any other suggestion to address this concurrency issue beside the second canRun check in tryNode method?
[~pbacsko] I tested in a real k8s cluster with 85nodes, not minikube.
was (Author: rainieli):
hi [~wilfreds] I addressed your comments for [https://github.com/apache/yunikorn-core/pull/429] && updated change in [https://github.com/apache/yunikorn-core/pull/435 |https://github.com/apache/yunikorn-core/pull/435]
Can you please review #435? I will abandon #429.
*Summary:*
The reason I added a new method *incAllocatingAcceptedAppsIfCanRun()* is we increase runningApps when application enters _Starting_ state, this logic is executed in a separate thread from where we check canRun.
During large scale load testing, we saw concurrency issue: several apps can be accepted and passed canRun check at the same time. After they got allocated resource, they will be added to runningApps. It can exceed MaxApplications value.
*Potential issue & proposed solution:*
With my change, this concurrency issue will still happen if I ran large scale testing in extreme case: I set 5 Yunikorn queues, each queue maxApplication value is set to 12. Then I launched 1000 spark jobs concurrently , each queue got launched 200 jobs. After 1 hour, I saw runningApps exceeded maxApplication after test ran for an hour.
To address this issue, I added an additional check [https://github.com/apache/yunikorn-core/pull/435/files#diff-eeab7f6e1845e8d0ca68c02daa7151e8d68a5e7fc6cf4247b74632956d54f359R1201 |https://github.com/apache/yunikorn-core/pull/435/files#diff-eeab7f6e1845e8d0ca68c02daa7151e8d68a5e7fc6cf4247b74632956d54f359R1201]
When we check canRun again in tryNode, we can guarantee runningApps in each queue always be limited to maxApplication value. I have ran several large scale load tests to validate it's working. However, with this additional check, it will slow down the test. That's why I commented this change in PR #435.
*Questions:*
1.Is concurrency issue in large scale load extreme case is accepted? (We will not have same extreme case in our production environment, not sure others)
2.If 1 is not accepted, any other suggestion to address this concurrency issue beside the second canRun check in tryNode method?
[~pbacsko] I tested in a real k8s cluster with 85nodes, not minikube.
> Implement MaxApplications enforcement
> -------------------------------------
>
> Key: YUNIKORN-790
> URL: https://issues.apache.org/jira/browse/YUNIKORN-790
> Project: Apache YuniKorn
> Issue Type: New Feature
> Components: core - scheduler
> Reporter: Wilfred Spiegelenburg
> Assignee: Rainie Li
> Priority: Major
> Labels: pull-request-available
>
> Queues have an option to set the MaxApplications that can run in a queue. There is currently no code in the scheduler that checks this setting.
> As a new feature we should add the enforcement for this setting:
> * enforce the setting on a leaf queue
> * enforce the setting on a parent, the apps running in a parent queue is defined as the sum of all the apps running in all leaf queues of the parent.
> As a side note from a config check: we need to make sure that the parent setting cannot be lower than any of the child queues it has. We _must not_ enforce that the parent setting must be larger than sum of all leafs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org