[jira] [Created] (YUNIKORN-921) Applications should be able to opt-out of state-aware scheduling

["Craig Condit (Jira)" <ji...@apache.org>]

Craig Condit created YUNIKORN-921:
-------------------------------------

             Summary: Applications should be able to opt-out of state-aware scheduling
                 Key: YUNIKORN-921
                 URL: https://issues.apache.org/jira/browse/YUNIKORN-921
             Project: Apache YuniKorn
          Issue Type: Improvement
          Components: core - scheduler, shim - kubernetes
            Reporter: Craig Condit
            Assignee: Craig Condit
             Fix For: 1.0.0


When pods are submitted to YuniKorn without an associated applicationId label, the admission controller assigns a generated applicationId for that pod. However, if this pod is the first (or only) pod submitted, the generated application will block other applications from executing until a second pod is submitted, or until 5 minutes have elapsed. Since we have no way to know if another pod will be scheduled for the generated application, we should have a way to skip state-aware scheduling in this case and avoid the 5 minute delay.

To fix this:

1) Scheduler core: Add a new tag "application.stateaware.disable" which if present, will prevent waiting for a second task to transition to Running state.

2) Admission controller: Add new label "stateAware: false" to a pod if neither applicationId nor spark-app-selector is provided.

3) K8S Shim: When creating a New application in the core, if the stateAware label exists and is false, set the "application.stateaware.disable" tag.

In addition to bypassing state-aware scheduling for generated apps, the addition of this label / tag also gives users a mechanism to opt out on a per-application basis from state-aware scheduling if necessary, such as an application containing only a single pod, or one where a second pod may not be launched in a timely manner.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org