You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Wilfred Spiegelenburg (Jira)" <ji...@apache.org> on 2024/09/12 03:50:00 UTC
[jira] [Commented] (YUNIKORN-2860) submit gang applications Simultaneously may cause unexpected pending apps
[ https://issues.apache.org/jira/browse/YUNIKORN-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881159#comment-17881159 ]
Wilfred Spiegelenburg commented on YUNIKORN-2860:
-------------------------------------------------
This is partially a K8s issue and partially ours. There is no guarantee that the pods we create for gang scheduling placeholders come back in a pre-defined order. Even if and when we process thing serially there is no guarantee that we get them back in that order.
I have seen before that we create 10 placeholders and the first pod that K8s returns to us is the last pod we created. That becomes worse when multiple origins (applications) are involved. We pass on from the k8shim to the core serially.
I think we can improve this if we track the applications that have requested gangs in a queue in order of creation and only service the ones that fit in the queue or just the first one out of that list until all placeholders are allocated. That could be investigated for a next release. Not sure if it can work as something like this might have side effects we do not want or cause bigger issues. For instance if the placeholder pods continually fail placements due to predicates or something we need to "escape" this.
> submit gang applications Simultaneously may cause unexpected pending apps
> ---------------------------------------------------------------------------
>
> Key: YUNIKORN-2860
> URL: https://issues.apache.org/jira/browse/YUNIKORN-2860
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Affects Versions: 1.3.0, 1.4.0, 1.5.0, 1.5.1, 1.5.2
> Reporter: shawn
> Priority: Major
> Attachments: image-2024-09-11-15-41-12-142.png, image-2024-09-11-15-42-07-739.png, state-dump.txt, yunikorn-scheduler.txt
>
>
>
> I Simultaneously submit 4 gang apps to yunikorn，sometimes 4 apps get pending, while two pgs get running, that's not expected.
> It can be reproduced as follows：
> queues
> 1.kubectl create configmap yunikorn-configs --from-file=queues.yaml -n yunikorn
> * queues.yaml
> {code:java}
> partitions:
> - name: default
> queues:
> - name: root
> queues:
> - name: my-dev
> submitacl: "*"
> resources:
> guaranteed: { memory: 1G, vcore: 1 }
> max: { memory: 2G, vcore: 2 }{code}
> 2.Simultaneously submit gang-scheduling-job-example1-4.yaml, while
> gang-scheduling-job-example1-4.yaml only differ in name and applicationId
> {code:java}
> apiVersion: batch/v1
> kind: Job
> metadata:
> name: gang-scheduling-job-example1
> spec:
> completions: 2
> parallelism: 2
> template:
> metadata:
> labels:
> app: sleep
> applicationId: "gang-scheduling-job-example1"
> queue: root.my-dev
> annotations:
> yunikorn.apache.org/task-group-name: task-group-example-0
> yunikorn.apache.org/task-groups: |-
> [{
> "name": "task-group-example-0",
> "minMember": 2,
> "minResource": {
> "cpu": "1",
> "memory": "1G"
> },
> "nodeSelector": {},
> "tolerations": [],
> "affinity": {}
> }]
> spec:
> schedulerName: yunikorn
> restartPolicy: Never
> containers:
> - name: sleep30
> image: "nginx:latest"
> command: ["sleep", "999999999"]
> resources:
> requests:
> cpu: "1"
> memory: "1G" {code}
> finally，kubectl get pods -n default gets unexpected result(not always reproducible)
> !image-2024-09-11-15-41-12-142.png!
>
> app state as follows
> !image-2024-09-11-15-42-07-739.png|width=754,height=280!
> full state dump as state-dump.txt, yunikorn scheduler logs are in yunikorn-scheduler.txt
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org