You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Craig Condit (Jira)" <ji...@apache.org> on 2022/04/07 01:41:00 UTC

[jira] [Commented] (YUNIKORN-1173) Basic scheduling fails on an existing cluster

    [ https://issues.apache.org/jira/browse/YUNIKORN-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17518537#comment-17518537 ] 

Craig Condit commented on YUNIKORN-1173:
----------------------------------------

The first thing that stands out is that this appears to be a very large pod (32 GB RAM), but the queue capacities are all very small. You might need to check your queue configuration as the base unit of measurement of memory has changed from 1,000,000 bytes in older versions to bytes in newer versions, while CPU cores are now stored internally as millicores. This was done to avoid mismatches between resource consumption when interpreted by K8s vs. YuniKorn, but it is a breaking change.

The queue configuration has been updated to support K8s-style suffixes as well.

From [https://yunikorn.apache.org/docs/next/user_guide/queue_config#resources]:
{noformat}
An optional suffix may be specified for resource quantities. Valid suffixes are k, M, G, T, P, and E for SI powers of 10, and Ki, Mi, Gi, Ti, Pi, and Ei for SI powers of 2. Additionally, resources of type vcore may have a suffix of m to indicate millicores. For example, 500m is 50% of a vcore. Units of type memory are interpreted in bytes by default. All other resource types have no designated base unit.

Note that this is a behavioral change as of YuniKorn 1.0. Prior versions interpreted memory as units of 1000000 (1 million) bytes and vcore as millicores.{noformat}
I suspect your queue configuration is undersizing RAM by a factor of 1 million or so.

> Basic scheduling fails on an existing cluster
> ---------------------------------------------
>
>                 Key: YUNIKORN-1173
>                 URL: https://issues.apache.org/jira/browse/YUNIKORN-1173
>             Project: Apache YuniKorn
>          Issue Type: Bug
>          Components: shim - kubernetes
>            Reporter: Chaoran Yu
>            Priority: Blocker
>         Attachments: logs.txt, statedump.txt
>
>
> Environment: EKS K8s 1.20. 
> K8shim built based on commit: [https://github.com/apache/yunikorn-k8shim/commit/be3bb70d9757b27d0c40d446306b928c79c80a9f]
> Core version used: v0.0.0-20220325135453-73d55282f052
> After YuniKorn is deployed, I deleted one of the pods managed by K8s deployment, but YK didn't schedule the new pod that's created: 
> *spo-og60-03-spark-operator-86cc7ff747-9vzxl* 
> is the name of the new pod. It's stuck in pending and its event said "spark-operator/spo-og60-03-spark-operator-86cc7ff747-9vzxl is queued and waiting for allocation"
> State dump and scheduler logs are attached



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org