You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/03/16 07:25:00 UTC
[jira] [Updated] (YUNIKORN-333) Reduce the number events published
to K8s event system when predicate fails
[ https://issues.apache.org/jira/browse/YUNIKORN-333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated YUNIKORN-333:
------------------------------------
Labels: pull-request-available (was: )
> Reduce the number events published to K8s event system when predicate fails
> ---------------------------------------------------------------------------
>
> Key: YUNIKORN-333
> URL: https://issues.apache.org/jira/browse/YUNIKORN-333
> Project: Apache YuniKorn
> Issue Type: Sub-task
> Components: core - scheduler
> Reporter: Adam Antal
> Assignee: Ting Yao,Huang
> Priority: Major
> Labels: pull-request-available
>
> The problem today is we are publishing too many events to K8s.
> If you look at the code: https://github.com/apache/incubator-yunikorn-k8shim/blob/86cc199c00d44c1dde71c9f2faf5bc17ff28bbb7/pkg/plugin/predicates/predictor.go#L303-L304, this is called in the core scheduling logic upon each allocation, which could happen thousands of times per sec. For example, if a pod could not be allocated onto any of the nodes due to some node taints, it runs a while and we will see a huge number of dup events when we do "kubectl describe pod". So this gives us:
> - good: we do not lose any of events, all pushed to K8s
> - bad: overhead to the K8s event system (but gladly it aggregates the dup events)
> I think there are a few options we can evaluation:
> - Shall we cache such events via the event cache system, and then push them in 1s interval just like what we have done for headRoom check?
> - Add some rate-limit mechanism to reduce number of dup events
> could you pls take a look and let me know your thought. thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org