[jira] [Commented] (FLINK-15648) Support to configure limit for CPU requirement

["Xintong Song (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087666#comment-17087666 ] 

Xintong Song commented on FLINK-15648:
--------------------------------------

[~felixzheng], sorry to jump into the discussion late.

I agree with you that it is a good thing to leverage the Kubernetes feature of container request/limit cpu. However, I'm not entirely sure about the proposed approach that introduces configuration options for the absolute cpu limit for JM/TM containers.

It might be ok with the current Flink resource management, where we have only two kinds of containers. But it may not work well with FLINK-14106, where we no longer assumes that all the TMs have the same amount of resources. With the efforts towards fine grained resource management, we might eventually have SlotManager plugin that dynamically decides TM resources based on the workload. In such cases, we may not have a proper `kubernetes.taskmanager.limit.cpu` value that works well with all TMs with different resources, and the SlotManager plugin should not be aware of the specific underlying resource manager (K8s, Yarn or Mesos) and decide the Kubernetes specific 'limit.cpu'.

Alternatively, we might have a configuration option 'kubernetes.container.cpu-limit-ratio' or so, that calculate the 'limit.cpu' from the 'request.cpu'. E.g., if the requested cpu is 2.0 and the ratio is 1.5, then the limit would be 3.0. I think this gives you practically the same controllability as the proposed approach, while naturally works with dynamic TM resources.

WDYT?

> Support to configure limit for CPU requirement
> ----------------------------------------------
>
>                 Key: FLINK-15648
>                 URL: https://issues.apache.org/jira/browse/FLINK-15648
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Deployment / Kubernetes
>            Reporter: Canbin Zheng
>            Assignee: Canbin Zheng
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.11.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> The current branch use kubernetes.xx.cpu to configure request and limit resource requirement for a Container, it's an important improvement to separate these two configurations, we can use kubernetes.xx.request.cpu and kubernetes.xx.limit.cpu to specify request and limit resource requirements.{color:#6a8759}
> {color}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)