[jira] [Comment Edited] (FLINK-15648) Support to configure request or limit for CPU requirement

["Canbin Zheng (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028017#comment-17028017 ] 

Canbin Zheng edited comment on FLINK-15648 at 2/1/20 9:10 AM:
--------------------------------------------------------------

Thanks [~fly_in_gis] . This ticket is about the CPU resource requirement, it’s good to narrow the scope.

>>> Could you share the key advantages of separate the request and limit configuration

As you said, one can set the resource limit larger than the request to make the most of the resources, such as share CPU resource when the machine CPU load is low. And of course, one can just disables the CPU limit, just like what you do for the internal use case, so that the Pods could use the CPU resource from the landing machine as much as possible if they need. However it’s not always suitable(also not particular recommend) to disable the CPU limit in production, especially for the machine with large specification and has many Pods running on, because the potential CPU competition brings extra overhead due to the CPU scheduling mechanism, so in some case one needs to provide specific value for the limit configuration option, instead of just disabling the limit.

As to the limit configuration option, it can satisfy your internal use case also, the typical usage is as below:
 # if not set, fallback to the CPU request configuration option
 # if set with number no more than zero, disable the CPU limit
 # set with number larger than the CPU request configuration option

>>> Since it will make kubernetes configuration very different from others(standalone, Yarn, mesos).

Could you share more details why it will make K8s configuration very different from the standalone and mesos cluster managers?

As to the Yarn cluster manager, it do not support well for the request and limit CPU settings, so the {{yarn.xxx.vcores}} configuration option is just good enough.

Lastly, it could be further discussed whether we should deprecate the {{kubernets.xxx.cpu}} configuration option, anyway I propose to provide setting entrypoint for the CPU limit.


was (Author: felixzheng):
Thanks [~fly_in_gis] . This ticket is about the CPU resource requirement, let’s narrow the scope.

>>> Could you share the key advantages of separate the request and limit configuration

As you said, one can set the resource limit larger than the request to make the most of the resources, such as share CPU resource when the machine CPU load is low. And of course, one can just disables the CPU limit, just like what you do for the internal use case, so that the Pods could use the CPU resource from the landing machine as much as possible if they need. However it’s not always suitable(also not particular recommend) to disable the CPU limit in production, especially for the machine with large specification and has many Pods running on, because the potential CPU competition brings extra overhead due to the CPU scheduling mechanism, so in some case one needs to provide specific value for the limit configuration option, instead of just disabling the limit.

As to the limit configuration option, it can satisfy your internal use case also, the typical usage is as below:
 # if not set, fallback to the CPU request configuration option
 # if set with number no more than zero, disable the CPU limit
 # set with number larger than the CPU request configuration option


>>> Since it will make kubernetes configuration very different from others(standalone, Yarn, mesos).

Could you share more details why it will make K8s configuration very different from the standalone and mesos cluster managers?

As to the Yarn cluster manager, it do not support well for the request and limit CPU settings, so the \{{yarn.xxx.vcores}} configuration option is just good enough.


Lastly, it could be further discussed whether we should deprecate the \{{kubernets.xxx.cpu}} configuration option, anyway I propose to provide setting entrypoint for the CPU limit.

> Support to configure request or limit for CPU requirement
> ---------------------------------------------------------
>
>                 Key: FLINK-15648
>                 URL: https://issues.apache.org/jira/browse/FLINK-15648
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Deployment / Kubernetes
>            Reporter: Canbin Zheng
>            Priority: Major
>
> The current branch use kubernetes.xx.cpu to configure request and limit resource requirement for a Container, it's an important improvement to separate these two configurations, we can use kubernetes.xx.request.cpu and kubernetes.xx.limit.cpu to specify request and limit resource requirements.{color:#6a8759}
> {color}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)