[jira] [Comment Edited] (YARN-5179) Issue of CPU usage of containers

["Manikandan R (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15873308#comment-15873308 ] 

Manikandan R edited comment on YARN-5179 at 3/1/17 5:36 PM:
------------------------------------------------------------

[~asuresh]

(edited with more details based on investigation)

Based on my understanding, resourceCalculatorPlugin.getNumProcessors() and maxVCoresAllottedForContainers difference cause this millivcores calculation issue. resourceCalculatorPlugin.getNumProcessors() simply holds of logical processors count. If a node has 4 logical cpu's, then getNumProcessors returns 4.

When yarn.nodemanager.resource.cpu-vcores is -1,

maxVCoresAllottedForContainers has values based on yarn.nodemanager.resource.count-logical-processors-as-cores property. If yarn.nodemanager.resource.count-logical-processors-as-cores is true, then maxVCoresAllottedForContainers is equal to resourceCalculatorPlugin.getNumProcessors(), otherwise, it differs. I am assuming yarn.nodemanager.resource.detect-hardware-capabilities has been enabled and yarn.nodemanager.resource.percentage-physical-cpu-limit is 100 in this case.

When yarn.nodemanager.resource.cpu-vcores is not equal to -1,

maxVCoresAllottedForContainers simply holds value of yarn.nodemanager.resource.cpu-vcores property. Since there is no validation in place, it can have any arbitrary no. (for ex, 100 etc)

Like Memory overflow limit, currently containers won't get killed when cpu usage exceeds. A container can use all the logical cores available in the node. Given this situation, I don't think using maxVCoresAllottedForContainers for millivcores calculation is correct as it can either lead to low or high cpu usage with respect to actual cpu usage when maxVCoresAllottedForContainers != resourceCalculatorPlugin.getNumProcessors().

Approach 1: Can we use actual vcores being used?

Instead of 

{code}float cpuUsageTotalCoresPercentage = cpuUsagePercentPerCore / resourceCalculatorPlugin.getNumProcessors();
int milliVcoresUsed = (int) (cpuUsageTotalCoresPercentage * 1000 * maxVCoresAllottedForContainers /nodeCpuPercentageForYARN);{code}

Can we use this?

{code}float cpuUsageTotalCoresPercentage = cpuUsagePercentPerCore / resourceCalculatorPlugin.getNumVcoresUsed();
int milliVcoresUsed = (int) (cpuUsageTotalCoresPercentage * 1000 * resourceCalculatorPlugin.getNumVcoresUsed()/nodeCpuPercentageForYARN);{code}

Approach 2:

Instead of 

{code}float cpuUsageTotalCoresPercentage = cpuUsagePercentPerCore / resourceCalculatorPlugin.getNumProcessors();
int milliVcoresUsed = (int) (cpuUsageTotalCoresPercentage * 1000 * maxVCoresAllottedForContainers /nodeCpuPercentageForYARN);{code}

Can we use?

{code}float cpuUsageTotalCoresPercentage = cpuUsagePercentPerCore / resourceCalculatorPlugin.getNumProcessors();
int milliVcoresUsed = (int) ((int) (cpuUsageTotalCoresPercentage * resourceCalculatorPlugin.getNumProcessors() * 1000)/100.0f);{code}

Please go through these above options and let me know your suggestions.


was (Author: manirajv06@gmail.com):
[~asuresh]

Based on my understanding, resourceCalculatorPlugin.getNumProcessors() and maxVCoresAllottedForContainers difference cause this millivcores calculation issue. resourceCalculatorPlugin.getNumProcessors() simply return no. of logical processors count, whereas maxVCoresAllottedForContainers has values based on yarn.nodemanager.resource.count-logical-processors-as-cores property. If above property is true, then maxVCoresAllottedForContainers is equal to resourceCalculatorPlugin.getNumProcessors(), otherwise, it differs. I am assuming yarn.nodemanager.resource.detect-hardware-capabilities has been enabled in this case.

Can we use actual vcores being used?

Instead of 

{code}float cpuUsageTotalCoresPercentage = cpuUsagePercentPerCore / resourceCalculatorPlugin.getNumProcessors();
int milliVcoresUsed = (int) (cpuUsageTotalCoresPercentage * 1000 * maxVCoresAllottedForContainers /nodeCpuPercentageForYARN);{code}

Can we use this?

{code}float cpuUsageTotalCoresPercentage = cpuUsagePercentPerCore / resourceCalculatorPlugin.getNumVcoresUsed();
int milliVcoresUsed = (int) (cpuUsageTotalCoresPercentage * 1000 * resourceCalculatorPlugin.getNumVcoresUsed()/nodeCpuPercentageForYARN);{code}

Please provide your thoughts.

> Issue of CPU usage of containers
> --------------------------------
>
>                 Key: YARN-5179
>                 URL: https://issues.apache.org/jira/browse/YARN-5179
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.7.0
>         Environment: Both on Windows and Linux
>            Reporter: Zhongkai Mi
>
> // Multiply by 1000 to avoid losing data when converting to int 
>        int milliVcoresUsed = (int) (cpuUsageTotalCoresPercentage * 1000 
>               * maxVCoresAllottedForContainers /nodeCpuPercentageForYARN); 
> This formula will not get right CPU usage based vcore if vcores != physical cores. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org