You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2020/03/02 12:05:02 UTC
[GitHub] [cloudstack] DennisKonrad opened a new issue #3923: cputune shares
in domain xml set without CPU cap enabled
DennisKonrad opened a new issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923
##### ISSUE TYPE
* Bug Report
##### COMPONENT NAME
KVM Domain XML
LibvirtComputingResource
##### CLOUDSTACK VERSION
current master
##### CONFIGURATION
We see VMs with an compute offering that has CPU cap disabled that have CPU shares set in their domain XML.
Also we see that VMs that use very similar offerings that have "offer HA" enabled, do not have see cpu shares in their XML.
##### SUMMARY
cputune parameter should not be enabled when cpu cap is disabled for a vm.
##### STEPS TO REPRODUCE
Create offering with "offer HA" disabled and check domain XML file of an VM created with it. cputune and shares are set.
Create offering with "offer HA" enabled and check domain XML file of an VM created with it. cputune and shares are set.
##### EXPECTED RESULTS
When cpu cap is disabled there should be no cputune with shares in the domain xml.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3923: cputune shares
in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
DennisKonrad commented on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-595693355
@weizhouapache We are very likely affected by this:
https://bugzilla.redhat.com/show_bug.cgi?id=1789824
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3923: cputune shares
in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
DennisKonrad commented on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-593407909
@weizhouapache That's really weird...
Okay, do you think this should be like this? In my opinion cputune should not be set when cpu cap is disabled. Also it causes huge issues for us when VMs are maxing out on CPU usage. Steal time goes up to 50% with cputune shares
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] weizhouapache commented on issue #3923: cputune shares
in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-593587796
> schedinfo
@DennisKonrad
cpu_shares in "virsh schedinfo <instance name>"
= virsh dumpxml <instance name> | grep shares
= cpu cores * cpu speed / cpu.overprovisioning.factor
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3923: cputune shares
in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
DennisKonrad commented on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-595672233
Maybe it's not the OS but qemu or libvirt version. It looks like it's "right" way to configure it. But it could also be this worked some time before so we didn't notice. And after some update it stopped to write the shares to cgroup. Only speculation though
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] DennisKonrad edited a comment on issue #3923: cputune
shares in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
DennisKonrad edited a comment on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-593866590
2 Cores * 1000 MHz / 10 = 200
In reality theres 66 set. Did I get you right on the calculation? @weizhouapache
**EDIT**
I found why the shares are not set correctly anymore. In our last update the qemu.conf got replaced and there were no controllers configured. After repairing that the values match beween XML, schedinfo and cgroup.
<details>
<summary>Click to expand!</summary>
```
<domain type='kvm' id='148'>
...
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static'>2</vcpu>
<cputune>
<shares>66</shares>
</cputune>
```
Schedinfo is already wrong:
```
Scheduler : posix
cpu_shares : 1024
vcpu_period : 100000
vcpu_quota : -1
...
```
Cgroup (or CFS) shares are set falsely set also (as one would expect):
```
# cat cpu.shares
1024
# pwd
/sys/fs/cgroup/cpu/machine.slice/machine-qemu\x2d148\x2di\x2d331\x2d2003\x2dVM.scope
```
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] DennisKonrad edited a comment on issue #3923: cputune
shares in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
DennisKonrad edited a comment on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-593866590
2 Cores * 1000 MHz / 10 = 200
In reality theres 66 set. Did I get you right on the calculation? @weizhouapache
**EDIT**
I found why the shares are not set correctly anymore. In our last update the qemu.conf got replaced and there were no controllers configured. After repairing that the values match beween XML, schedinfo and cgroup.
<details>
<summary>Click to expand!</summary>
```
<domain type='kvm' id='148'>
...
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static'>2</vcpu>
<cputune>
<shares>66</shares>
</cputune>
```
Schedinfo is already wrong:
```
Scheduler : posix
cpu_shares : 1024
vcpu_period : 100000
vcpu_quota : -1
...
```
Cgroup (or CFS) shares are set falsely set also (as one would expect):
```
# cat cpu.shares
1024
# pwd
/sys/fs/cgroup/cpu/machine.slice/machine-qemu\x2d148\x2di\x2d331\x2d2003\x2dVM.scope
```
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3923: cputune shares
in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
DennisKonrad commented on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-593413529
@weizhouapache Or maybe do you know a way to not have this cputune set at all? Looking at the code it is my understanding that it should not be set when limit_cpu is false
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] weizhouapache commented on issue #3923: cputune shares
in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-595670190
@DennisKonrad thanks for your info !
it is strange we do not configure cgroup in qemu.conf, but cpu_shares looks correctly.
I use ubuntu 18.04
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3923: cputune shares
in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
DennisKonrad commented on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-593456216
@weizhouapache I checked with
`virsh schedinfo [instance-name]`
what cpu_shares are really set (and double checked through looking into /sys/fs/cgroup/cpu/... )
and most of the VMs just have the default value of 1024.
Most of the values CS generates are way lower and probaly therefor discarded?! Just a few really get the value specified in the domain xml. Is this the same with your VMs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3923: cputune shares
in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
DennisKonrad commented on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-593866590
2 Cores * 1000 MHz / 10 = 200
In reality theres 66 set. Did I get you right on the calculation? @weizhouapache
```
<domain type='kvm' id='148'>
...
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static'>2</vcpu>
<cputune>
<shares>66</shares>
</cputune>
```
Schedinfo is already wrong:
```
Scheduler : posix
cpu_shares : 1024
vcpu_period : 100000
vcpu_quota : -1
...
```
Cgroup (or CFS) shares are set falsely set also (as one would expect):
```
# cat cpu.shares
1024
# pwd
/sys/fs/cgroup/cpu/machine.slice/machine-qemu\x2d148\x2di\x2d331\x2d2003\x2dVM.scope
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] DennisKonrad edited a comment on issue #3923: cputune
shares in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
DennisKonrad edited a comment on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-595672233
@weizhouapache Maybe it's not the OS but qemu or libvirt version. It looks like it's "right" way to configure it. But it could also be this worked some time before so we didn't notice. And after some update it stopped to write the shares to cgroup. Only speculation though
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3923: cputune shares
in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
DennisKonrad commented on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-593370805
@ustcweizhou @wido Did you maybe encounter an issue like this before?
We saw a lot of CPU steal on VMs and investigated the issue. The culprit is the cputune parameter described above that lead to the stealtime. After using a offering with "offer HA" the cputune is gone and the cpusteal also.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] weizhouapache commented on issue #3923: cputune shares
in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-595131036
> 2 Cores * 1000 MHz / 10 = 200
>
> In reality theres 66 set. Did I get you right on the calculation? @weizhouapache
>
> **EDIT**
> I found why the shares are not set correctly anymore. In our last update the qemu.conf got replaced and there were no controllers configured. After repairing that the values match beween XML, schedinfo and cgroup.
>
> Click to expand!
> ```
> <domain type='kvm' id='148'>
> ...
> <currentMemory unit='KiB'>4194304</currentMemory>
> <vcpu placement='static'>2</vcpu>
> <cputune>
> <shares>66</shares>
> </cputune>
> ```
>
> Schedinfo is already wrong:
>
> ```
> Scheduler : posix
> cpu_shares : 1024
> vcpu_period : 100000
> vcpu_quota : -1
> ...
> ```
>
> Cgroup (or CFS) shares are set falsely set also (as one would expect):
>
> ```
> # cat cpu.shares
> 1024
> # pwd
> /sys/fs/cgroup/cpu/machine.slice/machine-qemu\x2d148\x2di\x2d331\x2d2003\x2dVM.scope
> ```
@DennisKonrad could you give more details why controller is not configured and how you fixed it ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] weizhouapache commented on issue #3923: cputune shares
in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-593398639
@DennisKonrad
I have tested the offerings with HA and offering without HA, there is no difference with the vm xml definitions (cputune is defined with both offerings).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3923: cputune shares
in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
DennisKonrad commented on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-595667425
@weizhouapache The why is due to our way of provisioning the hosts. It does not seem CS related. Could be more documented maybe.
To solve the issue we just configured the cpu controllers in the qemu.conf on the hosts. There's already an example in the config file that you can adapt to your needs:
`cgroup_controllers = [ "cpu", "devices", "memory", "blkio", "cpuset", "cpuacct" ]`
To allow for the shares to be passed through corretly we needed to activate at least cpu and cpuacct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] weizhouapache commented on issue #3923: cputune shares
in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-595130041
> After all the share values are correctly set through libvirt/qemu the steal issue is gone.
> Also as there is a default value of shares set anyway it makes absolutely sense to set it regardless of the cpu cap setting.
good to know.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] DennisKonrad commented on issue #3923: cputune shares
in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
DennisKonrad commented on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-595129696
After all the share values are correctly set through libvirt/qemu the steal issue is gone.
Also as there is a default value of shares set anyway it makes absolutely sense to set it regardless of the cpu cap setting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] DennisKonrad edited a comment on issue #3923: cputune
shares in domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
DennisKonrad edited a comment on issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923#issuecomment-593866590
2 Cores * 1000 MHz / 10 = 200
In reality theres 66 set. Did I get you right on the calculation? @weizhouapache
**EDIT**
I found why the shares are not set correctly anymore. In our last update the qemu.conf got replaced and there were no controllers configured. After repairing that the values match beween XML, schedinfo and cgroup.
```
<domain type='kvm' id='148'>
...
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static'>2</vcpu>
<cputune>
<shares>66</shares>
</cputune>
```
Schedinfo is already wrong:
```
Scheduler : posix
cpu_shares : 1024
vcpu_period : 100000
vcpu_quota : -1
...
```
Cgroup (or CFS) shares are set falsely set also (as one would expect):
```
# cat cpu.shares
1024
# pwd
/sys/fs/cgroup/cpu/machine.slice/machine-qemu\x2d148\x2di\x2d331\x2d2003\x2dVM.scope
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [cloudstack] DennisKonrad closed issue #3923: cputune shares in
domain xml set without CPU cap enabled
Posted by GitBox <gi...@apache.org>.
DennisKonrad closed issue #3923: cputune shares in domain xml set without CPU cap enabled
URL: https://github.com/apache/cloudstack/issues/3923
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services