You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by remibergsma <gi...@git.apache.org> on 2015/08/23 21:02:34 UTC
[GitHub] cloudstack pull request: CLOUDSTACK-8443: Support CentOS 7 as KVM ...
GitHub user remibergsma opened a pull request:
https://github.com/apache/cloudstack/pull/731
CLOUDSTACK-8443: Support CentOS 7 as KVM hypervisor
This adds support for CentOS 7 as a KVM hypervisor. As discussed in the Jira issue, the root cause of the issue was the `cloudstack-setup-agent` script that added this line to `/etc/libvirt/qemu.conf`:
`cgroup_controllers=["cpu"]`
On CentOS 6 this 'works', but since by default no cgroups are mounted, this setting is ignored by `libvirtd`. On CentOS 7 is does not work, because the `cpu` and `cpuacct` are 'co-mounted'. This simpy means you cannot use one without the order. The config line above instructs `libvirtd` to only use the `cpu` cgroup, which it can't.
There was code added to make sure the `cpu` cgroup was not co-mounted. It did this with a shell script, but after a reboot it was gone. It wasn't reliable and not even a solution at all. I went ahead and removed the code, because the co-mounted cgroups work fine. The real issue was the config line we put in `/etc/libvirt/qemu.conf` so much better to fix that instead.
The default for `libvirtd` is to use only cgroups that are mounted. When you remove the 'cgroup_controllers' line, the default kicks in. According to the `qemu.conf` file, the default is:
`cgroup_controllers = ["cpu", "devices", "memory", "blkio", "cpuacct", "net_cls"]`
Then again, if it's not mounted, it's not used. So, that's a nice way to control it.
This works just fine for both RHEL/CentOS 6 and RHEL/CentOS 7.
Along the debugging, I came across some small issues here and there which I also fixed in this PR.
If you test this, be sure to build RPMs from this branch and install those on the KVM hypervisor.
At Schuberg Philis we run KVM hypervisors on CentOS 7 for months with these fixes. Now I took the time to fix the issues in CloudStack, rather than work around them.
@bhaisaab @karuturi could you please review this? Thanks!
Please note:
The next step is to fix CLOUDSTACK-8625 (Systemd profile for KVM Agent) as currently a sysvinit script is still used. Although it works, we do not have any systemd benefits. As it is a separate issue, I'll try to address it in a separate PR.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/remibergsma/cloudstack centos7-kvm
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/731.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #731
----
commit e8fcaa8736d4f870d28fdb8617402c78b0fe0a0a
Author: Remi Bergsma <gi...@remi.nl>
Date: 2015-08-20T20:31:55Z
CLOUDSTACK-8443: no need to specify cgroup_controllers
This setting works on CentOS 6 / RHEL 6 but does nothing, as
"cpu" cgroup is not mounted. On CentOS 7 / RHEL 7 systemd does
mount cgroups and "cpu" is co-mounted with "cpuacc". Hence, if
we specify "cpu" then this results in an error because it can
only use them both, or none.
By removing the setting, we rely on the default of qemu, which
is:
cgroup_controllers = ["cpu", "devices", "memory", "blkio", "cpuacct", "net_cls"]
Only if they are really mounted, they will be used. So, this will
work on both version 6 and 7.
commit 3ec82a598c6aed0f72fc15b35de32575168e40ab
Author: Remi Bergsma <gi...@remi.nl>
Date: 2015-08-20T20:51:53Z
CLOUDSTACK-8443: remove setting up cgroup_controllers
commit 8196cc82542b9b2038ae3c5db88358249b3ce5ad
Author: Remi Bergsma <gi...@remi.nl>
Date: 2015-08-20T20:53:31Z
CLOUDSTACK-8443: don't try to fix co-mounted cgroups
This didn't work well, as after a reboot you'd still have qemu
throwing errors. We'll make sure we can handle the co-mounted
cgroups.
commit 7d7778faa79823342c0468318ec26d8e3df84a9f
Author: Remi Bergsma <gi...@remi.nl>
Date: 2015-08-22T20:54:01Z
CLOUDSTACK-8443: display the right hypervisor type
commit 313988c76a9524159b8a12a569a818f0b715c14e
Author: Remi Bergsma <gi...@remi.nl>
Date: 2015-08-23T10:44:34Z
CLOUDSTACK-8443: mention the correct logfile
commit 3696fd37d34a3db962c6785530aefb3574dd874f
Author: Remi Bergsma <gi...@remi.nl>
Date: 2015-08-22T20:30:46Z
CLOUDSTACK-8443: detect CentOS 7.x as RHEL 7 alike
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8443: Support CentOS 7 as KVM ...
Posted by karuturi <gi...@git.apache.org>.
Github user karuturi commented on the pull request:
https://github.com/apache/cloudstack/pull/731#issuecomment-134502526
This is what I have done to test
installed a fresh setup with centos63 rpms on MS with 6.3 and KVM host with 6.3.
verified that systemvms, VR and user instances successfully started.
verified that cgroup_controllers=["cpu"] doesnt exists at host:/etc/libvirt/qemu.conf
installed a fresh setup with centos7 rpms on MS with 7 and KVM host with 7.
verified that systemvms, VR and user instances successfully started.
verified that cgroup_controllers=["cpu"] doesnt exists at host:/etc/libvirt/qemu.conf
:+1:
Thanks for the detailed description :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8443: Support CentOS 7 as KVM ...
Posted by remibergsma <gi...@git.apache.org>.
Github user remibergsma commented on the pull request:
https://github.com/apache/cloudstack/pull/731#issuecomment-134363088
@bhaisaab No worries, took me some time as well to understand how it is supposed to work and I'm also still learning.
In the Jira issue you mention two issues. The first:
```
org.libvirt.LibvirtException: Controller 'cpuacct' is not wanted, but 'cpu' is co-mounted: Invalid argument
```
This is due to the co-mount: both 'cpu' and 'cpuacct' are on one mount point, and our previous config instructs it to use only 'cpu'. So, the 'cpuacct' is `not wanted` by our config. This was solved in my PR by relying on the default.
The second issue:
```
unsupported configuration: CPU tuning is not available on this host
```
This issue also went away by updating the `libvirtd` config. When I look at the Libvirt source (https://github.com/virtualopensystems/libvirt/blob/master/src/qemu/qemu_cgroup.c#L635) I see this error is thrown in qemu_cgroup.c I assume the VIR_ERR_CONFIG_UNSUPPORTED aka `unsupported configuration` also has to do with the rather weird cgroup config that we put there before but I don't know for sure.
If you have some time, please give it a go. I'll also ask @wilderrodrigues to review it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8443: Support CentOS 7 as KVM ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/cloudstack/pull/731
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8443: Support CentOS 7 as KVM ...
Posted by remibergsma <gi...@git.apache.org>.
Github user remibergsma commented on the pull request:
https://github.com/apache/cloudstack/pull/731#issuecomment-134664222
@kishankavala Thanks, that would be great! Let me know if I can be of any help.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8443: Support CentOS 7 as KVM ...
Posted by bhaisaab <gi...@git.apache.org>.
Github user bhaisaab commented on the pull request:
https://github.com/apache/cloudstack/pull/731#issuecomment-134169583
just to clarify I've not tested this against a real env, just reviewed the code; there are few cgroup related things I don't completely grok but I'll invest time on them
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8443: Support CentOS 7 as KVM ...
Posted by bhaisaab <gi...@git.apache.org>.
Github user bhaisaab commented on the pull request:
https://github.com/apache/cloudstack/pull/731#issuecomment-134107765
@remibergsma looks good, but let me test it and get back to you on this; also for a small patch a single commit is preferred as multiple commits dilute the main motive of the fix when someone is browsing git history or referencing in future - 'Supporting CentOS7 as KVM hypervisor host'
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8443: Support CentOS 7 as KVM ...
Posted by remibergsma <gi...@git.apache.org>.
Github user remibergsma commented on the pull request:
https://github.com/apache/cloudstack/pull/731#issuecomment-134211424
Squashed some commits that belong to each other.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8443: Support CentOS 7 as KVM ...
Posted by bhaisaab <gi...@git.apache.org>.
Github user bhaisaab commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/731#discussion_r37855588
--- Diff: plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java ---
@@ -699,17 +698,6 @@ public boolean configure(final String name, final Map<String, Object> params) th
_hypervisorType = HypervisorType.KVM;
}
- //Verify that cpu,cpuacct cgroups are not co-mounted
- if(HypervisorType.LXC.equals(getHypervisorType())){
--- End diff --
@kishankavala can you review this patch, it removes some of the cgroup setting stuff for LXC?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8443: Support CentOS 7 as KVM ...
Posted by remibergsma <gi...@git.apache.org>.
Github user remibergsma commented on the pull request:
https://github.com/apache/cloudstack/pull/731#issuecomment-134523092
Thanks for testing and verifying @karuturi :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8443: Support CentOS 7 as KVM ...
Posted by kishankavala <gi...@git.apache.org>.
Github user kishankavala commented on the pull request:
https://github.com/apache/cloudstack/pull/731#issuecomment-134636230
@remibergsma changes looks fine. I would like to test it and get back to you, hopefully by tomorrow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8443: Support CentOS 7 as KVM ...
Posted by borisroman <gi...@git.apache.org>.
Github user borisroman commented on the pull request:
https://github.com/apache/cloudstack/pull/731#issuecomment-134071270
LGTM -> Deployed on Centos7 started 2 systemvms, 1 router and 1 guest vm without libvirt errors.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8443: Support CentOS 7 as KVM ...
Posted by remibergsma <gi...@git.apache.org>.
Github user remibergsma commented on the pull request:
https://github.com/apache/cloudstack/pull/731#issuecomment-134154979
@borisroman @bhaisaab Thanks for testing!
I'll have another look at the commits and see if it makes sense to quash some of them together (I'll make them atomic).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8443: Support CentOS 7 as KVM ...
Posted by remibergsma <gi...@git.apache.org>.
Github user remibergsma commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/731#discussion_r37874182
--- Diff: plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java ---
@@ -699,17 +698,6 @@ public boolean configure(final String name, final Map<String, Object> params) th
_hypervisorType = HypervisorType.KVM;
}
- //Verify that cpu,cpuacct cgroups are not co-mounted
- if(HypervisorType.LXC.equals(getHypervisorType())){
--- End diff --
@bhaisaab I just tested LXC and it works just fine with the co-mounted cgroups. The problem this code works-around has been solved. I think we're fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---