You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Diego Spinola Castro <sp...@gmail.com> on 2012/05/22 20:30:22 UTC
Vmware CPU Cap
I believe that is a bug with cpu cap and vmware.
To reproduce:
Create a offering with 2 cores and 1000mhz.
Enable CPU CAP.
After created instance , cs create a vm with 2 cores and 1000mhz of limit.
I don't know for sure if is a bug, but vmware gives 1000mhz shared with
cores.
Diego
Re: Vmware CPU Cap
Posted by Diego Spinola Castro <sp...@gmail.com>.
Any chance to apply on 2.2.x ?
2012/5/25 Edison Su <Ed...@citrix.com>
> Two bugs are been created:
> http://bugs.cloudstack.org/browse/CS-15095
> http://bugs.cloudstack.org/browse/CS-15094
> Let me fix it on the master branch, but not sure will it be pushed into
> next release(3.0.3).
>
> > -----Original Message-----
> > From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
> > Sent: Friday, May 25, 2012 5:07 AM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > Edison. What we are going to do? I believe this is a important path
> > inspite
> > of cs isnt using the right way of resource allocation.
> >
> > []'s
> > Diego Castro
> > Em 23/05/2012 19:29, "Edison Su" <Ed...@citrix.com> escreveu:
> >
> > > ...Sorry, I am not read your last email carefully...
> > > As I said before, in VMware, the limit parameter is set on the whole
> > VM,
> > > not per core. So we need to set the limit as " core_number*core_speed
> > ",
> > > which means
> > > the VM as a whole will not use as much as " core_number*core_speed "
> > > shares. It doesn't break the rule 2.
> > >
> > > > -----Original Message-----
> > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > Sent: Wednesday, May 23, 2012 3:20 PM
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > I'm afraid I don't follow.
> > > > I was talking about multiple core VMs and not single core ones.
> > > > I said if I have a 4 core VM with 3Ghz core speed and I run 4
> > cpuburn
> > > > threads I could get approximately the performance of a single core
> > at
> > > > 12Ghz.
> > > >
> > > > -----Original Message-----
> > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > Sent: 23 May 2012 23:06
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > Single core means there is only one scheduling entity, that
> > hypervisor
> > > > can schedule for the VM, so there is no parallel processing at all.
> > > > Hypervisor can not schedule a single core VM to multiple physical
> > cpu
> > > > cores at the same time.
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > Sent: Wednesday, May 23, 2012 2:59 PM
> > > > > To: cloudstack-dev@incubator.apache.org
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > Hi,
> > > > >
> > > > > Before we would touch that rule 2:
> > > > > While utilizing 12Ghz on a single core is not possible it is
> > possible
> > > > > to have effective 12Ghz on 4 cores at 3Ghz via parallel
> > processing.
> > > > > If I start 4 cpuburn threads on a quad core it will use all four
> > 3Ghz
> > > > > cores flat out "giving similar speed as 1 core@12Ghz would be"*.
> > As
> > > > > vmware uses the limit parameter as limit to the whole CPU
> > utilization
> > > > > rule 2 approach would prevent me effectively utilizing the CPU on
> > > > > vmware because if we say the CPU speed in Mhz cannot be larger
> > than
> > > > > the core speed I would be limited to an effective speed of core-
> > > > > speed/number_of_cores. This would mean 3000/4=750Mhz per core on
> > a
> > > > > 4vCPU VM. If you set "core_number*core_speed" as limit you
> > already
> > > > > broke rule 2 as the CPU is set higher than its core speed.
> > > > >
> > > > > I believe it good as it is now.
> > > > >
> > > > > Regards
> > > > >
> > > > > -----Original Message-----
> > > > > From: Prachi Damle [mailto:Prachi.Damle@citrix.com]
> > > > > Sent: 23 May 2012 22:36
> > > > > To: cloudstack-dev@incubator.apache.org
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > Yeah, rule#1 and #3 are considered by the current Host Allocator;
> > > > > rule#2 to compare just the speed is missing.
> > > > >
> > > > > Prachi
> > > > >
> > > > > -----Original Message-----
> > > > > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> > > > > Sent: Wednesday, May 23, 2012 2:06 PM
> > > > > To: cloudstack-dev@incubator.apache.org
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > Checked the code, I think we miss rule 2 in allocator.
> > > > >
> > > > > Anthony
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> > > > > > Sent: Wednesday, May 23, 2012 1:50 PM
> > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > I'm pretty sure we have that before we redesign allocator.
> > > > > >
> > > > > > Anthony
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > > > > > Sent: Wednesday, May 23, 2012 1:44 PM
> > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > I'm surprised. I definitely recall having that conversation
> > > > > > > before about never exceeding the physical limits of the
> > > > > > > hypervisor. I'm pretty sure we have test cases against that
> > as
> > > > well.
> > > > > > >
> > > > > > > --Alex
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > > Sent: Wednesday, May 23, 2012 1:39 PM
> > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > >
> > > > > > > > I think it's a general issue for all the hypervisors we
> > > > supported.
> > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > > > > > > > Sent: Wednesday, May 23, 2012 1:32 PM
> > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > >
> > > > > > > > > That's because no one wrote a host allocator for VmWare.
> > > > > > > > >
> > > > > > > > > The relationship between DeploymentPlanner and
> > HostAllocator
> > > > > and
> > > > > > > > > StoragePoolAllocator is as follows:
> > > > > > > > >
> > > > > > > > > DeploymentPlanner deals with heuristics and affinity
> > rules of
> > > > > > > placing
> > > > > > > > > a VM. Once it determines a set of hosts that matches, it
> > > > then
> > > > > > asks
> > > > > > > > > the HostAllocator to work on the limitations of the type
> > of
> > > > > > > hypervisor.
> > > > > > > > > There was never a HostAllocator written for VmWare. It
> > just
> > > > > > > > > uses
> > > > > > > the
> > > > > > > > > one that was written originally for XenServer. Hence the
> > bug.
> > > > > > > > >
> > > > > > > > > There's similar differentiation for DeploymentPlanner and
> > > > > > > > > StoragePoolAllocator and similar bugs exists, especially
> > if
> > > > > > someone
> > > > > > > > > adds a completely new type of storage pool but decides to
> > > > just
> > > > > > > reuse
> > > > > > > > > the current set of StoragePoolAllocators.
> > > > > > > > >
> > > > > > > > > --Alex
> > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > > > > Sent: Wednesday, May 23, 2012 1:24 PM
> > > > > > > > > > To: cloudstack-dev@incubator.apache.org;
> > > > 'tamasm@veber.co.uk'
> > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > >
> > > > > > > > > > Thanks for your input, I find another bug in
> > cloudstack...
> > > > > > > > > > Let's dive little bit deeper into how the CPU
> > allocation
> > > > > works.
> > > > > > > > > > VMware(the same for other hypervisors(Xen/KVM)) uses
> > > > > > proportional
> > > > > > > > > share
> > > > > > > > > > based scheduling algorithm([1],[2]). It means "If a
> > virtual
> > > > > > > machine
> > > > > > > > > has twice
> > > > > > > > > > as many shares of a resource as another virtual machine,
> > it
> > > > > is
> > > > > > > > > entitled to
> > > > > > > > > > consume twice as much of that resource when these two
> > > > > > > > > > virtual
> > > > > > > > > machines
> > > > > > > > > > are competing for resources."
> > > > > > > > > > There are two questions:
> > > > > > > > > > 1. Where is the share coming from? In Cloudstack, the
> > share
> > > > > is
> > > > > > > > > calculated
> > > > > > > > > > from (CPU MHz * CPU cores).
> > > > > > > > > > 2. How the share of a VM is mapped to the physical CPU
> > by
> > > > > > > > > > the
> > > > > > > > > hypervisor?
> > > > > > > > > > Of cause, it depends on the hypervisor implementation,
> > but
> > > > > > there
> > > > > > > are
> > > > > > > > > some
> > > > > > > > > > general rules that we need to follow, as we are living
> > in
> > > > > > > > > > the
> > > > > > > real
> > > > > > > > > world:)
> > > > > > > > > > Rule 1: num of cpu core of a VM should be <= num of
> > cpu
> > > > > > > > > > core
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > hypervisor host. It doesn't make sense to running a 12
> > core
> > > > > VM
> > > > > > on
> > > > > > > a
> > > > > > > > > single
> > > > > > > > > > core host, even some hypervisors, such as KVM, can do
> > that.
> > > > > > > > > Hypervisor will
> > > > > > > > > > be busy at scheduling VCPU(the VCPU context switch is
> > much
> > > > > > > heavier
> > > > > > > > > than
> > > > > > > > > > process context switch), thus bad performance for the
> > VM.
> > > > > > > > > > Rule 2: The frequency of a VCPU should be <= the
> > > > > > > > > > frequency
> > > > > > of
> > > > > > > the
> > > > > > > > > host
> > > > > > > > > > hypervisor host. Can you run an one core * 12Ghz VM on
> > a
> > > > > > > > > > 3Ghz
> > > > > > > > > > *
> > > > > > 4
> > > > > > > > > core
> > > > > > > > > > physical CPU? Nope, in an unit time, the max freq of
> > VCPU
> > > > > > > > > > can
> > > > > > get
> > > > > > > is
> > > > > > > > > the max
> > > > > > > > > > freq of physical cpu core, that's the physical LAW.
> > > > > > > > > > Rule 3: the total share of VMs <= total share of
> > host
> > > > > > > > > hypervisor(in case of
> > > > > > > > > > no CPU overcommit).
> > > > > > > > > >
> > > > > > > > > > Currently, in cloudstack host allocator, only rule 3 is
> > > > > > > > > > taken
> > > > > > > into
> > > > > > > > > consideration.
> > > > > > > > > >
> > > > > > > > > > [1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM-
> > > > > > resource.pdf
> > > > > > > > > > [2]
> > > > > > > > > >
> > > > > > > >
> > http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf
> > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > > > Sent: Tuesday, May 22, 2012 5:41 PM
> > > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > I still don't think this is an issue as the CPU Mhz
> > limit
> > > > > > > > > > > and
> > > > > > > the
> > > > > > > > > > > number of cores are independent.
> > > > > > > > > > > CPU manufacturers sell 2,4,6 cores at 3Ghz and not
> > > > > > > > > > > 6,12,18Ghz
> > > > > > > CPUs.
> > > > > > > > > > >
> > > > > > > > > > > So I think it is good how it works but the
> > > > > > > "number_of_cores*Mhz"
> > > > > > > > > while
> > > > > > > > > > > allocating should not multiply so that is the bug :)
> > What
> > > > > > > vmware
> > > > > > > > > does
> > > > > > > > > > > with multiplying the cores with the core speed is bad
> > as
> > > > I
> > > > > > > can't
> > > > > > > > > have
> > > > > > > > > > > a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know
> > where
> > > > > > > > > > > I'm
> > > > > > > > > coming
> > > > > > > > > > > from....
> > > > > > > > > > >
> > > > > > > > > > > Regards
> > > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > > > Sent: 23 May 2012 01:27
> > > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > I have confused myself too because if I have a look
> > the
> > > > > > > database
> > > > > > > > > and
> > > > > > > > > > > dump the service_offerings table the limit for me is
> > set
> > > > > > > > > > > to
> > > > > > > > > > > 2000,3000,4000 and not 1000.
> > > > > > > > > > > So a Mhz limit set to 4000 and 4 cores will end up as
> > a
> > > > > quad
> > > > > > > core
> > > > > > > > > box
> > > > > > > > > > > at 4Ghz. I remember now I had to set CPU
> > overprovisioning
> > > > > to
> > > > > > 4
> > > > > > > as
> > > > > > > > > > > CloudStack took away 4x4Ghz=16Ghz of the available
> > CPU.
> > > > > > > > > > > So Cloudstack sets the VM CPU Mhz limit what the
> > actual
> > > > > > > > > > > limit
> > > > > > > is
> > > > > > > > > set
> > > > > > > > > > > to in the offering but does not multiply the Mhz
> > limit by
> > > > > > > > > > > the
> > > > > > > > > number
> > > > > > > > > > > of cores when setting the limit on vmware. However
> > takes
> > > > > > > > > > > away "number_of_cores*Mhz limit" from the available
> > CPU
> > > > > > > > > > > capacity
> > > > > > > when
> > > > > > > > > > > allocating.
> > > > > > > > > > >
> > > > > > > > > > > I'm getting confused myself so I'm not sure if this
> > is
> > > > bug
> > > > > > now
> > > > > > > or
> > > > > > > > > not
> > > > > > > > > > > either :) I'm using 3.0.3
> > > > > > > > > > >
> > > > > > > > > > > Regards
> > > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > > > > > Sent: 23 May 2012 00:32
> > > > > > > > > > > To: cloudstack-dev@incubator.apache.org; Tamas Monos
> > > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > > >
> > > > > > > > > > > But in Diego'case, the limit is set as 1000Mhz, while
> > his
> > > > > > > service
> > > > > > > > > > > offering is 1000Mhz * 2 cores.
> > > > > > > > > > > Which version of cloudstack are you using? Maybe it's
> > a
> > > > > > > regression.
> > > > > > > > > > >
> > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > > > > Sent: Tuesday, May 22, 2012 4:26 PM
> > > > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > I have a service offering with 1000Mhz limit and 4
> > cpu
> > > > > > cores.
> > > > > > > > > That
> > > > > > > > > > > > sums up to 4000Mhz.
> > > > > > > > > > > > Cloudstack sets the limit to 4Ghz on the virtual
> > > > machine
> > > > > > and
> > > > > > > > > > > > when
> > > > > > > > > I
> > > > > > > > > > > > put load on it vmware balances the load between the
> > 4
> > > > > > > > > > > > cores
> > > > > > > > > allowing
> > > > > > > > > > > > them to use 1000Mhz each.
> > > > > > > > > > > > I do not see any bugs here.
> > > > > > > > > > > >
> > > > > > > > > > > > Apologies if the "meant per CPU core" was incorrect.
> > > > > > > > > > > > What I meant
> > > > > > > > > is
> > > > > > > > > > > > described above.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards
> > > > > > > > > > > >
> > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > > > > > > Sent: 23 May 2012 00:05
> > > > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > > > >
> > > > > > > > > > > > Nope, from the document, the limit is set on whole
> > vm:
> > > > > > > > > > > > CPU Limits
> > > > > > > > > > > >
> > > > > > > > > > > > When a CPU Limit is set on a virtual machine
> > resource
> > > > > > > settings,
> > > > > > > > > the
> > > > > > > > > > > > virtual machine is deliberately held from being
> > > > > > > > > > > > scheduled
> > > > > > to
> > > > > > > a
> > > > > > > > > PCPU
> > > > > > > > > > > > when it has used up its allocated CPU resource.
> > This
> > > > > > happens
> > > > > > > > > > > > regardless of the CPU utilization. If the limit is
> > set
> > > > > > > > > > > > to 500MHz, the virtual machine is descheduled from
> > the
> > > > > > > > > > > > PCPU
> > > > > > and
> > > > > > > has
> > > > > > > > > > > > to wait before
> > > > > > > > > > > it
> > > > > > > > > > > > is allowed to be scheduled again. As such, the
> > virtual
> > > > > > > machine
> > > > > > > > > might
> > > > > > > > > > > > experience performance degradation.
> > > > > > > > > > > >
> > > > > > > > > > > > Note: For an SMP virtual machine, the sum of all
> > vCPUs
> > > > > > cannot
> > > > > > > > > exceed
> > > > > > > > > > > > the specified limit. For example, 4 vCPU virtual
> > > > machine
> > > > > > with
> > > > > > > a
> > > > > > > > > > > > limit of 1200MHz and equal load among vCPUs would
> > > > result
> > > > > > > > > > > > in
> > > > > > a
> > > > > > > > > > > > max
> > > > > > > > > of
> > > > > > > > > > > > 300MHz per vCPU.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&pop
> > > > > > > > > > u
> > > > > > > > > > p
> > > > > > > > > > > > =
> > > > > > > > > > > > true&languageId=&externalID=1033115
> > > > > > > > > > > >
> > > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > > > > > Sent: Tuesday, May 22, 2012 3:43 PM
> > > > > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > >
> > > > > > > > > > > > > No I don't think this is a bug. When you set
> > 1000Mhz
> > > > > > > > > > > > > as
> > > > > > CPU
> > > > > > > > > > > > > cap
> > > > > > > > > > > that
> > > > > > > > > > > > > is meant per core. vmWare will limit each CPU
> > core to
> > > > > > > 1000Mhz.
> > > > > > > > > > > > > As you gave 2 CPU cores that is 2000Mhz effective.
> > > > > > > > > > > > > That
> > > > > > is
> > > > > > > how
> > > > > > > > > > > > > vmware works.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have setup my offerings all to 1000Mhz as speed
> > and
> > > > > > just
> > > > > > > > > > > > > increasing the number of cores.
> > > > > > > > > > > > > 1 core ends up being 1x1000Mhz
> > > > > > > > > > > > > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > > > > > > > > > > > > ...
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm actually using it in production and works
> > quite
> > > > > well
> > > > > > as
> > > > > > > > > > > > Cloudstack
> > > > > > > > > > > > > "allocates" 4000Mhz when I'm using 4x1000Mhz
> > cores
> > > > and
> > > > > > > vmware
> > > > > > > > > > > > cleverly
> > > > > > > > > > > > > balances between the cores as you put load on it
> > and
> > > > > not
> > > > > > > > > letting
> > > > > > > > > > > any
> > > > > > > > > > > > > of the cores above the set limit of 1000Mhz.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards
> > > > > > > > > > > > >
> > > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > > From: Diego Spinola Castro
> > > > > > [mailto:spinolacastro@gmail.com]
> > > > > > > > > > > > > Sent: 22 May 2012 19:34
> > > > > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > > > > Subject: Re: Vmware CPU Cap
> > > > > > > > > > > > >
> > > > > > > > > > > > > I forgot the link:
> > > > > > > > > > > > > http://imageshack.us/photo/my-
> > > > images/848/cpulimit.png/
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2012/5/22 Diego Spinola Castro
> > > > > <sp...@gmail.com>
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I believe that is a bug with cpu cap and vmware.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > To reproduce:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Create a offering with 2 cores and 1000mhz.
> > > > > > > > > > > > > > Enable CPU CAP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > After created instance , cs create a vm with 2
> > > > cores
> > > > > > and
> > > > > > > > > 1000mhz
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > limit.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I don't know for sure if is a bug, but vmware
> > gives
> > > > > > > 1000mhz
> > > > > > > > > > > shared
> > > > > > > > > > > > > > with cores.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Diego
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
>
RE: Vmware CPU Cap
Posted by Edison Su <Ed...@citrix.com>.
Two bugs are been created:
http://bugs.cloudstack.org/browse/CS-15095
http://bugs.cloudstack.org/browse/CS-15094
Let me fix it on the master branch, but not sure will it be pushed into next release(3.0.3).
> -----Original Message-----
> From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
> Sent: Friday, May 25, 2012 5:07 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Edison. What we are going to do? I believe this is a important path
> inspite
> of cs isnt using the right way of resource allocation.
>
> []'s
> Diego Castro
> Em 23/05/2012 19:29, "Edison Su" <Ed...@citrix.com> escreveu:
>
> > ...Sorry, I am not read your last email carefully...
> > As I said before, in VMware, the limit parameter is set on the whole
> VM,
> > not per core. So we need to set the limit as " core_number*core_speed
> ",
> > which means
> > the VM as a whole will not use as much as " core_number*core_speed "
> > shares. It doesn't break the rule 2.
> >
> > > -----Original Message-----
> > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > Sent: Wednesday, May 23, 2012 3:20 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > I'm afraid I don't follow.
> > > I was talking about multiple core VMs and not single core ones.
> > > I said if I have a 4 core VM with 3Ghz core speed and I run 4
> cpuburn
> > > threads I could get approximately the performance of a single core
> at
> > > 12Ghz.
> > >
> > > -----Original Message-----
> > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > Sent: 23 May 2012 23:06
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > Single core means there is only one scheduling entity, that
> hypervisor
> > > can schedule for the VM, so there is no parallel processing at all.
> > > Hypervisor can not schedule a single core VM to multiple physical
> cpu
> > > cores at the same time.
> > >
> > >
> > > > -----Original Message-----
> > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > Sent: Wednesday, May 23, 2012 2:59 PM
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > Hi,
> > > >
> > > > Before we would touch that rule 2:
> > > > While utilizing 12Ghz on a single core is not possible it is
> possible
> > > > to have effective 12Ghz on 4 cores at 3Ghz via parallel
> processing.
> > > > If I start 4 cpuburn threads on a quad core it will use all four
> 3Ghz
> > > > cores flat out "giving similar speed as 1 core@12Ghz would be"*.
> As
> > > > vmware uses the limit parameter as limit to the whole CPU
> utilization
> > > > rule 2 approach would prevent me effectively utilizing the CPU on
> > > > vmware because if we say the CPU speed in Mhz cannot be larger
> than
> > > > the core speed I would be limited to an effective speed of core-
> > > > speed/number_of_cores. This would mean 3000/4=750Mhz per core on
> a
> > > > 4vCPU VM. If you set "core_number*core_speed" as limit you
> already
> > > > broke rule 2 as the CPU is set higher than its core speed.
> > > >
> > > > I believe it good as it is now.
> > > >
> > > > Regards
> > > >
> > > > -----Original Message-----
> > > > From: Prachi Damle [mailto:Prachi.Damle@citrix.com]
> > > > Sent: 23 May 2012 22:36
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > Yeah, rule#1 and #3 are considered by the current Host Allocator;
> > > > rule#2 to compare just the speed is missing.
> > > >
> > > > Prachi
> > > >
> > > > -----Original Message-----
> > > > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> > > > Sent: Wednesday, May 23, 2012 2:06 PM
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > Checked the code, I think we miss rule 2 in allocator.
> > > >
> > > > Anthony
> > > >
> > > > > -----Original Message-----
> > > > > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> > > > > Sent: Wednesday, May 23, 2012 1:50 PM
> > > > > To: cloudstack-dev@incubator.apache.org
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > I'm pretty sure we have that before we redesign allocator.
> > > > >
> > > > > Anthony
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > > > > Sent: Wednesday, May 23, 2012 1:44 PM
> > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > I'm surprised. I definitely recall having that conversation
> > > > > > before about never exceeding the physical limits of the
> > > > > > hypervisor. I'm pretty sure we have test cases against that
> as
> > > well.
> > > > > >
> > > > > > --Alex
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > Sent: Wednesday, May 23, 2012 1:39 PM
> > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > I think it's a general issue for all the hypervisors we
> > > supported.
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > > > > > > Sent: Wednesday, May 23, 2012 1:32 PM
> > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > >
> > > > > > > > That's because no one wrote a host allocator for VmWare.
> > > > > > > >
> > > > > > > > The relationship between DeploymentPlanner and
> HostAllocator
> > > > and
> > > > > > > > StoragePoolAllocator is as follows:
> > > > > > > >
> > > > > > > > DeploymentPlanner deals with heuristics and affinity
> rules of
> > > > > > placing
> > > > > > > > a VM. Once it determines a set of hosts that matches, it
> > > then
> > > > > asks
> > > > > > > > the HostAllocator to work on the limitations of the type
> of
> > > > > > hypervisor.
> > > > > > > > There was never a HostAllocator written for VmWare. It
> just
> > > > > > > > uses
> > > > > > the
> > > > > > > > one that was written originally for XenServer. Hence the
> bug.
> > > > > > > >
> > > > > > > > There's similar differentiation for DeploymentPlanner and
> > > > > > > > StoragePoolAllocator and similar bugs exists, especially
> if
> > > > > someone
> > > > > > > > adds a completely new type of storage pool but decides to
> > > just
> > > > > > reuse
> > > > > > > > the current set of StoragePoolAllocators.
> > > > > > > >
> > > > > > > > --Alex
> > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > > > Sent: Wednesday, May 23, 2012 1:24 PM
> > > > > > > > > To: cloudstack-dev@incubator.apache.org;
> > > 'tamasm@veber.co.uk'
> > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > >
> > > > > > > > > Thanks for your input, I find another bug in
> cloudstack...
> > > > > > > > > Let's dive little bit deeper into how the CPU
> allocation
> > > > works.
> > > > > > > > > VMware(the same for other hypervisors(Xen/KVM)) uses
> > > > > proportional
> > > > > > > > share
> > > > > > > > > based scheduling algorithm([1],[2]). It means "If a
> virtual
> > > > > > machine
> > > > > > > > has twice
> > > > > > > > > as many shares of a resource as another virtual machine,
> it
> > > > is
> > > > > > > > entitled to
> > > > > > > > > consume twice as much of that resource when these two
> > > > > > > > > virtual
> > > > > > > > machines
> > > > > > > > > are competing for resources."
> > > > > > > > > There are two questions:
> > > > > > > > > 1. Where is the share coming from? In Cloudstack, the
> share
> > > > is
> > > > > > > > calculated
> > > > > > > > > from (CPU MHz * CPU cores).
> > > > > > > > > 2. How the share of a VM is mapped to the physical CPU
> by
> > > > > > > > > the
> > > > > > > > hypervisor?
> > > > > > > > > Of cause, it depends on the hypervisor implementation,
> but
> > > > > there
> > > > > > are
> > > > > > > > some
> > > > > > > > > general rules that we need to follow, as we are living
> in
> > > > > > > > > the
> > > > > > real
> > > > > > > > world:)
> > > > > > > > > Rule 1: num of cpu core of a VM should be <= num of
> cpu
> > > > > > > > > core
> > > > > > of
> > > > > > > > the
> > > > > > > > > hypervisor host. It doesn't make sense to running a 12
> core
> > > > VM
> > > > > on
> > > > > > a
> > > > > > > > single
> > > > > > > > > core host, even some hypervisors, such as KVM, can do
> that.
> > > > > > > > Hypervisor will
> > > > > > > > > be busy at scheduling VCPU(the VCPU context switch is
> much
> > > > > > heavier
> > > > > > > > than
> > > > > > > > > process context switch), thus bad performance for the
> VM.
> > > > > > > > > Rule 2: The frequency of a VCPU should be <= the
> > > > > > > > > frequency
> > > > > of
> > > > > > the
> > > > > > > > host
> > > > > > > > > hypervisor host. Can you run an one core * 12Ghz VM on
> a
> > > > > > > > > 3Ghz
> > > > > > > > > *
> > > > > 4
> > > > > > > > core
> > > > > > > > > physical CPU? Nope, in an unit time, the max freq of
> VCPU
> > > > > > > > > can
> > > > > get
> > > > > > is
> > > > > > > > the max
> > > > > > > > > freq of physical cpu core, that's the physical LAW.
> > > > > > > > > Rule 3: the total share of VMs <= total share of
> host
> > > > > > > > hypervisor(in case of
> > > > > > > > > no CPU overcommit).
> > > > > > > > >
> > > > > > > > > Currently, in cloudstack host allocator, only rule 3 is
> > > > > > > > > taken
> > > > > > into
> > > > > > > > consideration.
> > > > > > > > >
> > > > > > > > > [1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM-
> > > > > resource.pdf
> > > > > > > > > [2]
> > > > > > > > >
> > > > > > >
> http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf
> > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > > Sent: Tuesday, May 22, 2012 5:41 PM
> > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I still don't think this is an issue as the CPU Mhz
> limit
> > > > > > > > > > and
> > > > > > the
> > > > > > > > > > number of cores are independent.
> > > > > > > > > > CPU manufacturers sell 2,4,6 cores at 3Ghz and not
> > > > > > > > > > 6,12,18Ghz
> > > > > > CPUs.
> > > > > > > > > >
> > > > > > > > > > So I think it is good how it works but the
> > > > > > "number_of_cores*Mhz"
> > > > > > > > while
> > > > > > > > > > allocating should not multiply so that is the bug :)
> What
> > > > > > vmware
> > > > > > > > does
> > > > > > > > > > with multiplying the cores with the core speed is bad
> as
> > > I
> > > > > > can't
> > > > > > > > have
> > > > > > > > > > a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know
> where
> > > > > > > > > > I'm
> > > > > > > > coming
> > > > > > > > > > from....
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > > Sent: 23 May 2012 01:27
> > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I have confused myself too because if I have a look
> the
> > > > > > database
> > > > > > > > and
> > > > > > > > > > dump the service_offerings table the limit for me is
> set
> > > > > > > > > > to
> > > > > > > > > > 2000,3000,4000 and not 1000.
> > > > > > > > > > So a Mhz limit set to 4000 and 4 cores will end up as
> a
> > > > quad
> > > > > > core
> > > > > > > > box
> > > > > > > > > > at 4Ghz. I remember now I had to set CPU
> overprovisioning
> > > > to
> > > > > 4
> > > > > > as
> > > > > > > > > > CloudStack took away 4x4Ghz=16Ghz of the available
> CPU.
> > > > > > > > > > So Cloudstack sets the VM CPU Mhz limit what the
> actual
> > > > > > > > > > limit
> > > > > > is
> > > > > > > > set
> > > > > > > > > > to in the offering but does not multiply the Mhz
> limit by
> > > > > > > > > > the
> > > > > > > > number
> > > > > > > > > > of cores when setting the limit on vmware. However
> takes
> > > > > > > > > > away "number_of_cores*Mhz limit" from the available
> CPU
> > > > > > > > > > capacity
> > > > > > when
> > > > > > > > > > allocating.
> > > > > > > > > >
> > > > > > > > > > I'm getting confused myself so I'm not sure if this
> is
> > > bug
> > > > > now
> > > > > > or
> > > > > > > > not
> > > > > > > > > > either :) I'm using 3.0.3
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > > > > Sent: 23 May 2012 00:32
> > > > > > > > > > To: cloudstack-dev@incubator.apache.org; Tamas Monos
> > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > >
> > > > > > > > > > But in Diego'case, the limit is set as 1000Mhz, while
> his
> > > > > > service
> > > > > > > > > > offering is 1000Mhz * 2 cores.
> > > > > > > > > > Which version of cloudstack are you using? Maybe it's
> a
> > > > > > regression.
> > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > > > Sent: Tuesday, May 22, 2012 4:26 PM
> > > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > I have a service offering with 1000Mhz limit and 4
> cpu
> > > > > cores.
> > > > > > > > That
> > > > > > > > > > > sums up to 4000Mhz.
> > > > > > > > > > > Cloudstack sets the limit to 4Ghz on the virtual
> > > machine
> > > > > and
> > > > > > > > > > > when
> > > > > > > > I
> > > > > > > > > > > put load on it vmware balances the load between the
> 4
> > > > > > > > > > > cores
> > > > > > > > allowing
> > > > > > > > > > > them to use 1000Mhz each.
> > > > > > > > > > > I do not see any bugs here.
> > > > > > > > > > >
> > > > > > > > > > > Apologies if the "meant per CPU core" was incorrect.
> > > > > > > > > > > What I meant
> > > > > > > > is
> > > > > > > > > > > described above.
> > > > > > > > > > >
> > > > > > > > > > > Regards
> > > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > > > > > Sent: 23 May 2012 00:05
> > > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > > >
> > > > > > > > > > > Nope, from the document, the limit is set on whole
> vm:
> > > > > > > > > > > CPU Limits
> > > > > > > > > > >
> > > > > > > > > > > When a CPU Limit is set on a virtual machine
> resource
> > > > > > settings,
> > > > > > > > the
> > > > > > > > > > > virtual machine is deliberately held from being
> > > > > > > > > > > scheduled
> > > > > to
> > > > > > a
> > > > > > > > PCPU
> > > > > > > > > > > when it has used up its allocated CPU resource.
> This
> > > > > happens
> > > > > > > > > > > regardless of the CPU utilization. If the limit is
> set
> > > > > > > > > > > to 500MHz, the virtual machine is descheduled from
> the
> > > > > > > > > > > PCPU
> > > > > and
> > > > > > has
> > > > > > > > > > > to wait before
> > > > > > > > > > it
> > > > > > > > > > > is allowed to be scheduled again. As such, the
> virtual
> > > > > > machine
> > > > > > > > might
> > > > > > > > > > > experience performance degradation.
> > > > > > > > > > >
> > > > > > > > > > > Note: For an SMP virtual machine, the sum of all
> vCPUs
> > > > > cannot
> > > > > > > > exceed
> > > > > > > > > > > the specified limit. For example, 4 vCPU virtual
> > > machine
> > > > > with
> > > > > > a
> > > > > > > > > > > limit of 1200MHz and equal load among vCPUs would
> > > result
> > > > > > > > > > > in
> > > > > a
> > > > > > > > > > > max
> > > > > > > > of
> > > > > > > > > > > 300MHz per vCPU.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&pop
> > > > > > > > > u
> > > > > > > > > p
> > > > > > > > > > > =
> > > > > > > > > > > true&languageId=&externalID=1033115
> > > > > > > > > > >
> > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > > > > Sent: Tuesday, May 22, 2012 3:43 PM
> > > > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > No I don't think this is a bug. When you set
> 1000Mhz
> > > > > > > > > > > > as
> > > > > CPU
> > > > > > > > > > > > cap
> > > > > > > > > > that
> > > > > > > > > > > > is meant per core. vmWare will limit each CPU
> core to
> > > > > > 1000Mhz.
> > > > > > > > > > > > As you gave 2 CPU cores that is 2000Mhz effective.
> > > > > > > > > > > > That
> > > > > is
> > > > > > how
> > > > > > > > > > > > vmware works.
> > > > > > > > > > > >
> > > > > > > > > > > > I have setup my offerings all to 1000Mhz as speed
> and
> > > > > just
> > > > > > > > > > > > increasing the number of cores.
> > > > > > > > > > > > 1 core ends up being 1x1000Mhz
> > > > > > > > > > > > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > > > > > > > > > > > ...
> > > > > > > > > > > >
> > > > > > > > > > > > I'm actually using it in production and works
> quite
> > > > well
> > > > > as
> > > > > > > > > > > Cloudstack
> > > > > > > > > > > > "allocates" 4000Mhz when I'm using 4x1000Mhz
> cores
> > > and
> > > > > > vmware
> > > > > > > > > > > cleverly
> > > > > > > > > > > > balances between the cores as you put load on it
> and
> > > > not
> > > > > > > > letting
> > > > > > > > > > any
> > > > > > > > > > > > of the cores above the set limit of 1000Mhz.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Regards
> > > > > > > > > > > >
> > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > From: Diego Spinola Castro
> > > > > [mailto:spinolacastro@gmail.com]
> > > > > > > > > > > > Sent: 22 May 2012 19:34
> > > > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > > > Subject: Re: Vmware CPU Cap
> > > > > > > > > > > >
> > > > > > > > > > > > I forgot the link:
> > > > > > > > > > > > http://imageshack.us/photo/my-
> > > images/848/cpulimit.png/
> > > > > > > > > > > >
> > > > > > > > > > > > 2012/5/22 Diego Spinola Castro
> > > > <sp...@gmail.com>
> > > > > > > > > > > >
> > > > > > > > > > > > > I believe that is a bug with cpu cap and vmware.
> > > > > > > > > > > > >
> > > > > > > > > > > > > To reproduce:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Create a offering with 2 cores and 1000mhz.
> > > > > > > > > > > > > Enable CPU CAP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > After created instance , cs create a vm with 2
> > > cores
> > > > > and
> > > > > > > > 1000mhz
> > > > > > > > > > > > > of
> > > > > > > > > > > > limit.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I don't know for sure if is a bug, but vmware
> gives
> > > > > > 1000mhz
> > > > > > > > > > shared
> > > > > > > > > > > > > with cores.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Diego
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > >
> > > >
> > >
> > >
> >
> >
RE: Vmware CPU Cap
Posted by Diego Spinola Castro <sp...@gmail.com>.
Edison. What we are going to do? I believe this is a important path inspite
of cs isnt using the right way of resource allocation.
[]'s
Diego Castro
Em 23/05/2012 19:29, "Edison Su" <Ed...@citrix.com> escreveu:
> ...Sorry, I am not read your last email carefully...
> As I said before, in VMware, the limit parameter is set on the whole VM,
> not per core. So we need to set the limit as " core_number*core_speed ",
> which means
> the VM as a whole will not use as much as " core_number*core_speed "
> shares. It doesn't break the rule 2.
>
> > -----Original Message-----
> > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > Sent: Wednesday, May 23, 2012 3:20 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > I'm afraid I don't follow.
> > I was talking about multiple core VMs and not single core ones.
> > I said if I have a 4 core VM with 3Ghz core speed and I run 4 cpuburn
> > threads I could get approximately the performance of a single core at
> > 12Ghz.
> >
> > -----Original Message-----
> > From: Edison Su [mailto:Edison.su@citrix.com]
> > Sent: 23 May 2012 23:06
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > Single core means there is only one scheduling entity, that hypervisor
> > can schedule for the VM, so there is no parallel processing at all.
> > Hypervisor can not schedule a single core VM to multiple physical cpu
> > cores at the same time.
> >
> >
> > > -----Original Message-----
> > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > Sent: Wednesday, May 23, 2012 2:59 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > Hi,
> > >
> > > Before we would touch that rule 2:
> > > While utilizing 12Ghz on a single core is not possible it is possible
> > > to have effective 12Ghz on 4 cores at 3Ghz via parallel processing.
> > > If I start 4 cpuburn threads on a quad core it will use all four 3Ghz
> > > cores flat out "giving similar speed as 1 core@12Ghz would be"*. As
> > > vmware uses the limit parameter as limit to the whole CPU utilization
> > > rule 2 approach would prevent me effectively utilizing the CPU on
> > > vmware because if we say the CPU speed in Mhz cannot be larger than
> > > the core speed I would be limited to an effective speed of core-
> > > speed/number_of_cores. This would mean 3000/4=750Mhz per core on a
> > > 4vCPU VM. If you set "core_number*core_speed" as limit you already
> > > broke rule 2 as the CPU is set higher than its core speed.
> > >
> > > I believe it good as it is now.
> > >
> > > Regards
> > >
> > > -----Original Message-----
> > > From: Prachi Damle [mailto:Prachi.Damle@citrix.com]
> > > Sent: 23 May 2012 22:36
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > Yeah, rule#1 and #3 are considered by the current Host Allocator;
> > > rule#2 to compare just the speed is missing.
> > >
> > > Prachi
> > >
> > > -----Original Message-----
> > > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> > > Sent: Wednesday, May 23, 2012 2:06 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > Checked the code, I think we miss rule 2 in allocator.
> > >
> > > Anthony
> > >
> > > > -----Original Message-----
> > > > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> > > > Sent: Wednesday, May 23, 2012 1:50 PM
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > I'm pretty sure we have that before we redesign allocator.
> > > >
> > > > Anthony
> > > >
> > > > > -----Original Message-----
> > > > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > > > Sent: Wednesday, May 23, 2012 1:44 PM
> > > > > To: cloudstack-dev@incubator.apache.org
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > I'm surprised. I definitely recall having that conversation
> > > > > before about never exceeding the physical limits of the
> > > > > hypervisor. I'm pretty sure we have test cases against that as
> > well.
> > > > >
> > > > > --Alex
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > Sent: Wednesday, May 23, 2012 1:39 PM
> > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > I think it's a general issue for all the hypervisors we
> > supported.
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > > > > > Sent: Wednesday, May 23, 2012 1:32 PM
> > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > That's because no one wrote a host allocator for VmWare.
> > > > > > >
> > > > > > > The relationship between DeploymentPlanner and HostAllocator
> > > and
> > > > > > > StoragePoolAllocator is as follows:
> > > > > > >
> > > > > > > DeploymentPlanner deals with heuristics and affinity rules of
> > > > > placing
> > > > > > > a VM. Once it determines a set of hosts that matches, it
> > then
> > > > asks
> > > > > > > the HostAllocator to work on the limitations of the type of
> > > > > hypervisor.
> > > > > > > There was never a HostAllocator written for VmWare. It just
> > > > > > > uses
> > > > > the
> > > > > > > one that was written originally for XenServer. Hence the bug.
> > > > > > >
> > > > > > > There's similar differentiation for DeploymentPlanner and
> > > > > > > StoragePoolAllocator and similar bugs exists, especially if
> > > > someone
> > > > > > > adds a completely new type of storage pool but decides to
> > just
> > > > > reuse
> > > > > > > the current set of StoragePoolAllocators.
> > > > > > >
> > > > > > > --Alex
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > > Sent: Wednesday, May 23, 2012 1:24 PM
> > > > > > > > To: cloudstack-dev@incubator.apache.org;
> > 'tamasm@veber.co.uk'
> > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > >
> > > > > > > > Thanks for your input, I find another bug in cloudstack...
> > > > > > > > Let's dive little bit deeper into how the CPU allocation
> > > works.
> > > > > > > > VMware(the same for other hypervisors(Xen/KVM)) uses
> > > > proportional
> > > > > > > share
> > > > > > > > based scheduling algorithm([1],[2]). It means "If a virtual
> > > > > machine
> > > > > > > has twice
> > > > > > > > as many shares of a resource as another virtual machine, it
> > > is
> > > > > > > entitled to
> > > > > > > > consume twice as much of that resource when these two
> > > > > > > > virtual
> > > > > > > machines
> > > > > > > > are competing for resources."
> > > > > > > > There are two questions:
> > > > > > > > 1. Where is the share coming from? In Cloudstack, the share
> > > is
> > > > > > > calculated
> > > > > > > > from (CPU MHz * CPU cores).
> > > > > > > > 2. How the share of a VM is mapped to the physical CPU by
> > > > > > > > the
> > > > > > > hypervisor?
> > > > > > > > Of cause, it depends on the hypervisor implementation, but
> > > > there
> > > > > are
> > > > > > > some
> > > > > > > > general rules that we need to follow, as we are living in
> > > > > > > > the
> > > > > real
> > > > > > > world:)
> > > > > > > > Rule 1: num of cpu core of a VM should be <= num of cpu
> > > > > > > > core
> > > > > of
> > > > > > > the
> > > > > > > > hypervisor host. It doesn't make sense to running a 12 core
> > > VM
> > > > on
> > > > > a
> > > > > > > single
> > > > > > > > core host, even some hypervisors, such as KVM, can do that.
> > > > > > > Hypervisor will
> > > > > > > > be busy at scheduling VCPU(the VCPU context switch is much
> > > > > heavier
> > > > > > > than
> > > > > > > > process context switch), thus bad performance for the VM.
> > > > > > > > Rule 2: The frequency of a VCPU should be <= the
> > > > > > > > frequency
> > > > of
> > > > > the
> > > > > > > host
> > > > > > > > hypervisor host. Can you run an one core * 12Ghz VM on a
> > > > > > > > 3Ghz
> > > > > > > > *
> > > > 4
> > > > > > > core
> > > > > > > > physical CPU? Nope, in an unit time, the max freq of VCPU
> > > > > > > > can
> > > > get
> > > > > is
> > > > > > > the max
> > > > > > > > freq of physical cpu core, that's the physical LAW.
> > > > > > > > Rule 3: the total share of VMs <= total share of host
> > > > > > > hypervisor(in case of
> > > > > > > > no CPU overcommit).
> > > > > > > >
> > > > > > > > Currently, in cloudstack host allocator, only rule 3 is
> > > > > > > > taken
> > > > > into
> > > > > > > consideration.
> > > > > > > >
> > > > > > > > [1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM-
> > > > resource.pdf
> > > > > > > > [2]
> > > > > > > >
> > > > > > http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf
> > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > Sent: Tuesday, May 22, 2012 5:41 PM
> > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I still don't think this is an issue as the CPU Mhz limit
> > > > > > > > > and
> > > > > the
> > > > > > > > > number of cores are independent.
> > > > > > > > > CPU manufacturers sell 2,4,6 cores at 3Ghz and not
> > > > > > > > > 6,12,18Ghz
> > > > > CPUs.
> > > > > > > > >
> > > > > > > > > So I think it is good how it works but the
> > > > > "number_of_cores*Mhz"
> > > > > > > while
> > > > > > > > > allocating should not multiply so that is the bug :) What
> > > > > vmware
> > > > > > > does
> > > > > > > > > with multiplying the cores with the core speed is bad as
> > I
> > > > > can't
> > > > > > > have
> > > > > > > > > a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know where
> > > > > > > > > I'm
> > > > > > > coming
> > > > > > > > > from....
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > Sent: 23 May 2012 01:27
> > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I have confused myself too because if I have a look the
> > > > > database
> > > > > > > and
> > > > > > > > > dump the service_offerings table the limit for me is set
> > > > > > > > > to
> > > > > > > > > 2000,3000,4000 and not 1000.
> > > > > > > > > So a Mhz limit set to 4000 and 4 cores will end up as a
> > > quad
> > > > > core
> > > > > > > box
> > > > > > > > > at 4Ghz. I remember now I had to set CPU overprovisioning
> > > to
> > > > 4
> > > > > as
> > > > > > > > > CloudStack took away 4x4Ghz=16Ghz of the available CPU.
> > > > > > > > > So Cloudstack sets the VM CPU Mhz limit what the actual
> > > > > > > > > limit
> > > > > is
> > > > > > > set
> > > > > > > > > to in the offering but does not multiply the Mhz limit by
> > > > > > > > > the
> > > > > > > number
> > > > > > > > > of cores when setting the limit on vmware. However takes
> > > > > > > > > away "number_of_cores*Mhz limit" from the available CPU
> > > > > > > > > capacity
> > > > > when
> > > > > > > > > allocating.
> > > > > > > > >
> > > > > > > > > I'm getting confused myself so I'm not sure if this is
> > bug
> > > > now
> > > > > or
> > > > > > > not
> > > > > > > > > either :) I'm using 3.0.3
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > > > Sent: 23 May 2012 00:32
> > > > > > > > > To: cloudstack-dev@incubator.apache.org; Tamas Monos
> > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > >
> > > > > > > > > But in Diego'case, the limit is set as 1000Mhz, while his
> > > > > service
> > > > > > > > > offering is 1000Mhz * 2 cores.
> > > > > > > > > Which version of cloudstack are you using? Maybe it's a
> > > > > regression.
> > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > > Sent: Tuesday, May 22, 2012 4:26 PM
> > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I have a service offering with 1000Mhz limit and 4 cpu
> > > > cores.
> > > > > > > That
> > > > > > > > > > sums up to 4000Mhz.
> > > > > > > > > > Cloudstack sets the limit to 4Ghz on the virtual
> > machine
> > > > and
> > > > > > > > > > when
> > > > > > > I
> > > > > > > > > > put load on it vmware balances the load between the 4
> > > > > > > > > > cores
> > > > > > > allowing
> > > > > > > > > > them to use 1000Mhz each.
> > > > > > > > > > I do not see any bugs here.
> > > > > > > > > >
> > > > > > > > > > Apologies if the "meant per CPU core" was incorrect.
> > > > > > > > > > What I meant
> > > > > > > is
> > > > > > > > > > described above.
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > > > > Sent: 23 May 2012 00:05
> > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > >
> > > > > > > > > > Nope, from the document, the limit is set on whole vm:
> > > > > > > > > > CPU Limits
> > > > > > > > > >
> > > > > > > > > > When a CPU Limit is set on a virtual machine resource
> > > > > settings,
> > > > > > > the
> > > > > > > > > > virtual machine is deliberately held from being
> > > > > > > > > > scheduled
> > > > to
> > > > > a
> > > > > > > PCPU
> > > > > > > > > > when it has used up its allocated CPU resource. This
> > > > happens
> > > > > > > > > > regardless of the CPU utilization. If the limit is set
> > > > > > > > > > to 500MHz, the virtual machine is descheduled from the
> > > > > > > > > > PCPU
> > > > and
> > > > > has
> > > > > > > > > > to wait before
> > > > > > > > > it
> > > > > > > > > > is allowed to be scheduled again. As such, the virtual
> > > > > machine
> > > > > > > might
> > > > > > > > > > experience performance degradation.
> > > > > > > > > >
> > > > > > > > > > Note: For an SMP virtual machine, the sum of all vCPUs
> > > > cannot
> > > > > > > exceed
> > > > > > > > > > the specified limit. For example, 4 vCPU virtual
> > machine
> > > > with
> > > > > a
> > > > > > > > > > limit of 1200MHz and equal load among vCPUs would
> > result
> > > > > > > > > > in
> > > > a
> > > > > > > > > > max
> > > > > > > of
> > > > > > > > > > 300MHz per vCPU.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&pop
> > > > > > > > u
> > > > > > > > p
> > > > > > > > > > =
> > > > > > > > > > true&languageId=&externalID=1033115
> > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > > > Sent: Tuesday, May 22, 2012 3:43 PM
> > > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > No I don't think this is a bug. When you set 1000Mhz
> > > > > > > > > > > as
> > > > CPU
> > > > > > > > > > > cap
> > > > > > > > > that
> > > > > > > > > > > is meant per core. vmWare will limit each CPU core to
> > > > > 1000Mhz.
> > > > > > > > > > > As you gave 2 CPU cores that is 2000Mhz effective.
> > > > > > > > > > > That
> > > > is
> > > > > how
> > > > > > > > > > > vmware works.
> > > > > > > > > > >
> > > > > > > > > > > I have setup my offerings all to 1000Mhz as speed and
> > > > just
> > > > > > > > > > > increasing the number of cores.
> > > > > > > > > > > 1 core ends up being 1x1000Mhz
> > > > > > > > > > > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > > > > > > > > > > ...
> > > > > > > > > > >
> > > > > > > > > > > I'm actually using it in production and works quite
> > > well
> > > > as
> > > > > > > > > > Cloudstack
> > > > > > > > > > > "allocates" 4000Mhz when I'm using 4x1000Mhz cores
> > and
> > > > > vmware
> > > > > > > > > > cleverly
> > > > > > > > > > > balances between the cores as you put load on it and
> > > not
> > > > > > > letting
> > > > > > > > > any
> > > > > > > > > > > of the cores above the set limit of 1000Mhz.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Regards
> > > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: Diego Spinola Castro
> > > > [mailto:spinolacastro@gmail.com]
> > > > > > > > > > > Sent: 22 May 2012 19:34
> > > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > > Subject: Re: Vmware CPU Cap
> > > > > > > > > > >
> > > > > > > > > > > I forgot the link:
> > > > > > > > > > > http://imageshack.us/photo/my-
> > images/848/cpulimit.png/
> > > > > > > > > > >
> > > > > > > > > > > 2012/5/22 Diego Spinola Castro
> > > <sp...@gmail.com>
> > > > > > > > > > >
> > > > > > > > > > > > I believe that is a bug with cpu cap and vmware.
> > > > > > > > > > > >
> > > > > > > > > > > > To reproduce:
> > > > > > > > > > > >
> > > > > > > > > > > > Create a offering with 2 cores and 1000mhz.
> > > > > > > > > > > > Enable CPU CAP.
> > > > > > > > > > > >
> > > > > > > > > > > > After created instance , cs create a vm with 2
> > cores
> > > > and
> > > > > > > 1000mhz
> > > > > > > > > > > > of
> > > > > > > > > > > limit.
> > > > > > > > > > > >
> > > > > > > > > > > > I don't know for sure if is a bug, but vmware gives
> > > > > 1000mhz
> > > > > > > > > shared
> > > > > > > > > > > > with cores.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Diego
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > >
> > >
> >
> >
>
>
RE: Vmware CPU Cap
Posted by Edison Su <Ed...@citrix.com>.
...Sorry, I am not read your last email carefully...
As I said before, in VMware, the limit parameter is set on the whole VM, not per core. So we need to set the limit as " core_number*core_speed ", which means
the VM as a whole will not use as much as " core_number*core_speed " shares. It doesn't break the rule 2.
> -----Original Message-----
> From: Tamas Monos [mailto:tamasm@veber.co.uk]
> Sent: Wednesday, May 23, 2012 3:20 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> I'm afraid I don't follow.
> I was talking about multiple core VMs and not single core ones.
> I said if I have a 4 core VM with 3Ghz core speed and I run 4 cpuburn
> threads I could get approximately the performance of a single core at
> 12Ghz.
>
> -----Original Message-----
> From: Edison Su [mailto:Edison.su@citrix.com]
> Sent: 23 May 2012 23:06
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Single core means there is only one scheduling entity, that hypervisor
> can schedule for the VM, so there is no parallel processing at all.
> Hypervisor can not schedule a single core VM to multiple physical cpu
> cores at the same time.
>
>
> > -----Original Message-----
> > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > Sent: Wednesday, May 23, 2012 2:59 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > Hi,
> >
> > Before we would touch that rule 2:
> > While utilizing 12Ghz on a single core is not possible it is possible
> > to have effective 12Ghz on 4 cores at 3Ghz via parallel processing.
> > If I start 4 cpuburn threads on a quad core it will use all four 3Ghz
> > cores flat out "giving similar speed as 1 core@12Ghz would be"*. As
> > vmware uses the limit parameter as limit to the whole CPU utilization
> > rule 2 approach would prevent me effectively utilizing the CPU on
> > vmware because if we say the CPU speed in Mhz cannot be larger than
> > the core speed I would be limited to an effective speed of core-
> > speed/number_of_cores. This would mean 3000/4=750Mhz per core on a
> > 4vCPU VM. If you set "core_number*core_speed" as limit you already
> > broke rule 2 as the CPU is set higher than its core speed.
> >
> > I believe it good as it is now.
> >
> > Regards
> >
> > -----Original Message-----
> > From: Prachi Damle [mailto:Prachi.Damle@citrix.com]
> > Sent: 23 May 2012 22:36
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > Yeah, rule#1 and #3 are considered by the current Host Allocator;
> > rule#2 to compare just the speed is missing.
> >
> > Prachi
> >
> > -----Original Message-----
> > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> > Sent: Wednesday, May 23, 2012 2:06 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > Checked the code, I think we miss rule 2 in allocator.
> >
> > Anthony
> >
> > > -----Original Message-----
> > > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> > > Sent: Wednesday, May 23, 2012 1:50 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > I'm pretty sure we have that before we redesign allocator.
> > >
> > > Anthony
> > >
> > > > -----Original Message-----
> > > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > > Sent: Wednesday, May 23, 2012 1:44 PM
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > I'm surprised. I definitely recall having that conversation
> > > > before about never exceeding the physical limits of the
> > > > hypervisor. I'm pretty sure we have test cases against that as
> well.
> > > >
> > > > --Alex
> > > >
> > > > > -----Original Message-----
> > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > Sent: Wednesday, May 23, 2012 1:39 PM
> > > > > To: cloudstack-dev@incubator.apache.org
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > I think it's a general issue for all the hypervisors we
> supported.
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > > > > Sent: Wednesday, May 23, 2012 1:32 PM
> > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > That's because no one wrote a host allocator for VmWare.
> > > > > >
> > > > > > The relationship between DeploymentPlanner and HostAllocator
> > and
> > > > > > StoragePoolAllocator is as follows:
> > > > > >
> > > > > > DeploymentPlanner deals with heuristics and affinity rules of
> > > > placing
> > > > > > a VM. Once it determines a set of hosts that matches, it
> then
> > > asks
> > > > > > the HostAllocator to work on the limitations of the type of
> > > > hypervisor.
> > > > > > There was never a HostAllocator written for VmWare. It just
> > > > > > uses
> > > > the
> > > > > > one that was written originally for XenServer. Hence the bug.
> > > > > >
> > > > > > There's similar differentiation for DeploymentPlanner and
> > > > > > StoragePoolAllocator and similar bugs exists, especially if
> > > someone
> > > > > > adds a completely new type of storage pool but decides to
> just
> > > > reuse
> > > > > > the current set of StoragePoolAllocators.
> > > > > >
> > > > > > --Alex
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > Sent: Wednesday, May 23, 2012 1:24 PM
> > > > > > > To: cloudstack-dev@incubator.apache.org;
> 'tamasm@veber.co.uk'
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > Thanks for your input, I find another bug in cloudstack...
> > > > > > > Let's dive little bit deeper into how the CPU allocation
> > works.
> > > > > > > VMware(the same for other hypervisors(Xen/KVM)) uses
> > > proportional
> > > > > > share
> > > > > > > based scheduling algorithm([1],[2]). It means "If a virtual
> > > > machine
> > > > > > has twice
> > > > > > > as many shares of a resource as another virtual machine, it
> > is
> > > > > > entitled to
> > > > > > > consume twice as much of that resource when these two
> > > > > > > virtual
> > > > > > machines
> > > > > > > are competing for resources."
> > > > > > > There are two questions:
> > > > > > > 1. Where is the share coming from? In Cloudstack, the share
> > is
> > > > > > calculated
> > > > > > > from (CPU MHz * CPU cores).
> > > > > > > 2. How the share of a VM is mapped to the physical CPU by
> > > > > > > the
> > > > > > hypervisor?
> > > > > > > Of cause, it depends on the hypervisor implementation, but
> > > there
> > > > are
> > > > > > some
> > > > > > > general rules that we need to follow, as we are living in
> > > > > > > the
> > > > real
> > > > > > world:)
> > > > > > > Rule 1: num of cpu core of a VM should be <= num of cpu
> > > > > > > core
> > > > of
> > > > > > the
> > > > > > > hypervisor host. It doesn't make sense to running a 12 core
> > VM
> > > on
> > > > a
> > > > > > single
> > > > > > > core host, even some hypervisors, such as KVM, can do that.
> > > > > > Hypervisor will
> > > > > > > be busy at scheduling VCPU(the VCPU context switch is much
> > > > heavier
> > > > > > than
> > > > > > > process context switch), thus bad performance for the VM.
> > > > > > > Rule 2: The frequency of a VCPU should be <= the
> > > > > > > frequency
> > > of
> > > > the
> > > > > > host
> > > > > > > hypervisor host. Can you run an one core * 12Ghz VM on a
> > > > > > > 3Ghz
> > > > > > > *
> > > 4
> > > > > > core
> > > > > > > physical CPU? Nope, in an unit time, the max freq of VCPU
> > > > > > > can
> > > get
> > > > is
> > > > > > the max
> > > > > > > freq of physical cpu core, that's the physical LAW.
> > > > > > > Rule 3: the total share of VMs <= total share of host
> > > > > > hypervisor(in case of
> > > > > > > no CPU overcommit).
> > > > > > >
> > > > > > > Currently, in cloudstack host allocator, only rule 3 is
> > > > > > > taken
> > > > into
> > > > > > consideration.
> > > > > > >
> > > > > > > [1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM-
> > > resource.pdf
> > > > > > > [2]
> > > > > > >
> > > > > http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > Sent: Tuesday, May 22, 2012 5:41 PM
> > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I still don't think this is an issue as the CPU Mhz limit
> > > > > > > > and
> > > > the
> > > > > > > > number of cores are independent.
> > > > > > > > CPU manufacturers sell 2,4,6 cores at 3Ghz and not
> > > > > > > > 6,12,18Ghz
> > > > CPUs.
> > > > > > > >
> > > > > > > > So I think it is good how it works but the
> > > > "number_of_cores*Mhz"
> > > > > > while
> > > > > > > > allocating should not multiply so that is the bug :) What
> > > > vmware
> > > > > > does
> > > > > > > > with multiplying the cores with the core speed is bad as
> I
> > > > can't
> > > > > > have
> > > > > > > > a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know where
> > > > > > > > I'm
> > > > > > coming
> > > > > > > > from....
> > > > > > > >
> > > > > > > > Regards
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > Sent: 23 May 2012 01:27
> > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I have confused myself too because if I have a look the
> > > > database
> > > > > > and
> > > > > > > > dump the service_offerings table the limit for me is set
> > > > > > > > to
> > > > > > > > 2000,3000,4000 and not 1000.
> > > > > > > > So a Mhz limit set to 4000 and 4 cores will end up as a
> > quad
> > > > core
> > > > > > box
> > > > > > > > at 4Ghz. I remember now I had to set CPU overprovisioning
> > to
> > > 4
> > > > as
> > > > > > > > CloudStack took away 4x4Ghz=16Ghz of the available CPU.
> > > > > > > > So Cloudstack sets the VM CPU Mhz limit what the actual
> > > > > > > > limit
> > > > is
> > > > > > set
> > > > > > > > to in the offering but does not multiply the Mhz limit by
> > > > > > > > the
> > > > > > number
> > > > > > > > of cores when setting the limit on vmware. However takes
> > > > > > > > away "number_of_cores*Mhz limit" from the available CPU
> > > > > > > > capacity
> > > > when
> > > > > > > > allocating.
> > > > > > > >
> > > > > > > > I'm getting confused myself so I'm not sure if this is
> bug
> > > now
> > > > or
> > > > > > not
> > > > > > > > either :) I'm using 3.0.3
> > > > > > > >
> > > > > > > > Regards
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > > Sent: 23 May 2012 00:32
> > > > > > > > To: cloudstack-dev@incubator.apache.org; Tamas Monos
> > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > >
> > > > > > > > But in Diego'case, the limit is set as 1000Mhz, while his
> > > > service
> > > > > > > > offering is 1000Mhz * 2 cores.
> > > > > > > > Which version of cloudstack are you using? Maybe it's a
> > > > regression.
> > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > Sent: Tuesday, May 22, 2012 4:26 PM
> > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I have a service offering with 1000Mhz limit and 4 cpu
> > > cores.
> > > > > > That
> > > > > > > > > sums up to 4000Mhz.
> > > > > > > > > Cloudstack sets the limit to 4Ghz on the virtual
> machine
> > > and
> > > > > > > > > when
> > > > > > I
> > > > > > > > > put load on it vmware balances the load between the 4
> > > > > > > > > cores
> > > > > > allowing
> > > > > > > > > them to use 1000Mhz each.
> > > > > > > > > I do not see any bugs here.
> > > > > > > > >
> > > > > > > > > Apologies if the "meant per CPU core" was incorrect.
> > > > > > > > > What I meant
> > > > > > is
> > > > > > > > > described above.
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > > > Sent: 23 May 2012 00:05
> > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > >
> > > > > > > > > Nope, from the document, the limit is set on whole vm:
> > > > > > > > > CPU Limits
> > > > > > > > >
> > > > > > > > > When a CPU Limit is set on a virtual machine resource
> > > > settings,
> > > > > > the
> > > > > > > > > virtual machine is deliberately held from being
> > > > > > > > > scheduled
> > > to
> > > > a
> > > > > > PCPU
> > > > > > > > > when it has used up its allocated CPU resource. This
> > > happens
> > > > > > > > > regardless of the CPU utilization. If the limit is set
> > > > > > > > > to 500MHz, the virtual machine is descheduled from the
> > > > > > > > > PCPU
> > > and
> > > > has
> > > > > > > > > to wait before
> > > > > > > > it
> > > > > > > > > is allowed to be scheduled again. As such, the virtual
> > > > machine
> > > > > > might
> > > > > > > > > experience performance degradation.
> > > > > > > > >
> > > > > > > > > Note: For an SMP virtual machine, the sum of all vCPUs
> > > cannot
> > > > > > exceed
> > > > > > > > > the specified limit. For example, 4 vCPU virtual
> machine
> > > with
> > > > a
> > > > > > > > > limit of 1200MHz and equal load among vCPUs would
> result
> > > > > > > > > in
> > > a
> > > > > > > > > max
> > > > > > of
> > > > > > > > > 300MHz per vCPU.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&pop
> > > > > > > u
> > > > > > > p
> > > > > > > > > =
> > > > > > > > > true&languageId=&externalID=1033115
> > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > > Sent: Tuesday, May 22, 2012 3:43 PM
> > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > No I don't think this is a bug. When you set 1000Mhz
> > > > > > > > > > as
> > > CPU
> > > > > > > > > > cap
> > > > > > > > that
> > > > > > > > > > is meant per core. vmWare will limit each CPU core to
> > > > 1000Mhz.
> > > > > > > > > > As you gave 2 CPU cores that is 2000Mhz effective.
> > > > > > > > > > That
> > > is
> > > > how
> > > > > > > > > > vmware works.
> > > > > > > > > >
> > > > > > > > > > I have setup my offerings all to 1000Mhz as speed and
> > > just
> > > > > > > > > > increasing the number of cores.
> > > > > > > > > > 1 core ends up being 1x1000Mhz
> > > > > > > > > > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > > > > > > > > > ...
> > > > > > > > > >
> > > > > > > > > > I'm actually using it in production and works quite
> > well
> > > as
> > > > > > > > > Cloudstack
> > > > > > > > > > "allocates" 4000Mhz when I'm using 4x1000Mhz cores
> and
> > > > vmware
> > > > > > > > > cleverly
> > > > > > > > > > balances between the cores as you put load on it and
> > not
> > > > > > letting
> > > > > > > > any
> > > > > > > > > > of the cores above the set limit of 1000Mhz.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Diego Spinola Castro
> > > [mailto:spinolacastro@gmail.com]
> > > > > > > > > > Sent: 22 May 2012 19:34
> > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > Subject: Re: Vmware CPU Cap
> > > > > > > > > >
> > > > > > > > > > I forgot the link:
> > > > > > > > > > http://imageshack.us/photo/my-
> images/848/cpulimit.png/
> > > > > > > > > >
> > > > > > > > > > 2012/5/22 Diego Spinola Castro
> > <sp...@gmail.com>
> > > > > > > > > >
> > > > > > > > > > > I believe that is a bug with cpu cap and vmware.
> > > > > > > > > > >
> > > > > > > > > > > To reproduce:
> > > > > > > > > > >
> > > > > > > > > > > Create a offering with 2 cores and 1000mhz.
> > > > > > > > > > > Enable CPU CAP.
> > > > > > > > > > >
> > > > > > > > > > > After created instance , cs create a vm with 2
> cores
> > > and
> > > > > > 1000mhz
> > > > > > > > > > > of
> > > > > > > > > > limit.
> > > > > > > > > > >
> > > > > > > > > > > I don't know for sure if is a bug, but vmware gives
> > > > 1000mhz
> > > > > > > > shared
> > > > > > > > > > > with cores.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Diego
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> >
> >
>
>
RE: Vmware CPU Cap
Posted by Tamas Monos <ta...@veber.co.uk>.
I'm afraid I don't follow.
I was talking about multiple core VMs and not single core ones.
I said if I have a 4 core VM with 3Ghz core speed and I run 4 cpuburn threads I could get approximately the performance of a single core at 12Ghz.
-----Original Message-----
From: Edison Su [mailto:Edison.su@citrix.com]
Sent: 23 May 2012 23:06
To: cloudstack-dev@incubator.apache.org
Subject: RE: Vmware CPU Cap
Single core means there is only one scheduling entity, that hypervisor can schedule for the VM, so there is no parallel processing at all. Hypervisor can not schedule a single core VM to multiple physical cpu cores at the same time.
> -----Original Message-----
> From: Tamas Monos [mailto:tamasm@veber.co.uk]
> Sent: Wednesday, May 23, 2012 2:59 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Hi,
>
> Before we would touch that rule 2:
> While utilizing 12Ghz on a single core is not possible it is possible
> to have effective 12Ghz on 4 cores at 3Ghz via parallel processing.
> If I start 4 cpuburn threads on a quad core it will use all four 3Ghz
> cores flat out "giving similar speed as 1 core@12Ghz would be"*. As
> vmware uses the limit parameter as limit to the whole CPU utilization
> rule 2 approach would prevent me effectively utilizing the CPU on
> vmware because if we say the CPU speed in Mhz cannot be larger than
> the core speed I would be limited to an effective speed of core-
> speed/number_of_cores. This would mean 3000/4=750Mhz per core on a
> 4vCPU VM. If you set "core_number*core_speed" as limit you already
> broke rule 2 as the CPU is set higher than its core speed.
>
> I believe it good as it is now.
>
> Regards
>
> -----Original Message-----
> From: Prachi Damle [mailto:Prachi.Damle@citrix.com]
> Sent: 23 May 2012 22:36
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Yeah, rule#1 and #3 are considered by the current Host Allocator;
> rule#2 to compare just the speed is missing.
>
> Prachi
>
> -----Original Message-----
> From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> Sent: Wednesday, May 23, 2012 2:06 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Checked the code, I think we miss rule 2 in allocator.
>
> Anthony
>
> > -----Original Message-----
> > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> > Sent: Wednesday, May 23, 2012 1:50 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > I'm pretty sure we have that before we redesign allocator.
> >
> > Anthony
> >
> > > -----Original Message-----
> > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > Sent: Wednesday, May 23, 2012 1:44 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > I'm surprised. I definitely recall having that conversation
> > > before about never exceeding the physical limits of the
> > > hypervisor. I'm pretty sure we have test cases against that as well.
> > >
> > > --Alex
> > >
> > > > -----Original Message-----
> > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > Sent: Wednesday, May 23, 2012 1:39 PM
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > I think it's a general issue for all the hypervisors we supported.
> > > >
> > > > > -----Original Message-----
> > > > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > > > Sent: Wednesday, May 23, 2012 1:32 PM
> > > > > To: cloudstack-dev@incubator.apache.org
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > That's because no one wrote a host allocator for VmWare.
> > > > >
> > > > > The relationship between DeploymentPlanner and HostAllocator
> and
> > > > > StoragePoolAllocator is as follows:
> > > > >
> > > > > DeploymentPlanner deals with heuristics and affinity rules of
> > > placing
> > > > > a VM. Once it determines a set of hosts that matches, it then
> > asks
> > > > > the HostAllocator to work on the limitations of the type of
> > > hypervisor.
> > > > > There was never a HostAllocator written for VmWare. It just
> > > > > uses
> > > the
> > > > > one that was written originally for XenServer. Hence the bug.
> > > > >
> > > > > There's similar differentiation for DeploymentPlanner and
> > > > > StoragePoolAllocator and similar bugs exists, especially if
> > someone
> > > > > adds a completely new type of storage pool but decides to just
> > > reuse
> > > > > the current set of StoragePoolAllocators.
> > > > >
> > > > > --Alex
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > Sent: Wednesday, May 23, 2012 1:24 PM
> > > > > > To: cloudstack-dev@incubator.apache.org; 'tamasm@veber.co.uk'
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > Thanks for your input, I find another bug in cloudstack...
> > > > > > Let's dive little bit deeper into how the CPU allocation
> works.
> > > > > > VMware(the same for other hypervisors(Xen/KVM)) uses
> > proportional
> > > > > share
> > > > > > based scheduling algorithm([1],[2]). It means "If a virtual
> > > machine
> > > > > has twice
> > > > > > as many shares of a resource as another virtual machine, it
> is
> > > > > entitled to
> > > > > > consume twice as much of that resource when these two
> > > > > > virtual
> > > > > machines
> > > > > > are competing for resources."
> > > > > > There are two questions:
> > > > > > 1. Where is the share coming from? In Cloudstack, the share
> is
> > > > > calculated
> > > > > > from (CPU MHz * CPU cores).
> > > > > > 2. How the share of a VM is mapped to the physical CPU by
> > > > > > the
> > > > > hypervisor?
> > > > > > Of cause, it depends on the hypervisor implementation, but
> > there
> > > are
> > > > > some
> > > > > > general rules that we need to follow, as we are living in
> > > > > > the
> > > real
> > > > > world:)
> > > > > > Rule 1: num of cpu core of a VM should be <= num of cpu
> > > > > > core
> > > of
> > > > > the
> > > > > > hypervisor host. It doesn't make sense to running a 12 core
> VM
> > on
> > > a
> > > > > single
> > > > > > core host, even some hypervisors, such as KVM, can do that.
> > > > > Hypervisor will
> > > > > > be busy at scheduling VCPU(the VCPU context switch is much
> > > heavier
> > > > > than
> > > > > > process context switch), thus bad performance for the VM.
> > > > > > Rule 2: The frequency of a VCPU should be <= the
> > > > > > frequency
> > of
> > > the
> > > > > host
> > > > > > hypervisor host. Can you run an one core * 12Ghz VM on a
> > > > > > 3Ghz
> > > > > > *
> > 4
> > > > > core
> > > > > > physical CPU? Nope, in an unit time, the max freq of VCPU
> > > > > > can
> > get
> > > is
> > > > > the max
> > > > > > freq of physical cpu core, that's the physical LAW.
> > > > > > Rule 3: the total share of VMs <= total share of host
> > > > > hypervisor(in case of
> > > > > > no CPU overcommit).
> > > > > >
> > > > > > Currently, in cloudstack host allocator, only rule 3 is
> > > > > > taken
> > > into
> > > > > consideration.
> > > > > >
> > > > > > [1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM-
> > resource.pdf
> > > > > > [2]
> > > > > >
> > > > http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > Sent: Tuesday, May 22, 2012 5:41 PM
> > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I still don't think this is an issue as the CPU Mhz limit
> > > > > > > and
> > > the
> > > > > > > number of cores are independent.
> > > > > > > CPU manufacturers sell 2,4,6 cores at 3Ghz and not
> > > > > > > 6,12,18Ghz
> > > CPUs.
> > > > > > >
> > > > > > > So I think it is good how it works but the
> > > "number_of_cores*Mhz"
> > > > > while
> > > > > > > allocating should not multiply so that is the bug :) What
> > > vmware
> > > > > does
> > > > > > > with multiplying the cores with the core speed is bad as I
> > > can't
> > > > > have
> > > > > > > a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know where
> > > > > > > I'm
> > > > > coming
> > > > > > > from....
> > > > > > >
> > > > > > > Regards
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > Sent: 23 May 2012 01:27
> > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I have confused myself too because if I have a look the
> > > database
> > > > > and
> > > > > > > dump the service_offerings table the limit for me is set
> > > > > > > to
> > > > > > > 2000,3000,4000 and not 1000.
> > > > > > > So a Mhz limit set to 4000 and 4 cores will end up as a
> quad
> > > core
> > > > > box
> > > > > > > at 4Ghz. I remember now I had to set CPU overprovisioning
> to
> > 4
> > > as
> > > > > > > CloudStack took away 4x4Ghz=16Ghz of the available CPU.
> > > > > > > So Cloudstack sets the VM CPU Mhz limit what the actual
> > > > > > > limit
> > > is
> > > > > set
> > > > > > > to in the offering but does not multiply the Mhz limit by
> > > > > > > the
> > > > > number
> > > > > > > of cores when setting the limit on vmware. However takes
> > > > > > > away "number_of_cores*Mhz limit" from the available CPU
> > > > > > > capacity
> > > when
> > > > > > > allocating.
> > > > > > >
> > > > > > > I'm getting confused myself so I'm not sure if this is bug
> > now
> > > or
> > > > > not
> > > > > > > either :) I'm using 3.0.3
> > > > > > >
> > > > > > > Regards
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > Sent: 23 May 2012 00:32
> > > > > > > To: cloudstack-dev@incubator.apache.org; Tamas Monos
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > But in Diego'case, the limit is set as 1000Mhz, while his
> > > service
> > > > > > > offering is 1000Mhz * 2 cores.
> > > > > > > Which version of cloudstack are you using? Maybe it's a
> > > regression.
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > Sent: Tuesday, May 22, 2012 4:26 PM
> > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I have a service offering with 1000Mhz limit and 4 cpu
> > cores.
> > > > > That
> > > > > > > > sums up to 4000Mhz.
> > > > > > > > Cloudstack sets the limit to 4Ghz on the virtual machine
> > and
> > > > > > > > when
> > > > > I
> > > > > > > > put load on it vmware balances the load between the 4
> > > > > > > > cores
> > > > > allowing
> > > > > > > > them to use 1000Mhz each.
> > > > > > > > I do not see any bugs here.
> > > > > > > >
> > > > > > > > Apologies if the "meant per CPU core" was incorrect.
> > > > > > > > What I meant
> > > > > is
> > > > > > > > described above.
> > > > > > > >
> > > > > > > > Regards
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > > Sent: 23 May 2012 00:05
> > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > >
> > > > > > > > Nope, from the document, the limit is set on whole vm:
> > > > > > > > CPU Limits
> > > > > > > >
> > > > > > > > When a CPU Limit is set on a virtual machine resource
> > > settings,
> > > > > the
> > > > > > > > virtual machine is deliberately held from being
> > > > > > > > scheduled
> > to
> > > a
> > > > > PCPU
> > > > > > > > when it has used up its allocated CPU resource. This
> > happens
> > > > > > > > regardless of the CPU utilization. If the limit is set
> > > > > > > > to 500MHz, the virtual machine is descheduled from the
> > > > > > > > PCPU
> > and
> > > has
> > > > > > > > to wait before
> > > > > > > it
> > > > > > > > is allowed to be scheduled again. As such, the virtual
> > > machine
> > > > > might
> > > > > > > > experience performance degradation.
> > > > > > > >
> > > > > > > > Note: For an SMP virtual machine, the sum of all vCPUs
> > cannot
> > > > > exceed
> > > > > > > > the specified limit. For example, 4 vCPU virtual machine
> > with
> > > a
> > > > > > > > limit of 1200MHz and equal load among vCPUs would result
> > > > > > > > in
> > a
> > > > > > > > max
> > > > > of
> > > > > > > > 300MHz per vCPU.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&pop
> > > > > > u
> > > > > > p
> > > > > > > > =
> > > > > > > > true&languageId=&externalID=1033115
> > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > Sent: Tuesday, May 22, 2012 3:43 PM
> > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > No I don't think this is a bug. When you set 1000Mhz
> > > > > > > > > as
> > CPU
> > > > > > > > > cap
> > > > > > > that
> > > > > > > > > is meant per core. vmWare will limit each CPU core to
> > > 1000Mhz.
> > > > > > > > > As you gave 2 CPU cores that is 2000Mhz effective.
> > > > > > > > > That
> > is
> > > how
> > > > > > > > > vmware works.
> > > > > > > > >
> > > > > > > > > I have setup my offerings all to 1000Mhz as speed and
> > just
> > > > > > > > > increasing the number of cores.
> > > > > > > > > 1 core ends up being 1x1000Mhz
> > > > > > > > > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > > > > > > > > ...
> > > > > > > > >
> > > > > > > > > I'm actually using it in production and works quite
> well
> > as
> > > > > > > > Cloudstack
> > > > > > > > > "allocates" 4000Mhz when I'm using 4x1000Mhz cores and
> > > vmware
> > > > > > > > cleverly
> > > > > > > > > balances between the cores as you put load on it and
> not
> > > > > letting
> > > > > > > any
> > > > > > > > > of the cores above the set limit of 1000Mhz.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Diego Spinola Castro
> > [mailto:spinolacastro@gmail.com]
> > > > > > > > > Sent: 22 May 2012 19:34
> > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > Subject: Re: Vmware CPU Cap
> > > > > > > > >
> > > > > > > > > I forgot the link:
> > > > > > > > > http://imageshack.us/photo/my-images/848/cpulimit.png/
> > > > > > > > >
> > > > > > > > > 2012/5/22 Diego Spinola Castro
> <sp...@gmail.com>
> > > > > > > > >
> > > > > > > > > > I believe that is a bug with cpu cap and vmware.
> > > > > > > > > >
> > > > > > > > > > To reproduce:
> > > > > > > > > >
> > > > > > > > > > Create a offering with 2 cores and 1000mhz.
> > > > > > > > > > Enable CPU CAP.
> > > > > > > > > >
> > > > > > > > > > After created instance , cs create a vm with 2 cores
> > and
> > > > > 1000mhz
> > > > > > > > > > of
> > > > > > > > > limit.
> > > > > > > > > >
> > > > > > > > > > I don't know for sure if is a bug, but vmware gives
> > > 1000mhz
> > > > > > > shared
> > > > > > > > > > with cores.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Diego
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
>
>
RE: Vmware CPU Cap
Posted by Edison Su <Ed...@citrix.com>.
Single core means there is only one scheduling entity, that hypervisor can schedule for the VM, so there is no parallel processing at all. Hypervisor can not schedule a single core VM to multiple physical cpu cores at the same time.
> -----Original Message-----
> From: Tamas Monos [mailto:tamasm@veber.co.uk]
> Sent: Wednesday, May 23, 2012 2:59 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Hi,
>
> Before we would touch that rule 2:
> While utilizing 12Ghz on a single core is not possible it is possible
> to have effective 12Ghz on 4 cores at 3Ghz via parallel processing.
> If I start 4 cpuburn threads on a quad core it will use all four 3Ghz
> cores flat out "giving similar speed as 1 core@12Ghz would be"*. As
> vmware uses the limit parameter as limit to the whole CPU utilization
> rule 2 approach would prevent me effectively utilizing the CPU on
> vmware because if we say the CPU speed in Mhz cannot be larger than the
> core speed I would be limited to an effective speed of core-
> speed/number_of_cores. This would mean 3000/4=750Mhz per core on a
> 4vCPU VM. If you set "core_number*core_speed" as limit you already
> broke rule 2 as the CPU is set higher than its core speed.
>
> I believe it good as it is now.
>
> Regards
>
> -----Original Message-----
> From: Prachi Damle [mailto:Prachi.Damle@citrix.com]
> Sent: 23 May 2012 22:36
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Yeah, rule#1 and #3 are considered by the current Host Allocator;
> rule#2 to compare just the speed is missing.
>
> Prachi
>
> -----Original Message-----
> From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> Sent: Wednesday, May 23, 2012 2:06 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Checked the code, I think we miss rule 2 in allocator.
>
> Anthony
>
> > -----Original Message-----
> > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> > Sent: Wednesday, May 23, 2012 1:50 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > I'm pretty sure we have that before we redesign allocator.
> >
> > Anthony
> >
> > > -----Original Message-----
> > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > Sent: Wednesday, May 23, 2012 1:44 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > I'm surprised. I definitely recall having that conversation before
> > > about never exceeding the physical limits of the hypervisor. I'm
> > > pretty sure we have test cases against that as well.
> > >
> > > --Alex
> > >
> > > > -----Original Message-----
> > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > Sent: Wednesday, May 23, 2012 1:39 PM
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > I think it's a general issue for all the hypervisors we supported.
> > > >
> > > > > -----Original Message-----
> > > > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > > > Sent: Wednesday, May 23, 2012 1:32 PM
> > > > > To: cloudstack-dev@incubator.apache.org
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > That's because no one wrote a host allocator for VmWare.
> > > > >
> > > > > The relationship between DeploymentPlanner and HostAllocator
> and
> > > > > StoragePoolAllocator is as follows:
> > > > >
> > > > > DeploymentPlanner deals with heuristics and affinity rules of
> > > placing
> > > > > a VM. Once it determines a set of hosts that matches, it then
> > asks
> > > > > the HostAllocator to work on the limitations of the type of
> > > hypervisor.
> > > > > There was never a HostAllocator written for VmWare. It just
> > > > > uses
> > > the
> > > > > one that was written originally for XenServer. Hence the bug.
> > > > >
> > > > > There's similar differentiation for DeploymentPlanner and
> > > > > StoragePoolAllocator and similar bugs exists, especially if
> > someone
> > > > > adds a completely new type of storage pool but decides to just
> > > reuse
> > > > > the current set of StoragePoolAllocators.
> > > > >
> > > > > --Alex
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > Sent: Wednesday, May 23, 2012 1:24 PM
> > > > > > To: cloudstack-dev@incubator.apache.org; 'tamasm@veber.co.uk'
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > Thanks for your input, I find another bug in cloudstack...
> > > > > > Let's dive little bit deeper into how the CPU allocation
> works.
> > > > > > VMware(the same for other hypervisors(Xen/KVM)) uses
> > proportional
> > > > > share
> > > > > > based scheduling algorithm([1],[2]). It means "If a virtual
> > > machine
> > > > > has twice
> > > > > > as many shares of a resource as another virtual machine, it
> is
> > > > > entitled to
> > > > > > consume twice as much of that resource when these two virtual
> > > > > machines
> > > > > > are competing for resources."
> > > > > > There are two questions:
> > > > > > 1. Where is the share coming from? In Cloudstack, the share
> is
> > > > > calculated
> > > > > > from (CPU MHz * CPU cores).
> > > > > > 2. How the share of a VM is mapped to the physical CPU by the
> > > > > hypervisor?
> > > > > > Of cause, it depends on the hypervisor implementation, but
> > there
> > > are
> > > > > some
> > > > > > general rules that we need to follow, as we are living in the
> > > real
> > > > > world:)
> > > > > > Rule 1: num of cpu core of a VM should be <= num of cpu
> > > > > > core
> > > of
> > > > > the
> > > > > > hypervisor host. It doesn't make sense to running a 12 core
> VM
> > on
> > > a
> > > > > single
> > > > > > core host, even some hypervisors, such as KVM, can do that.
> > > > > Hypervisor will
> > > > > > be busy at scheduling VCPU(the VCPU context switch is much
> > > heavier
> > > > > than
> > > > > > process context switch), thus bad performance for the VM.
> > > > > > Rule 2: The frequency of a VCPU should be <= the frequency
> > of
> > > the
> > > > > host
> > > > > > hypervisor host. Can you run an one core * 12Ghz VM on a 3Ghz
> > > > > > *
> > 4
> > > > > core
> > > > > > physical CPU? Nope, in an unit time, the max freq of VCPU can
> > get
> > > is
> > > > > the max
> > > > > > freq of physical cpu core, that's the physical LAW.
> > > > > > Rule 3: the total share of VMs <= total share of host
> > > > > hypervisor(in case of
> > > > > > no CPU overcommit).
> > > > > >
> > > > > > Currently, in cloudstack host allocator, only rule 3 is taken
> > > into
> > > > > consideration.
> > > > > >
> > > > > > [1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM-
> > resource.pdf
> > > > > > [2]
> > > > > >
> > > > http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > Sent: Tuesday, May 22, 2012 5:41 PM
> > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I still don't think this is an issue as the CPU Mhz limit
> > > > > > > and
> > > the
> > > > > > > number of cores are independent.
> > > > > > > CPU manufacturers sell 2,4,6 cores at 3Ghz and not
> > > > > > > 6,12,18Ghz
> > > CPUs.
> > > > > > >
> > > > > > > So I think it is good how it works but the
> > > "number_of_cores*Mhz"
> > > > > while
> > > > > > > allocating should not multiply so that is the bug :) What
> > > vmware
> > > > > does
> > > > > > > with multiplying the cores with the core speed is bad as I
> > > can't
> > > > > have
> > > > > > > a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know where
> > > > > > > I'm
> > > > > coming
> > > > > > > from....
> > > > > > >
> > > > > > > Regards
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > Sent: 23 May 2012 01:27
> > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I have confused myself too because if I have a look the
> > > database
> > > > > and
> > > > > > > dump the service_offerings table the limit for me is set to
> > > > > > > 2000,3000,4000 and not 1000.
> > > > > > > So a Mhz limit set to 4000 and 4 cores will end up as a
> quad
> > > core
> > > > > box
> > > > > > > at 4Ghz. I remember now I had to set CPU overprovisioning
> to
> > 4
> > > as
> > > > > > > CloudStack took away 4x4Ghz=16Ghz of the available CPU.
> > > > > > > So Cloudstack sets the VM CPU Mhz limit what the actual
> > > > > > > limit
> > > is
> > > > > set
> > > > > > > to in the offering but does not multiply the Mhz limit by
> > > > > > > the
> > > > > number
> > > > > > > of cores when setting the limit on vmware. However takes
> > > > > > > away "number_of_cores*Mhz limit" from the available CPU
> > > > > > > capacity
> > > when
> > > > > > > allocating.
> > > > > > >
> > > > > > > I'm getting confused myself so I'm not sure if this is bug
> > now
> > > or
> > > > > not
> > > > > > > either :) I'm using 3.0.3
> > > > > > >
> > > > > > > Regards
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > Sent: 23 May 2012 00:32
> > > > > > > To: cloudstack-dev@incubator.apache.org; Tamas Monos
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > But in Diego'case, the limit is set as 1000Mhz, while his
> > > service
> > > > > > > offering is 1000Mhz * 2 cores.
> > > > > > > Which version of cloudstack are you using? Maybe it's a
> > > regression.
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > Sent: Tuesday, May 22, 2012 4:26 PM
> > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I have a service offering with 1000Mhz limit and 4 cpu
> > cores.
> > > > > That
> > > > > > > > sums up to 4000Mhz.
> > > > > > > > Cloudstack sets the limit to 4Ghz on the virtual machine
> > and
> > > > > > > > when
> > > > > I
> > > > > > > > put load on it vmware balances the load between the 4
> > > > > > > > cores
> > > > > allowing
> > > > > > > > them to use 1000Mhz each.
> > > > > > > > I do not see any bugs here.
> > > > > > > >
> > > > > > > > Apologies if the "meant per CPU core" was incorrect. What
> > > > > > > > I meant
> > > > > is
> > > > > > > > described above.
> > > > > > > >
> > > > > > > > Regards
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > > Sent: 23 May 2012 00:05
> > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > >
> > > > > > > > Nope, from the document, the limit is set on whole vm:
> > > > > > > > CPU Limits
> > > > > > > >
> > > > > > > > When a CPU Limit is set on a virtual machine resource
> > > settings,
> > > > > the
> > > > > > > > virtual machine is deliberately held from being scheduled
> > to
> > > a
> > > > > PCPU
> > > > > > > > when it has used up its allocated CPU resource. This
> > happens
> > > > > > > > regardless of the CPU utilization. If the limit is set to
> > > > > > > > 500MHz, the virtual machine is descheduled from the PCPU
> > and
> > > has
> > > > > > > > to wait before
> > > > > > > it
> > > > > > > > is allowed to be scheduled again. As such, the virtual
> > > machine
> > > > > might
> > > > > > > > experience performance degradation.
> > > > > > > >
> > > > > > > > Note: For an SMP virtual machine, the sum of all vCPUs
> > cannot
> > > > > exceed
> > > > > > > > the specified limit. For example, 4 vCPU virtual machine
> > with
> > > a
> > > > > > > > limit of 1200MHz and equal load among vCPUs would result
> > > > > > > > in
> > a
> > > > > > > > max
> > > > > of
> > > > > > > > 300MHz per vCPU.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&pop
> > > > > > u
> > > > > > p
> > > > > > > > =
> > > > > > > > true&languageId=&externalID=1033115
> > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > Sent: Tuesday, May 22, 2012 3:43 PM
> > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > No I don't think this is a bug. When you set 1000Mhz as
> > CPU
> > > > > > > > > cap
> > > > > > > that
> > > > > > > > > is meant per core. vmWare will limit each CPU core to
> > > 1000Mhz.
> > > > > > > > > As you gave 2 CPU cores that is 2000Mhz effective. That
> > is
> > > how
> > > > > > > > > vmware works.
> > > > > > > > >
> > > > > > > > > I have setup my offerings all to 1000Mhz as speed and
> > just
> > > > > > > > > increasing the number of cores.
> > > > > > > > > 1 core ends up being 1x1000Mhz
> > > > > > > > > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > > > > > > > > ...
> > > > > > > > >
> > > > > > > > > I'm actually using it in production and works quite
> well
> > as
> > > > > > > > Cloudstack
> > > > > > > > > "allocates" 4000Mhz when I'm using 4x1000Mhz cores and
> > > vmware
> > > > > > > > cleverly
> > > > > > > > > balances between the cores as you put load on it and
> not
> > > > > letting
> > > > > > > any
> > > > > > > > > of the cores above the set limit of 1000Mhz.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Diego Spinola Castro
> > [mailto:spinolacastro@gmail.com]
> > > > > > > > > Sent: 22 May 2012 19:34
> > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > Subject: Re: Vmware CPU Cap
> > > > > > > > >
> > > > > > > > > I forgot the link:
> > > > > > > > > http://imageshack.us/photo/my-images/848/cpulimit.png/
> > > > > > > > >
> > > > > > > > > 2012/5/22 Diego Spinola Castro
> <sp...@gmail.com>
> > > > > > > > >
> > > > > > > > > > I believe that is a bug with cpu cap and vmware.
> > > > > > > > > >
> > > > > > > > > > To reproduce:
> > > > > > > > > >
> > > > > > > > > > Create a offering with 2 cores and 1000mhz.
> > > > > > > > > > Enable CPU CAP.
> > > > > > > > > >
> > > > > > > > > > After created instance , cs create a vm with 2 cores
> > and
> > > > > 1000mhz
> > > > > > > > > > of
> > > > > > > > > limit.
> > > > > > > > > >
> > > > > > > > > > I don't know for sure if is a bug, but vmware gives
> > > 1000mhz
> > > > > > > shared
> > > > > > > > > > with cores.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Diego
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
>
>
RE: Vmware CPU Cap
Posted by Tamas Monos <ta...@veber.co.uk>.
Hi,
Before we would touch that rule 2:
While utilizing 12Ghz on a single core is not possible it is possible to have effective 12Ghz on 4 cores at 3Ghz via parallel processing.
If I start 4 cpuburn threads on a quad core it will use all four 3Ghz cores flat out "giving similar speed as 1 core@12Ghz would be"*. As vmware uses the limit parameter as limit to the whole CPU utilization rule 2 approach would prevent me effectively utilizing the CPU on vmware because if we say the CPU speed in Mhz cannot be larger than the core speed I would be limited to an effective speed of core-speed/number_of_cores. This would mean 3000/4=750Mhz per core on a 4vCPU VM. If you set "core_number*core_speed" as limit you already broke rule 2 as the CPU is set higher than its core speed.
I believe it good as it is now.
Regards
-----Original Message-----
From: Prachi Damle [mailto:Prachi.Damle@citrix.com]
Sent: 23 May 2012 22:36
To: cloudstack-dev@incubator.apache.org
Subject: RE: Vmware CPU Cap
Yeah, rule#1 and #3 are considered by the current Host Allocator; rule#2 to compare just the speed is missing.
Prachi
-----Original Message-----
From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
Sent: Wednesday, May 23, 2012 2:06 PM
To: cloudstack-dev@incubator.apache.org
Subject: RE: Vmware CPU Cap
Checked the code, I think we miss rule 2 in allocator.
Anthony
> -----Original Message-----
> From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> Sent: Wednesday, May 23, 2012 1:50 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> I'm pretty sure we have that before we redesign allocator.
>
> Anthony
>
> > -----Original Message-----
> > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > Sent: Wednesday, May 23, 2012 1:44 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > I'm surprised. I definitely recall having that conversation before
> > about never exceeding the physical limits of the hypervisor. I'm
> > pretty sure we have test cases against that as well.
> >
> > --Alex
> >
> > > -----Original Message-----
> > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > Sent: Wednesday, May 23, 2012 1:39 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > I think it's a general issue for all the hypervisors we supported.
> > >
> > > > -----Original Message-----
> > > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > > Sent: Wednesday, May 23, 2012 1:32 PM
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > That's because no one wrote a host allocator for VmWare.
> > > >
> > > > The relationship between DeploymentPlanner and HostAllocator and
> > > > StoragePoolAllocator is as follows:
> > > >
> > > > DeploymentPlanner deals with heuristics and affinity rules of
> > placing
> > > > a VM. Once it determines a set of hosts that matches, it then
> asks
> > > > the HostAllocator to work on the limitations of the type of
> > hypervisor.
> > > > There was never a HostAllocator written for VmWare. It just
> > > > uses
> > the
> > > > one that was written originally for XenServer. Hence the bug.
> > > >
> > > > There's similar differentiation for DeploymentPlanner and
> > > > StoragePoolAllocator and similar bugs exists, especially if
> someone
> > > > adds a completely new type of storage pool but decides to just
> > reuse
> > > > the current set of StoragePoolAllocators.
> > > >
> > > > --Alex
> > > >
> > > > > -----Original Message-----
> > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > Sent: Wednesday, May 23, 2012 1:24 PM
> > > > > To: cloudstack-dev@incubator.apache.org; 'tamasm@veber.co.uk'
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > Thanks for your input, I find another bug in cloudstack...
> > > > > Let's dive little bit deeper into how the CPU allocation works.
> > > > > VMware(the same for other hypervisors(Xen/KVM)) uses
> proportional
> > > > share
> > > > > based scheduling algorithm([1],[2]). It means "If a virtual
> > machine
> > > > has twice
> > > > > as many shares of a resource as another virtual machine, it is
> > > > entitled to
> > > > > consume twice as much of that resource when these two virtual
> > > > machines
> > > > > are competing for resources."
> > > > > There are two questions:
> > > > > 1. Where is the share coming from? In Cloudstack, the share is
> > > > calculated
> > > > > from (CPU MHz * CPU cores).
> > > > > 2. How the share of a VM is mapped to the physical CPU by the
> > > > hypervisor?
> > > > > Of cause, it depends on the hypervisor implementation, but
> there
> > are
> > > > some
> > > > > general rules that we need to follow, as we are living in the
> > real
> > > > world:)
> > > > > Rule 1: num of cpu core of a VM should be <= num of cpu
> > > > > core
> > of
> > > > the
> > > > > hypervisor host. It doesn't make sense to running a 12 core VM
> on
> > a
> > > > single
> > > > > core host, even some hypervisors, such as KVM, can do that.
> > > > Hypervisor will
> > > > > be busy at scheduling VCPU(the VCPU context switch is much
> > heavier
> > > > than
> > > > > process context switch), thus bad performance for the VM.
> > > > > Rule 2: The frequency of a VCPU should be <= the frequency
> of
> > the
> > > > host
> > > > > hypervisor host. Can you run an one core * 12Ghz VM on a 3Ghz
> > > > > *
> 4
> > > > core
> > > > > physical CPU? Nope, in an unit time, the max freq of VCPU can
> get
> > is
> > > > the max
> > > > > freq of physical cpu core, that's the physical LAW.
> > > > > Rule 3: the total share of VMs <= total share of host
> > > > hypervisor(in case of
> > > > > no CPU overcommit).
> > > > >
> > > > > Currently, in cloudstack host allocator, only rule 3 is taken
> > into
> > > > consideration.
> > > > >
> > > > > [1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM-
> resource.pdf
> > > > > [2]
> > > > >
> > > http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > Sent: Tuesday, May 22, 2012 5:41 PM
> > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I still don't think this is an issue as the CPU Mhz limit
> > > > > > and
> > the
> > > > > > number of cores are independent.
> > > > > > CPU manufacturers sell 2,4,6 cores at 3Ghz and not
> > > > > > 6,12,18Ghz
> > CPUs.
> > > > > >
> > > > > > So I think it is good how it works but the
> > "number_of_cores*Mhz"
> > > > while
> > > > > > allocating should not multiply so that is the bug :) What
> > vmware
> > > > does
> > > > > > with multiplying the cores with the core speed is bad as I
> > can't
> > > > have
> > > > > > a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know where
> > > > > > I'm
> > > > coming
> > > > > > from....
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > Sent: 23 May 2012 01:27
> > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I have confused myself too because if I have a look the
> > database
> > > > and
> > > > > > dump the service_offerings table the limit for me is set to
> > > > > > 2000,3000,4000 and not 1000.
> > > > > > So a Mhz limit set to 4000 and 4 cores will end up as a quad
> > core
> > > > box
> > > > > > at 4Ghz. I remember now I had to set CPU overprovisioning to
> 4
> > as
> > > > > > CloudStack took away 4x4Ghz=16Ghz of the available CPU.
> > > > > > So Cloudstack sets the VM CPU Mhz limit what the actual
> > > > > > limit
> > is
> > > > set
> > > > > > to in the offering but does not multiply the Mhz limit by
> > > > > > the
> > > > number
> > > > > > of cores when setting the limit on vmware. However takes
> > > > > > away "number_of_cores*Mhz limit" from the available CPU
> > > > > > capacity
> > when
> > > > > > allocating.
> > > > > >
> > > > > > I'm getting confused myself so I'm not sure if this is bug
> now
> > or
> > > > not
> > > > > > either :) I'm using 3.0.3
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > Sent: 23 May 2012 00:32
> > > > > > To: cloudstack-dev@incubator.apache.org; Tamas Monos
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > But in Diego'case, the limit is set as 1000Mhz, while his
> > service
> > > > > > offering is 1000Mhz * 2 cores.
> > > > > > Which version of cloudstack are you using? Maybe it's a
> > regression.
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > Sent: Tuesday, May 22, 2012 4:26 PM
> > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I have a service offering with 1000Mhz limit and 4 cpu
> cores.
> > > > That
> > > > > > > sums up to 4000Mhz.
> > > > > > > Cloudstack sets the limit to 4Ghz on the virtual machine
> and
> > > > > > > when
> > > > I
> > > > > > > put load on it vmware balances the load between the 4
> > > > > > > cores
> > > > allowing
> > > > > > > them to use 1000Mhz each.
> > > > > > > I do not see any bugs here.
> > > > > > >
> > > > > > > Apologies if the "meant per CPU core" was incorrect. What
> > > > > > > I meant
> > > > is
> > > > > > > described above.
> > > > > > >
> > > > > > > Regards
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > Sent: 23 May 2012 00:05
> > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > Nope, from the document, the limit is set on whole vm:
> > > > > > > CPU Limits
> > > > > > >
> > > > > > > When a CPU Limit is set on a virtual machine resource
> > settings,
> > > > the
> > > > > > > virtual machine is deliberately held from being scheduled
> to
> > a
> > > > PCPU
> > > > > > > when it has used up its allocated CPU resource. This
> happens
> > > > > > > regardless of the CPU utilization. If the limit is set to
> > > > > > > 500MHz, the virtual machine is descheduled from the PCPU
> and
> > has
> > > > > > > to wait before
> > > > > > it
> > > > > > > is allowed to be scheduled again. As such, the virtual
> > machine
> > > > might
> > > > > > > experience performance degradation.
> > > > > > >
> > > > > > > Note: For an SMP virtual machine, the sum of all vCPUs
> cannot
> > > > exceed
> > > > > > > the specified limit. For example, 4 vCPU virtual machine
> with
> > a
> > > > > > > limit of 1200MHz and equal load among vCPUs would result
> > > > > > > in
> a
> > > > > > > max
> > > > of
> > > > > > > 300MHz per vCPU.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&pop
> > > > > u
> > > > > p
> > > > > > > =
> > > > > > > true&languageId=&externalID=1033115
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > Sent: Tuesday, May 22, 2012 3:43 PM
> > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > No I don't think this is a bug. When you set 1000Mhz as
> CPU
> > > > > > > > cap
> > > > > > that
> > > > > > > > is meant per core. vmWare will limit each CPU core to
> > 1000Mhz.
> > > > > > > > As you gave 2 CPU cores that is 2000Mhz effective. That
> is
> > how
> > > > > > > > vmware works.
> > > > > > > >
> > > > > > > > I have setup my offerings all to 1000Mhz as speed and
> just
> > > > > > > > increasing the number of cores.
> > > > > > > > 1 core ends up being 1x1000Mhz
> > > > > > > > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > > > > > > > ...
> > > > > > > >
> > > > > > > > I'm actually using it in production and works quite well
> as
> > > > > > > Cloudstack
> > > > > > > > "allocates" 4000Mhz when I'm using 4x1000Mhz cores and
> > vmware
> > > > > > > cleverly
> > > > > > > > balances between the cores as you put load on it and not
> > > > letting
> > > > > > any
> > > > > > > > of the cores above the set limit of 1000Mhz.
> > > > > > > >
> > > > > > > >
> > > > > > > > Regards
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Diego Spinola Castro
> [mailto:spinolacastro@gmail.com]
> > > > > > > > Sent: 22 May 2012 19:34
> > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > Subject: Re: Vmware CPU Cap
> > > > > > > >
> > > > > > > > I forgot the link:
> > > > > > > > http://imageshack.us/photo/my-images/848/cpulimit.png/
> > > > > > > >
> > > > > > > > 2012/5/22 Diego Spinola Castro <sp...@gmail.com>
> > > > > > > >
> > > > > > > > > I believe that is a bug with cpu cap and vmware.
> > > > > > > > >
> > > > > > > > > To reproduce:
> > > > > > > > >
> > > > > > > > > Create a offering with 2 cores and 1000mhz.
> > > > > > > > > Enable CPU CAP.
> > > > > > > > >
> > > > > > > > > After created instance , cs create a vm with 2 cores
> and
> > > > 1000mhz
> > > > > > > > > of
> > > > > > > > limit.
> > > > > > > > >
> > > > > > > > > I don't know for sure if is a bug, but vmware gives
> > 1000mhz
> > > > > > shared
> > > > > > > > > with cores.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Diego
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
RE: Vmware CPU Cap
Posted by Prachi Damle <Pr...@citrix.com>.
Yeah, rule#1 and #3 are considered by the current Host Allocator; rule#2 to compare just the speed is missing.
Prachi
-----Original Message-----
From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
Sent: Wednesday, May 23, 2012 2:06 PM
To: cloudstack-dev@incubator.apache.org
Subject: RE: Vmware CPU Cap
Checked the code, I think we miss rule 2 in allocator.
Anthony
> -----Original Message-----
> From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> Sent: Wednesday, May 23, 2012 1:50 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> I'm pretty sure we have that before we redesign allocator.
>
> Anthony
>
> > -----Original Message-----
> > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > Sent: Wednesday, May 23, 2012 1:44 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > I'm surprised. I definitely recall having that conversation before
> > about never exceeding the physical limits of the hypervisor. I'm
> > pretty sure we have test cases against that as well.
> >
> > --Alex
> >
> > > -----Original Message-----
> > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > Sent: Wednesday, May 23, 2012 1:39 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > I think it's a general issue for all the hypervisors we supported.
> > >
> > > > -----Original Message-----
> > > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > > Sent: Wednesday, May 23, 2012 1:32 PM
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > That's because no one wrote a host allocator for VmWare.
> > > >
> > > > The relationship between DeploymentPlanner and HostAllocator and
> > > > StoragePoolAllocator is as follows:
> > > >
> > > > DeploymentPlanner deals with heuristics and affinity rules of
> > placing
> > > > a VM. Once it determines a set of hosts that matches, it then
> asks
> > > > the HostAllocator to work on the limitations of the type of
> > hypervisor.
> > > > There was never a HostAllocator written for VmWare. It just
> > > > uses
> > the
> > > > one that was written originally for XenServer. Hence the bug.
> > > >
> > > > There's similar differentiation for DeploymentPlanner and
> > > > StoragePoolAllocator and similar bugs exists, especially if
> someone
> > > > adds a completely new type of storage pool but decides to just
> > reuse
> > > > the current set of StoragePoolAllocators.
> > > >
> > > > --Alex
> > > >
> > > > > -----Original Message-----
> > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > Sent: Wednesday, May 23, 2012 1:24 PM
> > > > > To: cloudstack-dev@incubator.apache.org; 'tamasm@veber.co.uk'
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > Thanks for your input, I find another bug in cloudstack...
> > > > > Let's dive little bit deeper into how the CPU allocation works.
> > > > > VMware(the same for other hypervisors(Xen/KVM)) uses
> proportional
> > > > share
> > > > > based scheduling algorithm([1],[2]). It means "If a virtual
> > machine
> > > > has twice
> > > > > as many shares of a resource as another virtual machine, it is
> > > > entitled to
> > > > > consume twice as much of that resource when these two virtual
> > > > machines
> > > > > are competing for resources."
> > > > > There are two questions:
> > > > > 1. Where is the share coming from? In Cloudstack, the share is
> > > > calculated
> > > > > from (CPU MHz * CPU cores).
> > > > > 2. How the share of a VM is mapped to the physical CPU by the
> > > > hypervisor?
> > > > > Of cause, it depends on the hypervisor implementation, but
> there
> > are
> > > > some
> > > > > general rules that we need to follow, as we are living in the
> > real
> > > > world:)
> > > > > Rule 1: num of cpu core of a VM should be <= num of cpu
> > > > > core
> > of
> > > > the
> > > > > hypervisor host. It doesn't make sense to running a 12 core VM
> on
> > a
> > > > single
> > > > > core host, even some hypervisors, such as KVM, can do that.
> > > > Hypervisor will
> > > > > be busy at scheduling VCPU(the VCPU context switch is much
> > heavier
> > > > than
> > > > > process context switch), thus bad performance for the VM.
> > > > > Rule 2: The frequency of a VCPU should be <= the frequency
> of
> > the
> > > > host
> > > > > hypervisor host. Can you run an one core * 12Ghz VM on a 3Ghz
> > > > > *
> 4
> > > > core
> > > > > physical CPU? Nope, in an unit time, the max freq of VCPU can
> get
> > is
> > > > the max
> > > > > freq of physical cpu core, that's the physical LAW.
> > > > > Rule 3: the total share of VMs <= total share of host
> > > > hypervisor(in case of
> > > > > no CPU overcommit).
> > > > >
> > > > > Currently, in cloudstack host allocator, only rule 3 is taken
> > into
> > > > consideration.
> > > > >
> > > > > [1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM-
> resource.pdf
> > > > > [2]
> > > > >
> > > http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > Sent: Tuesday, May 22, 2012 5:41 PM
> > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I still don't think this is an issue as the CPU Mhz limit
> > > > > > and
> > the
> > > > > > number of cores are independent.
> > > > > > CPU manufacturers sell 2,4,6 cores at 3Ghz and not
> > > > > > 6,12,18Ghz
> > CPUs.
> > > > > >
> > > > > > So I think it is good how it works but the
> > "number_of_cores*Mhz"
> > > > while
> > > > > > allocating should not multiply so that is the bug :) What
> > vmware
> > > > does
> > > > > > with multiplying the cores with the core speed is bad as I
> > can't
> > > > have
> > > > > > a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know where
> > > > > > I'm
> > > > coming
> > > > > > from....
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > Sent: 23 May 2012 01:27
> > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I have confused myself too because if I have a look the
> > database
> > > > and
> > > > > > dump the service_offerings table the limit for me is set to
> > > > > > 2000,3000,4000 and not 1000.
> > > > > > So a Mhz limit set to 4000 and 4 cores will end up as a quad
> > core
> > > > box
> > > > > > at 4Ghz. I remember now I had to set CPU overprovisioning to
> 4
> > as
> > > > > > CloudStack took away 4x4Ghz=16Ghz of the available CPU.
> > > > > > So Cloudstack sets the VM CPU Mhz limit what the actual
> > > > > > limit
> > is
> > > > set
> > > > > > to in the offering but does not multiply the Mhz limit by
> > > > > > the
> > > > number
> > > > > > of cores when setting the limit on vmware. However takes
> > > > > > away "number_of_cores*Mhz limit" from the available CPU
> > > > > > capacity
> > when
> > > > > > allocating.
> > > > > >
> > > > > > I'm getting confused myself so I'm not sure if this is bug
> now
> > or
> > > > not
> > > > > > either :) I'm using 3.0.3
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > Sent: 23 May 2012 00:32
> > > > > > To: cloudstack-dev@incubator.apache.org; Tamas Monos
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > But in Diego'case, the limit is set as 1000Mhz, while his
> > service
> > > > > > offering is 1000Mhz * 2 cores.
> > > > > > Which version of cloudstack are you using? Maybe it's a
> > regression.
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > Sent: Tuesday, May 22, 2012 4:26 PM
> > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I have a service offering with 1000Mhz limit and 4 cpu
> cores.
> > > > That
> > > > > > > sums up to 4000Mhz.
> > > > > > > Cloudstack sets the limit to 4Ghz on the virtual machine
> and
> > > > > > > when
> > > > I
> > > > > > > put load on it vmware balances the load between the 4
> > > > > > > cores
> > > > allowing
> > > > > > > them to use 1000Mhz each.
> > > > > > > I do not see any bugs here.
> > > > > > >
> > > > > > > Apologies if the "meant per CPU core" was incorrect. What
> > > > > > > I meant
> > > > is
> > > > > > > described above.
> > > > > > >
> > > > > > > Regards
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > Sent: 23 May 2012 00:05
> > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > Nope, from the document, the limit is set on whole vm:
> > > > > > > CPU Limits
> > > > > > >
> > > > > > > When a CPU Limit is set on a virtual machine resource
> > settings,
> > > > the
> > > > > > > virtual machine is deliberately held from being scheduled
> to
> > a
> > > > PCPU
> > > > > > > when it has used up its allocated CPU resource. This
> happens
> > > > > > > regardless of the CPU utilization. If the limit is set to
> > > > > > > 500MHz, the virtual machine is descheduled from the PCPU
> and
> > has
> > > > > > > to wait before
> > > > > > it
> > > > > > > is allowed to be scheduled again. As such, the virtual
> > machine
> > > > might
> > > > > > > experience performance degradation.
> > > > > > >
> > > > > > > Note: For an SMP virtual machine, the sum of all vCPUs
> cannot
> > > > exceed
> > > > > > > the specified limit. For example, 4 vCPU virtual machine
> with
> > a
> > > > > > > limit of 1200MHz and equal load among vCPUs would result
> > > > > > > in
> a
> > > > > > > max
> > > > of
> > > > > > > 300MHz per vCPU.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&pop
> > > > > u
> > > > > p
> > > > > > > =
> > > > > > > true&languageId=&externalID=1033115
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > Sent: Tuesday, May 22, 2012 3:43 PM
> > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > No I don't think this is a bug. When you set 1000Mhz as
> CPU
> > > > > > > > cap
> > > > > > that
> > > > > > > > is meant per core. vmWare will limit each CPU core to
> > 1000Mhz.
> > > > > > > > As you gave 2 CPU cores that is 2000Mhz effective. That
> is
> > how
> > > > > > > > vmware works.
> > > > > > > >
> > > > > > > > I have setup my offerings all to 1000Mhz as speed and
> just
> > > > > > > > increasing the number of cores.
> > > > > > > > 1 core ends up being 1x1000Mhz
> > > > > > > > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > > > > > > > ...
> > > > > > > >
> > > > > > > > I'm actually using it in production and works quite well
> as
> > > > > > > Cloudstack
> > > > > > > > "allocates" 4000Mhz when I'm using 4x1000Mhz cores and
> > vmware
> > > > > > > cleverly
> > > > > > > > balances between the cores as you put load on it and not
> > > > letting
> > > > > > any
> > > > > > > > of the cores above the set limit of 1000Mhz.
> > > > > > > >
> > > > > > > >
> > > > > > > > Regards
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Diego Spinola Castro
> [mailto:spinolacastro@gmail.com]
> > > > > > > > Sent: 22 May 2012 19:34
> > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > Subject: Re: Vmware CPU Cap
> > > > > > > >
> > > > > > > > I forgot the link:
> > > > > > > > http://imageshack.us/photo/my-images/848/cpulimit.png/
> > > > > > > >
> > > > > > > > 2012/5/22 Diego Spinola Castro <sp...@gmail.com>
> > > > > > > >
> > > > > > > > > I believe that is a bug with cpu cap and vmware.
> > > > > > > > >
> > > > > > > > > To reproduce:
> > > > > > > > >
> > > > > > > > > Create a offering with 2 cores and 1000mhz.
> > > > > > > > > Enable CPU CAP.
> > > > > > > > >
> > > > > > > > > After created instance , cs create a vm with 2 cores
> and
> > > > 1000mhz
> > > > > > > > > of
> > > > > > > > limit.
> > > > > > > > >
> > > > > > > > > I don't know for sure if is a bug, but vmware gives
> > 1000mhz
> > > > > > shared
> > > > > > > > > with cores.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Diego
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
RE: Vmware CPU Cap
Posted by Anthony Xu <Xu...@citrix.com>.
Checked the code, I think we miss rule 2 in allocator.
Anthony
> -----Original Message-----
> From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> Sent: Wednesday, May 23, 2012 1:50 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> I'm pretty sure we have that before we redesign allocator.
>
> Anthony
>
> > -----Original Message-----
> > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > Sent: Wednesday, May 23, 2012 1:44 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > I'm surprised. I definitely recall having that conversation before
> > about never exceeding the physical limits of the hypervisor. I'm
> > pretty sure we have test cases against that as well.
> >
> > --Alex
> >
> > > -----Original Message-----
> > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > Sent: Wednesday, May 23, 2012 1:39 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > I think it's a general issue for all the hypervisors we supported.
> > >
> > > > -----Original Message-----
> > > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > > Sent: Wednesday, May 23, 2012 1:32 PM
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > That's because no one wrote a host allocator for VmWare.
> > > >
> > > > The relationship between DeploymentPlanner and HostAllocator and
> > > > StoragePoolAllocator is as follows:
> > > >
> > > > DeploymentPlanner deals with heuristics and affinity rules of
> > placing
> > > > a VM. Once it determines a set of hosts that matches, it then
> asks
> > > > the HostAllocator to work on the limitations of the type of
> > hypervisor.
> > > > There was never a HostAllocator written for VmWare. It just uses
> > the
> > > > one that was written originally for XenServer. Hence the bug.
> > > >
> > > > There's similar differentiation for DeploymentPlanner and
> > > > StoragePoolAllocator and similar bugs exists, especially if
> someone
> > > > adds a completely new type of storage pool but decides to just
> > reuse
> > > > the current set of StoragePoolAllocators.
> > > >
> > > > --Alex
> > > >
> > > > > -----Original Message-----
> > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > Sent: Wednesday, May 23, 2012 1:24 PM
> > > > > To: cloudstack-dev@incubator.apache.org; 'tamasm@veber.co.uk'
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > Thanks for your input, I find another bug in cloudstack...
> > > > > Let's dive little bit deeper into how the CPU allocation works.
> > > > > VMware(the same for other hypervisors(Xen/KVM)) uses
> proportional
> > > > share
> > > > > based scheduling algorithm([1],[2]). It means "If a virtual
> > machine
> > > > has twice
> > > > > as many shares of a resource as another virtual machine, it is
> > > > entitled to
> > > > > consume twice as much of that resource when these two virtual
> > > > machines
> > > > > are competing for resources."
> > > > > There are two questions:
> > > > > 1. Where is the share coming from? In Cloudstack, the share is
> > > > calculated
> > > > > from (CPU MHz * CPU cores).
> > > > > 2. How the share of a VM is mapped to the physical CPU by the
> > > > hypervisor?
> > > > > Of cause, it depends on the hypervisor implementation, but
> there
> > are
> > > > some
> > > > > general rules that we need to follow, as we are living in the
> > real
> > > > world:)
> > > > > Rule 1: num of cpu core of a VM should be <= num of cpu core
> > of
> > > > the
> > > > > hypervisor host. It doesn't make sense to running a 12 core VM
> on
> > a
> > > > single
> > > > > core host, even some hypervisors, such as KVM, can do that.
> > > > Hypervisor will
> > > > > be busy at scheduling VCPU(the VCPU context switch is much
> > heavier
> > > > than
> > > > > process context switch), thus bad performance for the VM.
> > > > > Rule 2: The frequency of a VCPU should be <= the frequency
> of
> > the
> > > > host
> > > > > hypervisor host. Can you run an one core * 12Ghz VM on a 3Ghz *
> 4
> > > > core
> > > > > physical CPU? Nope, in an unit time, the max freq of VCPU can
> get
> > is
> > > > the max
> > > > > freq of physical cpu core, that's the physical LAW.
> > > > > Rule 3: the total share of VMs <= total share of host
> > > > hypervisor(in case of
> > > > > no CPU overcommit).
> > > > >
> > > > > Currently, in cloudstack host allocator, only rule 3 is taken
> > into
> > > > consideration.
> > > > >
> > > > > [1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM-
> resource.pdf
> > > > > [2]
> > > > >
> > > http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > Sent: Tuesday, May 22, 2012 5:41 PM
> > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I still don't think this is an issue as the CPU Mhz limit and
> > the
> > > > > > number of cores are independent.
> > > > > > CPU manufacturers sell 2,4,6 cores at 3Ghz and not 6,12,18Ghz
> > CPUs.
> > > > > >
> > > > > > So I think it is good how it works but the
> > "number_of_cores*Mhz"
> > > > while
> > > > > > allocating should not multiply so that is the bug :) What
> > vmware
> > > > does
> > > > > > with multiplying the cores with the core speed is bad as I
> > can't
> > > > have
> > > > > > a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know where I'm
> > > > coming
> > > > > > from....
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > Sent: 23 May 2012 01:27
> > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I have confused myself too because if I have a look the
> > database
> > > > and
> > > > > > dump the service_offerings table the limit for me is set to
> > > > > > 2000,3000,4000 and not 1000.
> > > > > > So a Mhz limit set to 4000 and 4 cores will end up as a quad
> > core
> > > > box
> > > > > > at 4Ghz. I remember now I had to set CPU overprovisioning to
> 4
> > as
> > > > > > CloudStack took away 4x4Ghz=16Ghz of the available CPU.
> > > > > > So Cloudstack sets the VM CPU Mhz limit what the actual limit
> > is
> > > > set
> > > > > > to in the offering but does not multiply the Mhz limit by the
> > > > number
> > > > > > of cores when setting the limit on vmware. However takes away
> > > > > > "number_of_cores*Mhz limit" from the available CPU capacity
> > when
> > > > > > allocating.
> > > > > >
> > > > > > I'm getting confused myself so I'm not sure if this is bug
> now
> > or
> > > > not
> > > > > > either :) I'm using 3.0.3
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > Sent: 23 May 2012 00:32
> > > > > > To: cloudstack-dev@incubator.apache.org; Tamas Monos
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > But in Diego'case, the limit is set as 1000Mhz, while his
> > service
> > > > > > offering is 1000Mhz * 2 cores.
> > > > > > Which version of cloudstack are you using? Maybe it's a
> > regression.
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > Sent: Tuesday, May 22, 2012 4:26 PM
> > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I have a service offering with 1000Mhz limit and 4 cpu
> cores.
> > > > That
> > > > > > > sums up to 4000Mhz.
> > > > > > > Cloudstack sets the limit to 4Ghz on the virtual machine
> and
> > > > > > > when
> > > > I
> > > > > > > put load on it vmware balances the load between the 4 cores
> > > > allowing
> > > > > > > them to use 1000Mhz each.
> > > > > > > I do not see any bugs here.
> > > > > > >
> > > > > > > Apologies if the "meant per CPU core" was incorrect. What I
> > > > > > > meant
> > > > is
> > > > > > > described above.
> > > > > > >
> > > > > > > Regards
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > Sent: 23 May 2012 00:05
> > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > Nope, from the document, the limit is set on whole vm:
> > > > > > > CPU Limits
> > > > > > >
> > > > > > > When a CPU Limit is set on a virtual machine resource
> > settings,
> > > > the
> > > > > > > virtual machine is deliberately held from being scheduled
> to
> > a
> > > > PCPU
> > > > > > > when it has used up its allocated CPU resource. This
> happens
> > > > > > > regardless of the CPU utilization. If the limit is set to
> > > > > > > 500MHz, the virtual machine is descheduled from the PCPU
> and
> > has
> > > > > > > to wait before
> > > > > > it
> > > > > > > is allowed to be scheduled again. As such, the virtual
> > machine
> > > > might
> > > > > > > experience performance degradation.
> > > > > > >
> > > > > > > Note: For an SMP virtual machine, the sum of all vCPUs
> cannot
> > > > exceed
> > > > > > > the specified limit. For example, 4 vCPU virtual machine
> with
> > a
> > > > > > > limit of 1200MHz and equal load among vCPUs would result in
> a
> > > > > > > max
> > > > of
> > > > > > > 300MHz per vCPU.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&pop
> > > > > u
> > > > > p
> > > > > > > =
> > > > > > > true&languageId=&externalID=1033115
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > Sent: Tuesday, May 22, 2012 3:43 PM
> > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > No I don't think this is a bug. When you set 1000Mhz as
> CPU
> > > > > > > > cap
> > > > > > that
> > > > > > > > is meant per core. vmWare will limit each CPU core to
> > 1000Mhz.
> > > > > > > > As you gave 2 CPU cores that is 2000Mhz effective. That
> is
> > how
> > > > > > > > vmware works.
> > > > > > > >
> > > > > > > > I have setup my offerings all to 1000Mhz as speed and
> just
> > > > > > > > increasing the number of cores.
> > > > > > > > 1 core ends up being 1x1000Mhz
> > > > > > > > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > > > > > > > ...
> > > > > > > >
> > > > > > > > I'm actually using it in production and works quite well
> as
> > > > > > > Cloudstack
> > > > > > > > "allocates" 4000Mhz when I'm using 4x1000Mhz cores and
> > vmware
> > > > > > > cleverly
> > > > > > > > balances between the cores as you put load on it and not
> > > > letting
> > > > > > any
> > > > > > > > of the cores above the set limit of 1000Mhz.
> > > > > > > >
> > > > > > > >
> > > > > > > > Regards
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Diego Spinola Castro
> [mailto:spinolacastro@gmail.com]
> > > > > > > > Sent: 22 May 2012 19:34
> > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > Subject: Re: Vmware CPU Cap
> > > > > > > >
> > > > > > > > I forgot the link:
> > > > > > > > http://imageshack.us/photo/my-images/848/cpulimit.png/
> > > > > > > >
> > > > > > > > 2012/5/22 Diego Spinola Castro <sp...@gmail.com>
> > > > > > > >
> > > > > > > > > I believe that is a bug with cpu cap and vmware.
> > > > > > > > >
> > > > > > > > > To reproduce:
> > > > > > > > >
> > > > > > > > > Create a offering with 2 cores and 1000mhz.
> > > > > > > > > Enable CPU CAP.
> > > > > > > > >
> > > > > > > > > After created instance , cs create a vm with 2 cores
> and
> > > > 1000mhz
> > > > > > > > > of
> > > > > > > > limit.
> > > > > > > > >
> > > > > > > > > I don't know for sure if is a bug, but vmware gives
> > 1000mhz
> > > > > > shared
> > > > > > > > > with cores.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Diego
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
RE: Vmware CPU Cap
Posted by Anthony Xu <Xu...@citrix.com>.
I'm pretty sure we have that before we redesign allocator.
Anthony
> -----Original Message-----
> From: Alex Huang [mailto:Alex.Huang@citrix.com]
> Sent: Wednesday, May 23, 2012 1:44 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> I'm surprised. I definitely recall having that conversation before
> about never exceeding the physical limits of the hypervisor. I'm
> pretty sure we have test cases against that as well.
>
> --Alex
>
> > -----Original Message-----
> > From: Edison Su [mailto:Edison.su@citrix.com]
> > Sent: Wednesday, May 23, 2012 1:39 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > I think it's a general issue for all the hypervisors we supported.
> >
> > > -----Original Message-----
> > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > Sent: Wednesday, May 23, 2012 1:32 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > That's because no one wrote a host allocator for VmWare.
> > >
> > > The relationship between DeploymentPlanner and HostAllocator and
> > > StoragePoolAllocator is as follows:
> > >
> > > DeploymentPlanner deals with heuristics and affinity rules of
> placing
> > > a VM. Once it determines a set of hosts that matches, it then asks
> > > the HostAllocator to work on the limitations of the type of
> hypervisor.
> > > There was never a HostAllocator written for VmWare. It just uses
> the
> > > one that was written originally for XenServer. Hence the bug.
> > >
> > > There's similar differentiation for DeploymentPlanner and
> > > StoragePoolAllocator and similar bugs exists, especially if someone
> > > adds a completely new type of storage pool but decides to just
> reuse
> > > the current set of StoragePoolAllocators.
> > >
> > > --Alex
> > >
> > > > -----Original Message-----
> > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > Sent: Wednesday, May 23, 2012 1:24 PM
> > > > To: cloudstack-dev@incubator.apache.org; 'tamasm@veber.co.uk'
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > Thanks for your input, I find another bug in cloudstack...
> > > > Let's dive little bit deeper into how the CPU allocation works.
> > > > VMware(the same for other hypervisors(Xen/KVM)) uses proportional
> > > share
> > > > based scheduling algorithm([1],[2]). It means "If a virtual
> machine
> > > has twice
> > > > as many shares of a resource as another virtual machine, it is
> > > entitled to
> > > > consume twice as much of that resource when these two virtual
> > > machines
> > > > are competing for resources."
> > > > There are two questions:
> > > > 1. Where is the share coming from? In Cloudstack, the share is
> > > calculated
> > > > from (CPU MHz * CPU cores).
> > > > 2. How the share of a VM is mapped to the physical CPU by the
> > > hypervisor?
> > > > Of cause, it depends on the hypervisor implementation, but there
> are
> > > some
> > > > general rules that we need to follow, as we are living in the
> real
> > > world:)
> > > > Rule 1: num of cpu core of a VM should be <= num of cpu core
> of
> > > the
> > > > hypervisor host. It doesn't make sense to running a 12 core VM on
> a
> > > single
> > > > core host, even some hypervisors, such as KVM, can do that.
> > > Hypervisor will
> > > > be busy at scheduling VCPU(the VCPU context switch is much
> heavier
> > > than
> > > > process context switch), thus bad performance for the VM.
> > > > Rule 2: The frequency of a VCPU should be <= the frequency of
> the
> > > host
> > > > hypervisor host. Can you run an one core * 12Ghz VM on a 3Ghz * 4
> > > core
> > > > physical CPU? Nope, in an unit time, the max freq of VCPU can get
> is
> > > the max
> > > > freq of physical cpu core, that's the physical LAW.
> > > > Rule 3: the total share of VMs <= total share of host
> > > hypervisor(in case of
> > > > no CPU overcommit).
> > > >
> > > > Currently, in cloudstack host allocator, only rule 3 is taken
> into
> > > consideration.
> > > >
> > > > [1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM-resource.pdf
> > > > [2]
> > > >
> > http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf
> > > >
> > > > > -----Original Message-----
> > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > Sent: Tuesday, May 22, 2012 5:41 PM
> > > > > To: cloudstack-dev@incubator.apache.org
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > Hi,
> > > > >
> > > > > I still don't think this is an issue as the CPU Mhz limit and
> the
> > > > > number of cores are independent.
> > > > > CPU manufacturers sell 2,4,6 cores at 3Ghz and not 6,12,18Ghz
> CPUs.
> > > > >
> > > > > So I think it is good how it works but the
> "number_of_cores*Mhz"
> > > while
> > > > > allocating should not multiply so that is the bug :) What
> vmware
> > > does
> > > > > with multiplying the cores with the core speed is bad as I
> can't
> > > have
> > > > > a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know where I'm
> > > coming
> > > > > from....
> > > > >
> > > > > Regards
> > > > >
> > > > > -----Original Message-----
> > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > Sent: 23 May 2012 01:27
> > > > > To: cloudstack-dev@incubator.apache.org
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > Hi,
> > > > >
> > > > > I have confused myself too because if I have a look the
> database
> > > and
> > > > > dump the service_offerings table the limit for me is set to
> > > > > 2000,3000,4000 and not 1000.
> > > > > So a Mhz limit set to 4000 and 4 cores will end up as a quad
> core
> > > box
> > > > > at 4Ghz. I remember now I had to set CPU overprovisioning to 4
> as
> > > > > CloudStack took away 4x4Ghz=16Ghz of the available CPU.
> > > > > So Cloudstack sets the VM CPU Mhz limit what the actual limit
> is
> > > set
> > > > > to in the offering but does not multiply the Mhz limit by the
> > > number
> > > > > of cores when setting the limit on vmware. However takes away
> > > > > "number_of_cores*Mhz limit" from the available CPU capacity
> when
> > > > > allocating.
> > > > >
> > > > > I'm getting confused myself so I'm not sure if this is bug now
> or
> > > not
> > > > > either :) I'm using 3.0.3
> > > > >
> > > > > Regards
> > > > >
> > > > > -----Original Message-----
> > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > Sent: 23 May 2012 00:32
> > > > > To: cloudstack-dev@incubator.apache.org; Tamas Monos
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > But in Diego'case, the limit is set as 1000Mhz, while his
> service
> > > > > offering is 1000Mhz * 2 cores.
> > > > > Which version of cloudstack are you using? Maybe it's a
> regression.
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > Sent: Tuesday, May 22, 2012 4:26 PM
> > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I have a service offering with 1000Mhz limit and 4 cpu cores.
> > > That
> > > > > > sums up to 4000Mhz.
> > > > > > Cloudstack sets the limit to 4Ghz on the virtual machine and
> > > > > > when
> > > I
> > > > > > put load on it vmware balances the load between the 4 cores
> > > allowing
> > > > > > them to use 1000Mhz each.
> > > > > > I do not see any bugs here.
> > > > > >
> > > > > > Apologies if the "meant per CPU core" was incorrect. What I
> > > > > > meant
> > > is
> > > > > > described above.
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > Sent: 23 May 2012 00:05
> > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > Nope, from the document, the limit is set on whole vm:
> > > > > > CPU Limits
> > > > > >
> > > > > > When a CPU Limit is set on a virtual machine resource
> settings,
> > > the
> > > > > > virtual machine is deliberately held from being scheduled to
> a
> > > PCPU
> > > > > > when it has used up its allocated CPU resource. This happens
> > > > > > regardless of the CPU utilization. If the limit is set to
> > > > > > 500MHz, the virtual machine is descheduled from the PCPU and
> has
> > > > > > to wait before
> > > > > it
> > > > > > is allowed to be scheduled again. As such, the virtual
> machine
> > > might
> > > > > > experience performance degradation.
> > > > > >
> > > > > > Note: For an SMP virtual machine, the sum of all vCPUs cannot
> > > exceed
> > > > > > the specified limit. For example, 4 vCPU virtual machine with
> a
> > > > > > limit of 1200MHz and equal load among vCPUs would result in a
> > > > > > max
> > > of
> > > > > > 300MHz per vCPU.
> > > > > >
> > > > > >
> > > > >
> > > >
> > http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&pop
> > > > u
> > > > p
> > > > > > =
> > > > > > true&languageId=&externalID=1033115
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > Sent: Tuesday, May 22, 2012 3:43 PM
> > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > No I don't think this is a bug. When you set 1000Mhz as CPU
> > > > > > > cap
> > > > > that
> > > > > > > is meant per core. vmWare will limit each CPU core to
> 1000Mhz.
> > > > > > > As you gave 2 CPU cores that is 2000Mhz effective. That is
> how
> > > > > > > vmware works.
> > > > > > >
> > > > > > > I have setup my offerings all to 1000Mhz as speed and just
> > > > > > > increasing the number of cores.
> > > > > > > 1 core ends up being 1x1000Mhz
> > > > > > > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > > > > > > ...
> > > > > > >
> > > > > > > I'm actually using it in production and works quite well as
> > > > > > Cloudstack
> > > > > > > "allocates" 4000Mhz when I'm using 4x1000Mhz cores and
> vmware
> > > > > > cleverly
> > > > > > > balances between the cores as you put load on it and not
> > > letting
> > > > > any
> > > > > > > of the cores above the set limit of 1000Mhz.
> > > > > > >
> > > > > > >
> > > > > > > Regards
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
> > > > > > > Sent: 22 May 2012 19:34
> > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > Subject: Re: Vmware CPU Cap
> > > > > > >
> > > > > > > I forgot the link:
> > > > > > > http://imageshack.us/photo/my-images/848/cpulimit.png/
> > > > > > >
> > > > > > > 2012/5/22 Diego Spinola Castro <sp...@gmail.com>
> > > > > > >
> > > > > > > > I believe that is a bug with cpu cap and vmware.
> > > > > > > >
> > > > > > > > To reproduce:
> > > > > > > >
> > > > > > > > Create a offering with 2 cores and 1000mhz.
> > > > > > > > Enable CPU CAP.
> > > > > > > >
> > > > > > > > After created instance , cs create a vm with 2 cores and
> > > 1000mhz
> > > > > > > > of
> > > > > > > limit.
> > > > > > > >
> > > > > > > > I don't know for sure if is a bug, but vmware gives
> 1000mhz
> > > > > shared
> > > > > > > > with cores.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Diego
> > > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
RE: Vmware CPU Cap
Posted by Alex Huang <Al...@citrix.com>.
I'm surprised. I definitely recall having that conversation before about never exceeding the physical limits of the hypervisor. I'm pretty sure we have test cases against that as well.
--Alex
> -----Original Message-----
> From: Edison Su [mailto:Edison.su@citrix.com]
> Sent: Wednesday, May 23, 2012 1:39 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> I think it's a general issue for all the hypervisors we supported.
>
> > -----Original Message-----
> > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > Sent: Wednesday, May 23, 2012 1:32 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > That's because no one wrote a host allocator for VmWare.
> >
> > The relationship between DeploymentPlanner and HostAllocator and
> > StoragePoolAllocator is as follows:
> >
> > DeploymentPlanner deals with heuristics and affinity rules of placing
> > a VM. Once it determines a set of hosts that matches, it then asks
> > the HostAllocator to work on the limitations of the type of hypervisor.
> > There was never a HostAllocator written for VmWare. It just uses the
> > one that was written originally for XenServer. Hence the bug.
> >
> > There's similar differentiation for DeploymentPlanner and
> > StoragePoolAllocator and similar bugs exists, especially if someone
> > adds a completely new type of storage pool but decides to just reuse
> > the current set of StoragePoolAllocators.
> >
> > --Alex
> >
> > > -----Original Message-----
> > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > Sent: Wednesday, May 23, 2012 1:24 PM
> > > To: cloudstack-dev@incubator.apache.org; 'tamasm@veber.co.uk'
> > > Subject: RE: Vmware CPU Cap
> > >
> > > Thanks for your input, I find another bug in cloudstack...
> > > Let's dive little bit deeper into how the CPU allocation works.
> > > VMware(the same for other hypervisors(Xen/KVM)) uses proportional
> > share
> > > based scheduling algorithm([1],[2]). It means "If a virtual machine
> > has twice
> > > as many shares of a resource as another virtual machine, it is
> > entitled to
> > > consume twice as much of that resource when these two virtual
> > machines
> > > are competing for resources."
> > > There are two questions:
> > > 1. Where is the share coming from? In Cloudstack, the share is
> > calculated
> > > from (CPU MHz * CPU cores).
> > > 2. How the share of a VM is mapped to the physical CPU by the
> > hypervisor?
> > > Of cause, it depends on the hypervisor implementation, but there are
> > some
> > > general rules that we need to follow, as we are living in the real
> > world:)
> > > Rule 1: num of cpu core of a VM should be <= num of cpu core of
> > the
> > > hypervisor host. It doesn't make sense to running a 12 core VM on a
> > single
> > > core host, even some hypervisors, such as KVM, can do that.
> > Hypervisor will
> > > be busy at scheduling VCPU(the VCPU context switch is much heavier
> > than
> > > process context switch), thus bad performance for the VM.
> > > Rule 2: The frequency of a VCPU should be <= the frequency of the
> > host
> > > hypervisor host. Can you run an one core * 12Ghz VM on a 3Ghz * 4
> > core
> > > physical CPU? Nope, in an unit time, the max freq of VCPU can get is
> > the max
> > > freq of physical cpu core, that's the physical LAW.
> > > Rule 3: the total share of VMs <= total share of host
> > hypervisor(in case of
> > > no CPU overcommit).
> > >
> > > Currently, in cloudstack host allocator, only rule 3 is taken into
> > consideration.
> > >
> > > [1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM-resource.pdf
> > > [2]
> > >
> http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf
> > >
> > > > -----Original Message-----
> > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > Sent: Tuesday, May 22, 2012 5:41 PM
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > Hi,
> > > >
> > > > I still don't think this is an issue as the CPU Mhz limit and the
> > > > number of cores are independent.
> > > > CPU manufacturers sell 2,4,6 cores at 3Ghz and not 6,12,18Ghz CPUs.
> > > >
> > > > So I think it is good how it works but the "number_of_cores*Mhz"
> > while
> > > > allocating should not multiply so that is the bug :) What vmware
> > does
> > > > with multiplying the cores with the core speed is bad as I can't
> > have
> > > > a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know where I'm
> > coming
> > > > from....
> > > >
> > > > Regards
> > > >
> > > > -----Original Message-----
> > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > Sent: 23 May 2012 01:27
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > Hi,
> > > >
> > > > I have confused myself too because if I have a look the database
> > and
> > > > dump the service_offerings table the limit for me is set to
> > > > 2000,3000,4000 and not 1000.
> > > > So a Mhz limit set to 4000 and 4 cores will end up as a quad core
> > box
> > > > at 4Ghz. I remember now I had to set CPU overprovisioning to 4 as
> > > > CloudStack took away 4x4Ghz=16Ghz of the available CPU.
> > > > So Cloudstack sets the VM CPU Mhz limit what the actual limit is
> > set
> > > > to in the offering but does not multiply the Mhz limit by the
> > number
> > > > of cores when setting the limit on vmware. However takes away
> > > > "number_of_cores*Mhz limit" from the available CPU capacity when
> > > > allocating.
> > > >
> > > > I'm getting confused myself so I'm not sure if this is bug now or
> > not
> > > > either :) I'm using 3.0.3
> > > >
> > > > Regards
> > > >
> > > > -----Original Message-----
> > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > Sent: 23 May 2012 00:32
> > > > To: cloudstack-dev@incubator.apache.org; Tamas Monos
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > But in Diego'case, the limit is set as 1000Mhz, while his service
> > > > offering is 1000Mhz * 2 cores.
> > > > Which version of cloudstack are you using? Maybe it's a regression.
> > > >
> > > > > -----Original Message-----
> > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > Sent: Tuesday, May 22, 2012 4:26 PM
> > > > > To: cloudstack-dev@incubator.apache.org
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > Hi,
> > > > >
> > > > > I have a service offering with 1000Mhz limit and 4 cpu cores.
> > That
> > > > > sums up to 4000Mhz.
> > > > > Cloudstack sets the limit to 4Ghz on the virtual machine and
> > > > > when
> > I
> > > > > put load on it vmware balances the load between the 4 cores
> > allowing
> > > > > them to use 1000Mhz each.
> > > > > I do not see any bugs here.
> > > > >
> > > > > Apologies if the "meant per CPU core" was incorrect. What I
> > > > > meant
> > is
> > > > > described above.
> > > > >
> > > > > Regards
> > > > >
> > > > > -----Original Message-----
> > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > Sent: 23 May 2012 00:05
> > > > > To: cloudstack-dev@incubator.apache.org
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > Nope, from the document, the limit is set on whole vm:
> > > > > CPU Limits
> > > > >
> > > > > When a CPU Limit is set on a virtual machine resource settings,
> > the
> > > > > virtual machine is deliberately held from being scheduled to a
> > PCPU
> > > > > when it has used up its allocated CPU resource. This happens
> > > > > regardless of the CPU utilization. If the limit is set to
> > > > > 500MHz, the virtual machine is descheduled from the PCPU and has
> > > > > to wait before
> > > > it
> > > > > is allowed to be scheduled again. As such, the virtual machine
> > might
> > > > > experience performance degradation.
> > > > >
> > > > > Note: For an SMP virtual machine, the sum of all vCPUs cannot
> > exceed
> > > > > the specified limit. For example, 4 vCPU virtual machine with a
> > > > > limit of 1200MHz and equal load among vCPUs would result in a
> > > > > max
> > of
> > > > > 300MHz per vCPU.
> > > > >
> > > > >
> > > >
> > >
> http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&pop
> > > u
> > > p
> > > > > =
> > > > > true&languageId=&externalID=1033115
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > Sent: Tuesday, May 22, 2012 3:43 PM
> > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > No I don't think this is a bug. When you set 1000Mhz as CPU
> > > > > > cap
> > > > that
> > > > > > is meant per core. vmWare will limit each CPU core to 1000Mhz.
> > > > > > As you gave 2 CPU cores that is 2000Mhz effective. That is how
> > > > > > vmware works.
> > > > > >
> > > > > > I have setup my offerings all to 1000Mhz as speed and just
> > > > > > increasing the number of cores.
> > > > > > 1 core ends up being 1x1000Mhz
> > > > > > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > > > > > ...
> > > > > >
> > > > > > I'm actually using it in production and works quite well as
> > > > > Cloudstack
> > > > > > "allocates" 4000Mhz when I'm using 4x1000Mhz cores and vmware
> > > > > cleverly
> > > > > > balances between the cores as you put load on it and not
> > letting
> > > > any
> > > > > > of the cores above the set limit of 1000Mhz.
> > > > > >
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
> > > > > > Sent: 22 May 2012 19:34
> > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > Subject: Re: Vmware CPU Cap
> > > > > >
> > > > > > I forgot the link:
> > > > > > http://imageshack.us/photo/my-images/848/cpulimit.png/
> > > > > >
> > > > > > 2012/5/22 Diego Spinola Castro <sp...@gmail.com>
> > > > > >
> > > > > > > I believe that is a bug with cpu cap and vmware.
> > > > > > >
> > > > > > > To reproduce:
> > > > > > >
> > > > > > > Create a offering with 2 cores and 1000mhz.
> > > > > > > Enable CPU CAP.
> > > > > > >
> > > > > > > After created instance , cs create a vm with 2 cores and
> > 1000mhz
> > > > > > > of
> > > > > > limit.
> > > > > > >
> > > > > > > I don't know for sure if is a bug, but vmware gives 1000mhz
> > > > shared
> > > > > > > with cores.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Diego
> > > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
RE: Vmware CPU Cap
Posted by Edison Su <Ed...@citrix.com>.
I think it's a general issue for all the hypervisors we supported.
> -----Original Message-----
> From: Alex Huang [mailto:Alex.Huang@citrix.com]
> Sent: Wednesday, May 23, 2012 1:32 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> That's because no one wrote a host allocator for VmWare.
>
> The relationship between DeploymentPlanner and HostAllocator and
> StoragePoolAllocator is as follows:
>
> DeploymentPlanner deals with heuristics and affinity rules of placing a
> VM. Once it determines a set of hosts that matches, it then asks the
> HostAllocator to work on the limitations of the type of hypervisor.
> There was never a HostAllocator written for VmWare. It just uses the
> one that was written originally for XenServer. Hence the bug.
>
> There's similar differentiation for DeploymentPlanner and
> StoragePoolAllocator and similar bugs exists, especially if someone
> adds a completely new type of storage pool but decides to just reuse
> the current set of StoragePoolAllocators.
>
> --Alex
>
> > -----Original Message-----
> > From: Edison Su [mailto:Edison.su@citrix.com]
> > Sent: Wednesday, May 23, 2012 1:24 PM
> > To: cloudstack-dev@incubator.apache.org; 'tamasm@veber.co.uk'
> > Subject: RE: Vmware CPU Cap
> >
> > Thanks for your input, I find another bug in cloudstack...
> > Let's dive little bit deeper into how the CPU allocation works.
> > VMware(the same for other hypervisors(Xen/KVM)) uses proportional
> share
> > based scheduling algorithm([1],[2]). It means "If a virtual machine
> has twice
> > as many shares of a resource as another virtual machine, it is
> entitled to
> > consume twice as much of that resource when these two virtual
> machines
> > are competing for resources."
> > There are two questions:
> > 1. Where is the share coming from? In Cloudstack, the share is
> calculated
> > from (CPU MHz * CPU cores).
> > 2. How the share of a VM is mapped to the physical CPU by the
> hypervisor?
> > Of cause, it depends on the hypervisor implementation, but there are
> some
> > general rules that we need to follow, as we are living in the real
> world:)
> > Rule 1: num of cpu core of a VM should be <= num of cpu core of
> the
> > hypervisor host. It doesn't make sense to running a 12 core VM on a
> single
> > core host, even some hypervisors, such as KVM, can do that.
> Hypervisor will
> > be busy at scheduling VCPU(the VCPU context switch is much heavier
> than
> > process context switch), thus bad performance for the VM.
> > Rule 2: The frequency of a VCPU should be <= the frequency of the
> host
> > hypervisor host. Can you run an one core * 12Ghz VM on a 3Ghz * 4
> core
> > physical CPU? Nope, in an unit time, the max freq of VCPU can get is
> the max
> > freq of physical cpu core, that's the physical LAW.
> > Rule 3: the total share of VMs <= total share of host
> hypervisor(in case of
> > no CPU overcommit).
> >
> > Currently, in cloudstack host allocator, only rule 3 is taken into
> consideration.
> >
> > [1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM-resource.pdf
> > [2]
> > http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf
> >
> > > -----Original Message-----
> > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > Sent: Tuesday, May 22, 2012 5:41 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > Hi,
> > >
> > > I still don't think this is an issue as the CPU Mhz limit and the
> > > number of cores are independent.
> > > CPU manufacturers sell 2,4,6 cores at 3Ghz and not 6,12,18Ghz CPUs.
> > >
> > > So I think it is good how it works but the "number_of_cores*Mhz"
> while
> > > allocating should not multiply so that is the bug :) What vmware
> does
> > > with multiplying the cores with the core speed is bad as I can't
> have
> > > a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know where I'm
> coming
> > > from....
> > >
> > > Regards
> > >
> > > -----Original Message-----
> > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > Sent: 23 May 2012 01:27
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > Hi,
> > >
> > > I have confused myself too because if I have a look the database
> and
> > > dump the service_offerings table the limit for me is set to
> > > 2000,3000,4000 and not 1000.
> > > So a Mhz limit set to 4000 and 4 cores will end up as a quad core
> box
> > > at 4Ghz. I remember now I had to set CPU overprovisioning to 4 as
> > > CloudStack took away 4x4Ghz=16Ghz of the available CPU.
> > > So Cloudstack sets the VM CPU Mhz limit what the actual limit is
> set
> > > to in the offering but does not multiply the Mhz limit by the
> number
> > > of cores when setting the limit on vmware. However takes away
> > > "number_of_cores*Mhz limit" from the available CPU capacity when
> > > allocating.
> > >
> > > I'm getting confused myself so I'm not sure if this is bug now or
> not
> > > either :) I'm using 3.0.3
> > >
> > > Regards
> > >
> > > -----Original Message-----
> > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > Sent: 23 May 2012 00:32
> > > To: cloudstack-dev@incubator.apache.org; Tamas Monos
> > > Subject: RE: Vmware CPU Cap
> > >
> > > But in Diego'case, the limit is set as 1000Mhz, while his service
> > > offering is 1000Mhz * 2 cores.
> > > Which version of cloudstack are you using? Maybe it's a regression.
> > >
> > > > -----Original Message-----
> > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > Sent: Tuesday, May 22, 2012 4:26 PM
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > Hi,
> > > >
> > > > I have a service offering with 1000Mhz limit and 4 cpu cores.
> That
> > > > sums up to 4000Mhz.
> > > > Cloudstack sets the limit to 4Ghz on the virtual machine and when
> I
> > > > put load on it vmware balances the load between the 4 cores
> allowing
> > > > them to use 1000Mhz each.
> > > > I do not see any bugs here.
> > > >
> > > > Apologies if the "meant per CPU core" was incorrect. What I meant
> is
> > > > described above.
> > > >
> > > > Regards
> > > >
> > > > -----Original Message-----
> > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > Sent: 23 May 2012 00:05
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > Nope, from the document, the limit is set on whole vm:
> > > > CPU Limits
> > > >
> > > > When a CPU Limit is set on a virtual machine resource settings,
> the
> > > > virtual machine is deliberately held from being scheduled to a
> PCPU
> > > > when it has used up its allocated CPU resource. This happens
> > > > regardless of the CPU utilization. If the limit is set to 500MHz,
> > > > the virtual machine is descheduled from the PCPU and has to wait
> > > > before
> > > it
> > > > is allowed to be scheduled again. As such, the virtual machine
> might
> > > > experience performance degradation.
> > > >
> > > > Note: For an SMP virtual machine, the sum of all vCPUs cannot
> exceed
> > > > the specified limit. For example, 4 vCPU virtual machine with a
> > > > limit of 1200MHz and equal load among vCPUs would result in a max
> of
> > > > 300MHz per vCPU.
> > > >
> > > >
> > >
> > http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&popu
> > p
> > > > =
> > > > true&languageId=&externalID=1033115
> > > >
> > > > > -----Original Message-----
> > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > Sent: Tuesday, May 22, 2012 3:43 PM
> > > > > To: cloudstack-dev@incubator.apache.org
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > Hi,
> > > > >
> > > > > No I don't think this is a bug. When you set 1000Mhz as CPU cap
> > > that
> > > > > is meant per core. vmWare will limit each CPU core to 1000Mhz.
> > > > > As you gave 2 CPU cores that is 2000Mhz effective. That is how
> > > > > vmware works.
> > > > >
> > > > > I have setup my offerings all to 1000Mhz as speed and just
> > > > > increasing the number of cores.
> > > > > 1 core ends up being 1x1000Mhz
> > > > > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > > > > ...
> > > > >
> > > > > I'm actually using it in production and works quite well as
> > > > Cloudstack
> > > > > "allocates" 4000Mhz when I'm using 4x1000Mhz cores and vmware
> > > > cleverly
> > > > > balances between the cores as you put load on it and not
> letting
> > > any
> > > > > of the cores above the set limit of 1000Mhz.
> > > > >
> > > > >
> > > > > Regards
> > > > >
> > > > > -----Original Message-----
> > > > > From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
> > > > > Sent: 22 May 2012 19:34
> > > > > To: cloudstack-dev@incubator.apache.org
> > > > > Subject: Re: Vmware CPU Cap
> > > > >
> > > > > I forgot the link:
> > > > > http://imageshack.us/photo/my-images/848/cpulimit.png/
> > > > >
> > > > > 2012/5/22 Diego Spinola Castro <sp...@gmail.com>
> > > > >
> > > > > > I believe that is a bug with cpu cap and vmware.
> > > > > >
> > > > > > To reproduce:
> > > > > >
> > > > > > Create a offering with 2 cores and 1000mhz.
> > > > > > Enable CPU CAP.
> > > > > >
> > > > > > After created instance , cs create a vm with 2 cores and
> 1000mhz
> > > > > > of
> > > > > limit.
> > > > > >
> > > > > > I don't know for sure if is a bug, but vmware gives 1000mhz
> > > shared
> > > > > > with cores.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Diego
> > > > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
RE: Vmware CPU Cap
Posted by Alex Huang <Al...@citrix.com>.
That's because no one wrote a host allocator for VmWare.
The relationship between DeploymentPlanner and HostAllocator and StoragePoolAllocator is as follows:
DeploymentPlanner deals with heuristics and affinity rules of placing a VM. Once it determines a set of hosts that matches, it then asks the HostAllocator to work on the limitations of the type of hypervisor. There was never a HostAllocator written for VmWare. It just uses the one that was written originally for XenServer. Hence the bug.
There's similar differentiation for DeploymentPlanner and StoragePoolAllocator and similar bugs exists, especially if someone adds a completely new type of storage pool but decides to just reuse the current set of StoragePoolAllocators.
--Alex
> -----Original Message-----
> From: Edison Su [mailto:Edison.su@citrix.com]
> Sent: Wednesday, May 23, 2012 1:24 PM
> To: cloudstack-dev@incubator.apache.org; 'tamasm@veber.co.uk'
> Subject: RE: Vmware CPU Cap
>
> Thanks for your input, I find another bug in cloudstack...
> Let's dive little bit deeper into how the CPU allocation works.
> VMware(the same for other hypervisors(Xen/KVM)) uses proportional share
> based scheduling algorithm([1],[2]). It means "If a virtual machine has twice
> as many shares of a resource as another virtual machine, it is entitled to
> consume twice as much of that resource when these two virtual machines
> are competing for resources."
> There are two questions:
> 1. Where is the share coming from? In Cloudstack, the share is calculated
> from (CPU MHz * CPU cores).
> 2. How the share of a VM is mapped to the physical CPU by the hypervisor?
> Of cause, it depends on the hypervisor implementation, but there are some
> general rules that we need to follow, as we are living in the real world:)
> Rule 1: num of cpu core of a VM should be <= num of cpu core of the
> hypervisor host. It doesn't make sense to running a 12 core VM on a single
> core host, even some hypervisors, such as KVM, can do that. Hypervisor will
> be busy at scheduling VCPU(the VCPU context switch is much heavier than
> process context switch), thus bad performance for the VM.
> Rule 2: The frequency of a VCPU should be <= the frequency of the host
> hypervisor host. Can you run an one core * 12Ghz VM on a 3Ghz * 4 core
> physical CPU? Nope, in an unit time, the max freq of VCPU can get is the max
> freq of physical cpu core, that's the physical LAW.
> Rule 3: the total share of VMs <= total share of host hypervisor(in case of
> no CPU overcommit).
>
> Currently, in cloudstack host allocator, only rule 3 is taken into consideration.
>
> [1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM-resource.pdf
> [2]
> http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf
>
> > -----Original Message-----
> > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > Sent: Tuesday, May 22, 2012 5:41 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > Hi,
> >
> > I still don't think this is an issue as the CPU Mhz limit and the
> > number of cores are independent.
> > CPU manufacturers sell 2,4,6 cores at 3Ghz and not 6,12,18Ghz CPUs.
> >
> > So I think it is good how it works but the "number_of_cores*Mhz" while
> > allocating should not multiply so that is the bug :) What vmware does
> > with multiplying the cores with the core speed is bad as I can't have
> > a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know where I'm coming
> > from....
> >
> > Regards
> >
> > -----Original Message-----
> > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > Sent: 23 May 2012 01:27
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > Hi,
> >
> > I have confused myself too because if I have a look the database and
> > dump the service_offerings table the limit for me is set to
> > 2000,3000,4000 and not 1000.
> > So a Mhz limit set to 4000 and 4 cores will end up as a quad core box
> > at 4Ghz. I remember now I had to set CPU overprovisioning to 4 as
> > CloudStack took away 4x4Ghz=16Ghz of the available CPU.
> > So Cloudstack sets the VM CPU Mhz limit what the actual limit is set
> > to in the offering but does not multiply the Mhz limit by the number
> > of cores when setting the limit on vmware. However takes away
> > "number_of_cores*Mhz limit" from the available CPU capacity when
> > allocating.
> >
> > I'm getting confused myself so I'm not sure if this is bug now or not
> > either :) I'm using 3.0.3
> >
> > Regards
> >
> > -----Original Message-----
> > From: Edison Su [mailto:Edison.su@citrix.com]
> > Sent: 23 May 2012 00:32
> > To: cloudstack-dev@incubator.apache.org; Tamas Monos
> > Subject: RE: Vmware CPU Cap
> >
> > But in Diego'case, the limit is set as 1000Mhz, while his service
> > offering is 1000Mhz * 2 cores.
> > Which version of cloudstack are you using? Maybe it's a regression.
> >
> > > -----Original Message-----
> > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > Sent: Tuesday, May 22, 2012 4:26 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > Hi,
> > >
> > > I have a service offering with 1000Mhz limit and 4 cpu cores. That
> > > sums up to 4000Mhz.
> > > Cloudstack sets the limit to 4Ghz on the virtual machine and when I
> > > put load on it vmware balances the load between the 4 cores allowing
> > > them to use 1000Mhz each.
> > > I do not see any bugs here.
> > >
> > > Apologies if the "meant per CPU core" was incorrect. What I meant is
> > > described above.
> > >
> > > Regards
> > >
> > > -----Original Message-----
> > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > Sent: 23 May 2012 00:05
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > Nope, from the document, the limit is set on whole vm:
> > > CPU Limits
> > >
> > > When a CPU Limit is set on a virtual machine resource settings, the
> > > virtual machine is deliberately held from being scheduled to a PCPU
> > > when it has used up its allocated CPU resource. This happens
> > > regardless of the CPU utilization. If the limit is set to 500MHz,
> > > the virtual machine is descheduled from the PCPU and has to wait
> > > before
> > it
> > > is allowed to be scheduled again. As such, the virtual machine might
> > > experience performance degradation.
> > >
> > > Note: For an SMP virtual machine, the sum of all vCPUs cannot exceed
> > > the specified limit. For example, 4 vCPU virtual machine with a
> > > limit of 1200MHz and equal load among vCPUs would result in a max of
> > > 300MHz per vCPU.
> > >
> > >
> >
> http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&popu
> p
> > > =
> > > true&languageId=&externalID=1033115
> > >
> > > > -----Original Message-----
> > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > Sent: Tuesday, May 22, 2012 3:43 PM
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > Hi,
> > > >
> > > > No I don't think this is a bug. When you set 1000Mhz as CPU cap
> > that
> > > > is meant per core. vmWare will limit each CPU core to 1000Mhz.
> > > > As you gave 2 CPU cores that is 2000Mhz effective. That is how
> > > > vmware works.
> > > >
> > > > I have setup my offerings all to 1000Mhz as speed and just
> > > > increasing the number of cores.
> > > > 1 core ends up being 1x1000Mhz
> > > > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > > > ...
> > > >
> > > > I'm actually using it in production and works quite well as
> > > Cloudstack
> > > > "allocates" 4000Mhz when I'm using 4x1000Mhz cores and vmware
> > > cleverly
> > > > balances between the cores as you put load on it and not letting
> > any
> > > > of the cores above the set limit of 1000Mhz.
> > > >
> > > >
> > > > Regards
> > > >
> > > > -----Original Message-----
> > > > From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
> > > > Sent: 22 May 2012 19:34
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: Re: Vmware CPU Cap
> > > >
> > > > I forgot the link:
> > > > http://imageshack.us/photo/my-images/848/cpulimit.png/
> > > >
> > > > 2012/5/22 Diego Spinola Castro <sp...@gmail.com>
> > > >
> > > > > I believe that is a bug with cpu cap and vmware.
> > > > >
> > > > > To reproduce:
> > > > >
> > > > > Create a offering with 2 cores and 1000mhz.
> > > > > Enable CPU CAP.
> > > > >
> > > > > After created instance , cs create a vm with 2 cores and 1000mhz
> > > > > of
> > > > limit.
> > > > >
> > > > > I don't know for sure if is a bug, but vmware gives 1000mhz
> > shared
> > > > > with cores.
> > > > >
> > > > >
> > > > >
> > > > > Diego
> > > > >
> > >
> > >
> >
> >
> >
> >
RE: Vmware CPU Cap
Posted by Edison Su <Ed...@citrix.com>.
Thanks for your input, I find another bug in cloudstack...
Let's dive little bit deeper into how the CPU allocation works.
VMware(the same for other hypervisors(Xen/KVM)) uses proportional share based scheduling algorithm([1],[2]). It means
"If a virtual machine has twice as many shares of a resource as another virtual machine, it is entitled to consume twice as much of that resource when these two virtual machines are competing for resources."
There are two questions:
1. Where is the share coming from? In Cloudstack, the share is calculated from (CPU MHz * CPU cores).
2. How the share of a VM is mapped to the physical CPU by the hypervisor? Of cause, it depends on the hypervisor implementation, but there are some general rules that we need to follow, as we are living in the real world:)
Rule 1: num of cpu core of a VM should be <= num of cpu core of the hypervisor host. It doesn't make sense to running a 12 core VM on a single core host, even some hypervisors, such as KVM, can do that. Hypervisor will be busy at scheduling VCPU(the VCPU context switch is much heavier than process context switch), thus bad performance for the VM.
Rule 2: The frequency of a VCPU should be <= the frequency of the host hypervisor host. Can you run an one core * 12Ghz VM on a 3Ghz * 4 core physical CPU? Nope, in an unit time, the max freq of VCPU can get is the max freq of physical cpu core, that's the physical LAW.
Rule 3: the total share of VMs <= total share of host hypervisor(in case of no CPU overcommit).
Currently, in cloudstack host allocator, only rule 3 is taken into consideration.
[1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM-resource.pdf
[2] http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf
> -----Original Message-----
> From: Tamas Monos [mailto:tamasm@veber.co.uk]
> Sent: Tuesday, May 22, 2012 5:41 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Hi,
>
> I still don't think this is an issue as the CPU Mhz limit and the
> number of cores are independent.
> CPU manufacturers sell 2,4,6 cores at 3Ghz and not 6,12,18Ghz CPUs.
>
> So I think it is good how it works but the "number_of_cores*Mhz" while
> allocating should not multiply so that is the bug :)
> What vmware does with multiplying the cores with the core speed is bad
> as I can't have a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know
> where I'm coming from....
>
> Regards
>
> -----Original Message-----
> From: Tamas Monos [mailto:tamasm@veber.co.uk]
> Sent: 23 May 2012 01:27
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Hi,
>
> I have confused myself too because if I have a look the database and
> dump the service_offerings table the limit for me is set to
> 2000,3000,4000 and not 1000.
> So a Mhz limit set to 4000 and 4 cores will end up as a quad core box
> at 4Ghz. I remember now I had to set CPU overprovisioning to 4 as
> CloudStack took away 4x4Ghz=16Ghz of the available CPU.
> So Cloudstack sets the VM CPU Mhz limit what the actual limit is set to
> in the offering but does not multiply the Mhz limit by the number of
> cores when setting the limit on vmware. However takes away
> "number_of_cores*Mhz limit" from the available CPU capacity when
> allocating.
>
> I'm getting confused myself so I'm not sure if this is bug now or not
> either :) I'm using 3.0.3
>
> Regards
>
> -----Original Message-----
> From: Edison Su [mailto:Edison.su@citrix.com]
> Sent: 23 May 2012 00:32
> To: cloudstack-dev@incubator.apache.org; Tamas Monos
> Subject: RE: Vmware CPU Cap
>
> But in Diego'case, the limit is set as 1000Mhz, while his service
> offering is 1000Mhz * 2 cores.
> Which version of cloudstack are you using? Maybe it's a regression.
>
> > -----Original Message-----
> > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > Sent: Tuesday, May 22, 2012 4:26 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > Hi,
> >
> > I have a service offering with 1000Mhz limit and 4 cpu cores. That
> > sums up to 4000Mhz.
> > Cloudstack sets the limit to 4Ghz on the virtual machine and when I
> > put load on it vmware balances the load between the 4 cores allowing
> > them to use 1000Mhz each.
> > I do not see any bugs here.
> >
> > Apologies if the "meant per CPU core" was incorrect. What I meant is
> > described above.
> >
> > Regards
> >
> > -----Original Message-----
> > From: Edison Su [mailto:Edison.su@citrix.com]
> > Sent: 23 May 2012 00:05
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > Nope, from the document, the limit is set on whole vm:
> > CPU Limits
> >
> > When a CPU Limit is set on a virtual machine resource settings, the
> > virtual machine is deliberately held from being scheduled to a PCPU
> > when it has used up its allocated CPU resource. This happens
> > regardless of the CPU utilization. If the limit is set to 500MHz, the
> > virtual machine is descheduled from the PCPU and has to wait before
> it
> > is allowed to be scheduled again. As such, the virtual machine might
> > experience performance degradation.
> >
> > Note: For an SMP virtual machine, the sum of all vCPUs cannot exceed
> > the specified limit. For example, 4 vCPU virtual machine with a limit
> > of 1200MHz and equal load among vCPUs would result in a max of 300MHz
> > per vCPU.
> >
> >
> http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&popup
> > =
> > true&languageId=&externalID=1033115
> >
> > > -----Original Message-----
> > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > Sent: Tuesday, May 22, 2012 3:43 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > Hi,
> > >
> > > No I don't think this is a bug. When you set 1000Mhz as CPU cap
> that
> > > is meant per core. vmWare will limit each CPU core to 1000Mhz.
> > > As you gave 2 CPU cores that is 2000Mhz effective. That is how
> > > vmware works.
> > >
> > > I have setup my offerings all to 1000Mhz as speed and just
> > > increasing the number of cores.
> > > 1 core ends up being 1x1000Mhz
> > > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > > ...
> > >
> > > I'm actually using it in production and works quite well as
> > Cloudstack
> > > "allocates" 4000Mhz when I'm using 4x1000Mhz cores and vmware
> > cleverly
> > > balances between the cores as you put load on it and not letting
> any
> > > of the cores above the set limit of 1000Mhz.
> > >
> > >
> > > Regards
> > >
> > > -----Original Message-----
> > > From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
> > > Sent: 22 May 2012 19:34
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: Re: Vmware CPU Cap
> > >
> > > I forgot the link:
> > > http://imageshack.us/photo/my-images/848/cpulimit.png/
> > >
> > > 2012/5/22 Diego Spinola Castro <sp...@gmail.com>
> > >
> > > > I believe that is a bug with cpu cap and vmware.
> > > >
> > > > To reproduce:
> > > >
> > > > Create a offering with 2 cores and 1000mhz.
> > > > Enable CPU CAP.
> > > >
> > > > After created instance , cs create a vm with 2 cores and 1000mhz
> > > > of
> > > limit.
> > > >
> > > > I don't know for sure if is a bug, but vmware gives 1000mhz
> shared
> > > > with cores.
> > > >
> > > >
> > > >
> > > > Diego
> > > >
> >
> >
>
>
>
>
RE: Vmware CPU Cap
Posted by Tamas Monos <ta...@veber.co.uk>.
Hi,
I still don't think this is an issue as the CPU Mhz limit and the number of cores are independent.
CPU manufacturers sell 2,4,6 cores at 3Ghz and not 6,12,18Ghz CPUs.
So I think it is good how it works but the "number_of_cores*Mhz" while allocating should not multiply so that is the bug :)
What vmware does with multiplying the cores with the core speed is bad as I can't have a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know where I'm coming from....
Regards
-----Original Message-----
From: Tamas Monos [mailto:tamasm@veber.co.uk]
Sent: 23 May 2012 01:27
To: cloudstack-dev@incubator.apache.org
Subject: RE: Vmware CPU Cap
Hi,
I have confused myself too because if I have a look the database and dump the service_offerings table the limit for me is set to 2000,3000,4000 and not 1000.
So a Mhz limit set to 4000 and 4 cores will end up as a quad core box at 4Ghz. I remember now I had to set CPU overprovisioning to 4 as CloudStack took away 4x4Ghz=16Ghz of the available CPU.
So Cloudstack sets the VM CPU Mhz limit what the actual limit is set to in the offering but does not multiply the Mhz limit by the number of cores when setting the limit on vmware. However takes away "number_of_cores*Mhz limit" from the available CPU capacity when allocating.
I'm getting confused myself so I'm not sure if this is bug now or not either :) I'm using 3.0.3
Regards
-----Original Message-----
From: Edison Su [mailto:Edison.su@citrix.com]
Sent: 23 May 2012 00:32
To: cloudstack-dev@incubator.apache.org; Tamas Monos
Subject: RE: Vmware CPU Cap
But in Diego'case, the limit is set as 1000Mhz, while his service offering is 1000Mhz * 2 cores.
Which version of cloudstack are you using? Maybe it's a regression.
> -----Original Message-----
> From: Tamas Monos [mailto:tamasm@veber.co.uk]
> Sent: Tuesday, May 22, 2012 4:26 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Hi,
>
> I have a service offering with 1000Mhz limit and 4 cpu cores. That
> sums up to 4000Mhz.
> Cloudstack sets the limit to 4Ghz on the virtual machine and when I
> put load on it vmware balances the load between the 4 cores allowing
> them to use 1000Mhz each.
> I do not see any bugs here.
>
> Apologies if the "meant per CPU core" was incorrect. What I meant is
> described above.
>
> Regards
>
> -----Original Message-----
> From: Edison Su [mailto:Edison.su@citrix.com]
> Sent: 23 May 2012 00:05
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Nope, from the document, the limit is set on whole vm:
> CPU Limits
>
> When a CPU Limit is set on a virtual machine resource settings, the
> virtual machine is deliberately held from being scheduled to a PCPU
> when it has used up its allocated CPU resource. This happens
> regardless of the CPU utilization. If the limit is set to 500MHz, the
> virtual machine is descheduled from the PCPU and has to wait before it
> is allowed to be scheduled again. As such, the virtual machine might
> experience performance degradation.
>
> Note: For an SMP virtual machine, the sum of all vCPUs cannot exceed
> the specified limit. For example, 4 vCPU virtual machine with a limit
> of 1200MHz and equal load among vCPUs would result in a max of 300MHz
> per vCPU.
>
> http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&popup
> =
> true&languageId=&externalID=1033115
>
> > -----Original Message-----
> > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > Sent: Tuesday, May 22, 2012 3:43 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > Hi,
> >
> > No I don't think this is a bug. When you set 1000Mhz as CPU cap that
> > is meant per core. vmWare will limit each CPU core to 1000Mhz.
> > As you gave 2 CPU cores that is 2000Mhz effective. That is how
> > vmware works.
> >
> > I have setup my offerings all to 1000Mhz as speed and just
> > increasing the number of cores.
> > 1 core ends up being 1x1000Mhz
> > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > ...
> >
> > I'm actually using it in production and works quite well as
> Cloudstack
> > "allocates" 4000Mhz when I'm using 4x1000Mhz cores and vmware
> cleverly
> > balances between the cores as you put load on it and not letting any
> > of the cores above the set limit of 1000Mhz.
> >
> >
> > Regards
> >
> > -----Original Message-----
> > From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
> > Sent: 22 May 2012 19:34
> > To: cloudstack-dev@incubator.apache.org
> > Subject: Re: Vmware CPU Cap
> >
> > I forgot the link:
> > http://imageshack.us/photo/my-images/848/cpulimit.png/
> >
> > 2012/5/22 Diego Spinola Castro <sp...@gmail.com>
> >
> > > I believe that is a bug with cpu cap and vmware.
> > >
> > > To reproduce:
> > >
> > > Create a offering with 2 cores and 1000mhz.
> > > Enable CPU CAP.
> > >
> > > After created instance , cs create a vm with 2 cores and 1000mhz
> > > of
> > limit.
> > >
> > > I don't know for sure if is a bug, but vmware gives 1000mhz shared
> > > with cores.
> > >
> > >
> > >
> > > Diego
> > >
>
>
RE: Vmware CPU Cap
Posted by Tamas Monos <ta...@veber.co.uk>.
Hi,
I have confused myself too because if I have a look the database and dump the service_offerings table the limit for me is set to 2000,3000,4000 and not 1000.
So a Mhz limit set to 4000 and 4 cores will end up as a quad core box at 4Ghz. I remember now I had to set CPU overprovisioning to 4 as CloudStack took away 4x4Ghz=16Ghz of the available CPU.
So Cloudstack sets the VM CPU Mhz limit what the actual limit is set to in the offering but does not multiply the Mhz limit by the number of cores when setting the limit on vmware. However takes away "number_of_cores*Mhz limit" from the available CPU capacity when allocating.
I'm getting confused myself so I'm not sure if this is bug now or not either :) I'm using 3.0.3
Regards
-----Original Message-----
From: Edison Su [mailto:Edison.su@citrix.com]
Sent: 23 May 2012 00:32
To: cloudstack-dev@incubator.apache.org; Tamas Monos
Subject: RE: Vmware CPU Cap
But in Diego'case, the limit is set as 1000Mhz, while his service offering is 1000Mhz * 2 cores.
Which version of cloudstack are you using? Maybe it's a regression.
> -----Original Message-----
> From: Tamas Monos [mailto:tamasm@veber.co.uk]
> Sent: Tuesday, May 22, 2012 4:26 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Hi,
>
> I have a service offering with 1000Mhz limit and 4 cpu cores. That
> sums up to 4000Mhz.
> Cloudstack sets the limit to 4Ghz on the virtual machine and when I
> put load on it vmware balances the load between the 4 cores allowing
> them to use 1000Mhz each.
> I do not see any bugs here.
>
> Apologies if the "meant per CPU core" was incorrect. What I meant is
> described above.
>
> Regards
>
> -----Original Message-----
> From: Edison Su [mailto:Edison.su@citrix.com]
> Sent: 23 May 2012 00:05
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Nope, from the document, the limit is set on whole vm:
> CPU Limits
>
> When a CPU Limit is set on a virtual machine resource settings, the
> virtual machine is deliberately held from being scheduled to a PCPU
> when it has used up its allocated CPU resource. This happens
> regardless of the CPU utilization. If the limit is set to 500MHz, the
> virtual machine is descheduled from the PCPU and has to wait before it
> is allowed to be scheduled again. As such, the virtual machine might
> experience performance degradation.
>
> Note: For an SMP virtual machine, the sum of all vCPUs cannot exceed
> the specified limit. For example, 4 vCPU virtual machine with a limit
> of 1200MHz and equal load among vCPUs would result in a max of 300MHz
> per vCPU.
>
> http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&popup
> =
> true&languageId=&externalID=1033115
>
> > -----Original Message-----
> > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > Sent: Tuesday, May 22, 2012 3:43 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > Hi,
> >
> > No I don't think this is a bug. When you set 1000Mhz as CPU cap that
> > is meant per core. vmWare will limit each CPU core to 1000Mhz.
> > As you gave 2 CPU cores that is 2000Mhz effective. That is how
> > vmware works.
> >
> > I have setup my offerings all to 1000Mhz as speed and just
> > increasing the number of cores.
> > 1 core ends up being 1x1000Mhz
> > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > ...
> >
> > I'm actually using it in production and works quite well as
> Cloudstack
> > "allocates" 4000Mhz when I'm using 4x1000Mhz cores and vmware
> cleverly
> > balances between the cores as you put load on it and not letting any
> > of the cores above the set limit of 1000Mhz.
> >
> >
> > Regards
> >
> > -----Original Message-----
> > From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
> > Sent: 22 May 2012 19:34
> > To: cloudstack-dev@incubator.apache.org
> > Subject: Re: Vmware CPU Cap
> >
> > I forgot the link:
> > http://imageshack.us/photo/my-images/848/cpulimit.png/
> >
> > 2012/5/22 Diego Spinola Castro <sp...@gmail.com>
> >
> > > I believe that is a bug with cpu cap and vmware.
> > >
> > > To reproduce:
> > >
> > > Create a offering with 2 cores and 1000mhz.
> > > Enable CPU CAP.
> > >
> > > After created instance , cs create a vm with 2 cores and 1000mhz
> > > of
> > limit.
> > >
> > > I don't know for sure if is a bug, but vmware gives 1000mhz shared
> > > with cores.
> > >
> > >
> > >
> > > Diego
> > >
>
>
RE: Vmware CPU Cap
Posted by Edison Su <Ed...@citrix.com>.
But in Diego'case, the limit is set as 1000Mhz, while his service offering is 1000Mhz * 2 cores.
Which version of cloudstack are you using? Maybe it's a regression.
> -----Original Message-----
> From: Tamas Monos [mailto:tamasm@veber.co.uk]
> Sent: Tuesday, May 22, 2012 4:26 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Hi,
>
> I have a service offering with 1000Mhz limit and 4 cpu cores. That sums
> up to 4000Mhz.
> Cloudstack sets the limit to 4Ghz on the virtual machine and when I put
> load on it vmware balances the load between the 4 cores allowing them
> to use 1000Mhz each.
> I do not see any bugs here.
>
> Apologies if the "meant per CPU core" was incorrect. What I meant is
> described above.
>
> Regards
>
> -----Original Message-----
> From: Edison Su [mailto:Edison.su@citrix.com]
> Sent: 23 May 2012 00:05
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Nope, from the document, the limit is set on whole vm:
> CPU Limits
>
> When a CPU Limit is set on a virtual machine resource settings, the
> virtual machine is deliberately held from being scheduled to a PCPU
> when it has used up its allocated CPU resource. This happens regardless
> of the CPU utilization. If the limit is set to 500MHz, the virtual
> machine is descheduled from the PCPU and has to wait before it is
> allowed to be scheduled again. As such, the virtual machine might
> experience performance degradation.
>
> Note: For an SMP virtual machine, the sum of all vCPUs cannot exceed
> the specified limit. For example, 4 vCPU virtual machine with a limit
> of 1200MHz and equal load among vCPUs would result in a max of 300MHz
> per vCPU.
>
> http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&popup=
> true&languageId=&externalID=1033115
>
> > -----Original Message-----
> > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > Sent: Tuesday, May 22, 2012 3:43 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > Hi,
> >
> > No I don't think this is a bug. When you set 1000Mhz as CPU cap that
> > is meant per core. vmWare will limit each CPU core to 1000Mhz.
> > As you gave 2 CPU cores that is 2000Mhz effective. That is how vmware
> > works.
> >
> > I have setup my offerings all to 1000Mhz as speed and just increasing
> > the number of cores.
> > 1 core ends up being 1x1000Mhz
> > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > ...
> >
> > I'm actually using it in production and works quite well as
> Cloudstack
> > "allocates" 4000Mhz when I'm using 4x1000Mhz cores and vmware
> cleverly
> > balances between the cores as you put load on it and not letting any
> > of the cores above the set limit of 1000Mhz.
> >
> >
> > Regards
> >
> > -----Original Message-----
> > From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
> > Sent: 22 May 2012 19:34
> > To: cloudstack-dev@incubator.apache.org
> > Subject: Re: Vmware CPU Cap
> >
> > I forgot the link:
> > http://imageshack.us/photo/my-images/848/cpulimit.png/
> >
> > 2012/5/22 Diego Spinola Castro <sp...@gmail.com>
> >
> > > I believe that is a bug with cpu cap and vmware.
> > >
> > > To reproduce:
> > >
> > > Create a offering with 2 cores and 1000mhz.
> > > Enable CPU CAP.
> > >
> > > After created instance , cs create a vm with 2 cores and 1000mhz of
> > limit.
> > >
> > > I don't know for sure if is a bug, but vmware gives 1000mhz shared
> > > with cores.
> > >
> > >
> > >
> > > Diego
> > >
>
>
RE: Vmware CPU Cap
Posted by Tamas Monos <ta...@veber.co.uk>.
Hi,
I have a service offering with 1000Mhz limit and 4 cpu cores. That sums up to 4000Mhz.
Cloudstack sets the limit to 4Ghz on the virtual machine and when I put load on it vmware balances the load between the 4 cores allowing them to use 1000Mhz each.
I do not see any bugs here.
Apologies if the "meant per CPU core" was incorrect. What I meant is described above.
Regards
-----Original Message-----
From: Edison Su [mailto:Edison.su@citrix.com]
Sent: 23 May 2012 00:05
To: cloudstack-dev@incubator.apache.org
Subject: RE: Vmware CPU Cap
Nope, from the document, the limit is set on whole vm:
CPU Limits
When a CPU Limit is set on a virtual machine resource settings, the virtual machine is deliberately held from being scheduled to a PCPU when it has used up its allocated CPU resource. This happens regardless of the CPU utilization. If the limit is set to 500MHz, the virtual machine is descheduled from the PCPU and has to wait before it is allowed to be scheduled again. As such, the virtual machine might experience performance degradation.
Note: For an SMP virtual machine, the sum of all vCPUs cannot exceed the specified limit. For example, 4 vCPU virtual machine with a limit of 1200MHz and equal load among vCPUs would result in a max of 300MHz per vCPU.
http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&popup=true&languageId=&externalID=1033115
> -----Original Message-----
> From: Tamas Monos [mailto:tamasm@veber.co.uk]
> Sent: Tuesday, May 22, 2012 3:43 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Hi,
>
> No I don't think this is a bug. When you set 1000Mhz as CPU cap that
> is meant per core. vmWare will limit each CPU core to 1000Mhz.
> As you gave 2 CPU cores that is 2000Mhz effective. That is how vmware
> works.
>
> I have setup my offerings all to 1000Mhz as speed and just increasing
> the number of cores.
> 1 core ends up being 1x1000Mhz
> 2 core ends up being 2x1000Mhz=2000Mhz ...
> ...
>
> I'm actually using it in production and works quite well as Cloudstack
> "allocates" 4000Mhz when I'm using 4x1000Mhz cores and vmware cleverly
> balances between the cores as you put load on it and not letting any
> of the cores above the set limit of 1000Mhz.
>
>
> Regards
>
> -----Original Message-----
> From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
> Sent: 22 May 2012 19:34
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: Vmware CPU Cap
>
> I forgot the link:
> http://imageshack.us/photo/my-images/848/cpulimit.png/
>
> 2012/5/22 Diego Spinola Castro <sp...@gmail.com>
>
> > I believe that is a bug with cpu cap and vmware.
> >
> > To reproduce:
> >
> > Create a offering with 2 cores and 1000mhz.
> > Enable CPU CAP.
> >
> > After created instance , cs create a vm with 2 cores and 1000mhz of
> limit.
> >
> > I don't know for sure if is a bug, but vmware gives 1000mhz shared
> > with cores.
> >
> >
> >
> > Diego
> >
RE: Vmware CPU Cap
Posted by Edison Su <Ed...@citrix.com>.
Nope, from the document, the limit is set on whole vm:
CPU Limits
When a CPU Limit is set on a virtual machine resource settings, the virtual machine is deliberately held from being scheduled to a PCPU when it has used up its allocated CPU resource. This happens regardless of the CPU utilization. If the limit is set to 500MHz, the virtual machine is descheduled from the PCPU and has to wait before it is allowed to be scheduled again. As such, the virtual machine might experience performance degradation.
Note: For an SMP virtual machine, the sum of all vCPUs cannot exceed the specified limit. For example, 4 vCPU virtual machine with a limit of 1200MHz and equal load among vCPUs would result in a max of 300MHz per vCPU.
http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&popup=true&languageId=&externalID=1033115
> -----Original Message-----
> From: Tamas Monos [mailto:tamasm@veber.co.uk]
> Sent: Tuesday, May 22, 2012 3:43 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Hi,
>
> No I don't think this is a bug. When you set 1000Mhz as CPU cap that is
> meant per core. vmWare will limit each CPU core to 1000Mhz.
> As you gave 2 CPU cores that is 2000Mhz effective. That is how vmware
> works.
>
> I have setup my offerings all to 1000Mhz as speed and just increasing
> the number of cores.
> 1 core ends up being 1x1000Mhz
> 2 core ends up being 2x1000Mhz=2000Mhz
> ...
> ...
>
> I'm actually using it in production and works quite well as Cloudstack
> "allocates" 4000Mhz when I'm using 4x1000Mhz cores and vmware cleverly
> balances between the cores as you put load on it and not letting any of
> the cores above the set limit of 1000Mhz.
>
>
> Regards
>
> -----Original Message-----
> From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
> Sent: 22 May 2012 19:34
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: Vmware CPU Cap
>
> I forgot the link:
> http://imageshack.us/photo/my-images/848/cpulimit.png/
>
> 2012/5/22 Diego Spinola Castro <sp...@gmail.com>
>
> > I believe that is a bug with cpu cap and vmware.
> >
> > To reproduce:
> >
> > Create a offering with 2 cores and 1000mhz.
> > Enable CPU CAP.
> >
> > After created instance , cs create a vm with 2 cores and 1000mhz of
> limit.
> >
> > I don't know for sure if is a bug, but vmware gives 1000mhz shared
> > with cores.
> >
> >
> >
> > Diego
> >
RE: Vmware CPU Cap
Posted by Tamas Monos <ta...@veber.co.uk>.
Hi,
No I don't think this is a bug. When you set 1000Mhz as CPU cap that is meant per core. vmWare will limit each CPU core to 1000Mhz.
As you gave 2 CPU cores that is 2000Mhz effective. That is how vmware works.
I have setup my offerings all to 1000Mhz as speed and just increasing the number of cores.
1 core ends up being 1x1000Mhz
2 core ends up being 2x1000Mhz=2000Mhz
...
...
I'm actually using it in production and works quite well as Cloudstack "allocates" 4000Mhz when I'm using 4x1000Mhz cores and vmware cleverly balances between the cores as you put load on it and not letting any of the cores above the set limit of 1000Mhz.
Regards
-----Original Message-----
From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
Sent: 22 May 2012 19:34
To: cloudstack-dev@incubator.apache.org
Subject: Re: Vmware CPU Cap
I forgot the link:
http://imageshack.us/photo/my-images/848/cpulimit.png/
2012/5/22 Diego Spinola Castro <sp...@gmail.com>
> I believe that is a bug with cpu cap and vmware.
>
> To reproduce:
>
> Create a offering with 2 cores and 1000mhz.
> Enable CPU CAP.
>
> After created instance , cs create a vm with 2 cores and 1000mhz of limit.
>
> I don't know for sure if is a bug, but vmware gives 1000mhz shared
> with cores.
>
>
>
> Diego
>
Re: Vmware CPU Cap
Posted by Diego Spinola Castro <sp...@gmail.com>.
I forgot the link:
http://imageshack.us/photo/my-images/848/cpulimit.png/
2012/5/22 Diego Spinola Castro <sp...@gmail.com>
> I believe that is a bug with cpu cap and vmware.
>
> To reproduce:
>
> Create a offering with 2 cores and 1000mhz.
> Enable CPU CAP.
>
> After created instance , cs create a vm with 2 cores and 1000mhz of limit.
>
> I don't know for sure if is a bug, but vmware gives 1000mhz shared with
> cores.
>
>
>
> Diego
>
RE: Vmware CPU Cap
Posted by Diego Spinola Castro <sp...@gmail.com>.
Edison. What we are going to do? I believe this is a important path since
cs isnt using the right way of resource allocation.
[]'s
Diego Castro
Em 23/05/2012 19:29, "Edison Su" <Ed...@citrix.com> escreveu: