You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Devdeep Singh <de...@citrix.com> on 2012/07/11 11:50:47 UTC
Limits set for a resource can be exceeded
Hi,
I observed this issue while looking into how resource limiting works. When a request is placed to create a resource (instance, volume etc.) the check for resource limits and incrementing the resource count is done in the following order
1. Check the limits will not be exceeded by the addition of another resource. Raise an exception if so.
2. Create a db entry for the resource.
3. Increment the resource count.
The check and increment of a resource are not synchronized to be atomic. This can lead to scenarios where two different requests can successfully check the limits for the same resource and increment the count beyond the limits on boundary conditions.
For example: Suppose domain limits for a vm instance is set to 5 and there are already 4 instances created for the domain. Two requests are placed by two different users of the domain to create an instance.
1. User 1 request checks the limit. Verifies that domain limit of 5 will not be exceeded.
2. Similarly User 2 request checks the limit and verifies the domain limit of 5 will not be exceeded. This is because the first request still hasn't incremented the resource count.
3. Both the requests go ahead and increment the count (to 6) and exceed the limit.
I have observed this issue with instance creation. It may be present at other places too (volumes, public ips etc.). Before I go ahead and file a bug, I wanted to check if it is a valid concern or am I missing something.
Regards,
Devdeep
RE: Limits set for a resource can be exceeded
Posted by Devdeep Singh <de...@citrix.com>.
> -----Original Message-----
> From: Prasanna Santhanam [mailto:Prasanna.Santhanam@citrix.com]
> Sent: Thursday, July 12, 2012 4:34 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: Limits set for a resource can be exceeded
>
> On Wed, Jul 11, 2012 at 05:50:47AM -0400, Devdeep Singh wrote:
> > Hi,
> >
> > I observed this issue while looking into how resource limiting works.
> > When a request is placed to create a resource (instance, volume etc.)
> > the check for resource limits and incrementing the resource count is
> > done in the following order
> >
> > 1. Check the limits will not be exceeded by the addition of another
> > resource. Raise an exception if so. 2. Create a db entry for the
> > resource. 3. Increment the resource count.
> >
> > The check and increment of a resource are not synchronized to be
> > atomic. This can lead to scenarios where two different requests can
> > successfully check the limits for the same resource and increment the
> > count beyond the limits on boundary conditions.
> >
> > For example: Suppose domain limits for a vm instance is set to 5 and
> > there are already 4 instances created for the domain. Two requests are
> > placed by two different users of the domain to create an instance. 1.
> > User 1 request checks the limit. Verifies that domain limit of 5 will
> > not be exceeded. 2. Similarly User 2 request checks the limit and
> > verifies the domain limit of 5 will not be exceeded.
> > This is because the first request still hasn't incremented the
> > resource count. 3. Both the requests go ahead and increment the count
> > (to 6) and exceed the limit.
> >
> > I have observed this issue with instance creation. It may be present
> > at other places too (volumes, public ips etc.). Before I go ahead and
> > file a bug, I wanted to check if it is a valid concern or am I missing
> > something.
> >
>
> FWIW, this counter's atomicity used to be a real problem back in the last few
> 2.2.y series of releases and partly the reason why the updateResourceCount
> API came into being. But I remember this was fixed for accounts limits.
> Whether domain-limits and other resource limits follow the same
> increment/decrement code paths I'm not sure. If they don't we most likely
> have the same bug.
>
> Can we reproduce this trivially or will we have to hold the threads in the
> debugger to cause this situation?
>
> --
> Prasanna.,
I reproduced the bug for limits on instances in a debugger by suspending one thread
after it had checked for limits and letting the second request go through (and then
resuming the first thread).
The problem with limits on snapshots is easier to reproduce because snapshot upload to
secondary storage takes time and the increment is done after upload is successful.
http://bugs.cloudstack.org/browse/CS-15430 was filed for the same. A fix for this has
already been proposed.
Regards,
Devdeep
Re: Limits set for a resource can be exceeded
Posted by Prasanna Santhanam <Pr...@citrix.com>.
On Thu, Jul 12, 2012 at 07:14:39AM -0400, Somesh Naidu wrote:
> http://bugs.cloudstack.org/browse/CS-11930
>
> The above bug suggests similar for snapshots. Its marked as fixed in Acton.
It looks like a different problem. That's talking about the limit
being honored. The one Devdeep is talking about is a more serious
issue about concurrent updates to the same resource.
--
Prasanna.,
RE: Limits set for a resource can be exceeded
Posted by Somesh Naidu <So...@citrix.com>.
http://bugs.cloudstack.org/browse/CS-11930
The above bug suggests similar for snapshots. Its marked as fixed in Acton.
Regards,
Somesh
-----Original Message-----
From: Prasanna Santhanam [mailto:Prasanna.Santhanam@citrix.com]
Sent: Thursday, July 12, 2012 4:34 PM
To: cloudstack-dev@incubator.apache.org
Subject: Re: Limits set for a resource can be exceeded
On Wed, Jul 11, 2012 at 05:50:47AM -0400, Devdeep Singh wrote:
> Hi,
>
> I observed this issue while looking into how resource limiting
> works. When a request is placed to create a resource (instance,
> volume etc.) the check for resource limits and incrementing the
> resource count is done in the following order
>
> 1. Check the limits will not be exceeded by the addition of another
> resource. Raise an exception if so. 2. Create a db entry for the
> resource. 3. Increment the resource count.
>
> The check and increment of a resource are not synchronized to be
> atomic. This can lead to scenarios where two different requests can
> successfully check the limits for the same resource and increment
> the count beyond the limits on boundary conditions.
>
> For example: Suppose domain limits for a vm instance is set to 5 and
> there are already 4 instances created for the domain. Two requests
> are placed by two different users of the domain to create an
> instance. 1. User 1 request checks the limit. Verifies that domain
> limit of 5 will not be exceeded. 2. Similarly User 2 request checks
> the limit and verifies the domain limit of 5 will not be exceeded.
> This is because the first request still hasn't incremented the
> resource count. 3. Both the requests go ahead and increment the
> count (to 6) and exceed the limit.
>
> I have observed this issue with instance creation. It may be present
> at other places too (volumes, public ips etc.). Before I go ahead
> and file a bug, I wanted to check if it is a valid concern or am I
> missing something.
>
FWIW, this counter's atomicity used to be a real problem back in the
last few 2.2.y series of releases and partly the reason why the
updateResourceCount API came into being. But I remember this was fixed
for accounts limits. Whether domain-limits and other resource limits
follow the same increment/decrement code paths I'm not sure. If they
don't we most likely have the same bug.
Can we reproduce this trivially or will we have to hold the threads in
the debugger to cause this situation?
--
Prasanna.,
Re: Limits set for a resource can be exceeded
Posted by Prasanna Santhanam <Pr...@citrix.com>.
On Wed, Jul 11, 2012 at 05:50:47AM -0400, Devdeep Singh wrote:
> Hi,
>
> I observed this issue while looking into how resource limiting
> works. When a request is placed to create a resource (instance,
> volume etc.) the check for resource limits and incrementing the
> resource count is done in the following order
>
> 1. Check the limits will not be exceeded by the addition of another
> resource. Raise an exception if so. 2. Create a db entry for the
> resource. 3. Increment the resource count.
>
> The check and increment of a resource are not synchronized to be
> atomic. This can lead to scenarios where two different requests can
> successfully check the limits for the same resource and increment
> the count beyond the limits on boundary conditions.
>
> For example: Suppose domain limits for a vm instance is set to 5 and
> there are already 4 instances created for the domain. Two requests
> are placed by two different users of the domain to create an
> instance. 1. User 1 request checks the limit. Verifies that domain
> limit of 5 will not be exceeded. 2. Similarly User 2 request checks
> the limit and verifies the domain limit of 5 will not be exceeded.
> This is because the first request still hasn't incremented the
> resource count. 3. Both the requests go ahead and increment the
> count (to 6) and exceed the limit.
>
> I have observed this issue with instance creation. It may be present
> at other places too (volumes, public ips etc.). Before I go ahead
> and file a bug, I wanted to check if it is a valid concern or am I
> missing something.
>
FWIW, this counter's atomicity used to be a real problem back in the
last few 2.2.y series of releases and partly the reason why the
updateResourceCount API came into being. But I remember this was fixed
for accounts limits. Whether domain-limits and other resource limits
follow the same increment/decrement code paths I'm not sure. If they
don't we most likely have the same bug.
Can we reproduce this trivially or will we have to hold the threads in
the debugger to cause this situation?
--
Prasanna.,