You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Leo Simons <LS...@schubergphilis.com> on 2014/11/18 14:52:45 UTC
No event publish can be wrapped within db transaction...why?
Hi Min, hi Koushik,
Cloudstack is shouting at me:
NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!
(full stack trace below). I've learned this is happening on our systemvm-persistent-config feature branch because it has commit ffaabdc13fde0f0f7b2667a483006e2a4b805f63 but it does not have commit f585dd266188a134a9c8b911376b066b9d3806e8 yet.
I'm now trying to understand what's happening here -- the transaction / concurrency / messaging logic gave me significant headache with its triple negatives, nested transaction scoping and home-grown gates, but I think I got it now.
As I understand it, in the olde world, creating an account:
* opens a database transaction
* creates an account in the db
* creates the first user in that account in the db
* publishes an event
* which is listened to by 0 subscribers
* commmits the database transaction
* check the user is there
* opens a database transaction
* find the created user in the database
* (auto)closes transaction
* returns success if the user is in the db
this, err, works, but in some other cases, apparently, there are concerns that the db transaction is open too long while message handling happens. So that's why the warning was added, and follow up on, and so now, creating an account:
* opens a database transaction
* creates an account in the db
* creates the first user in that account in the db
* commmits the database transaction
* publishes an event
* which is still listened to by on average 0 subscribers,
but there could be an IAM subscriber
* check the user is there
* opens a database transaction
* find the created user in the database
* (auto)closes transaction
* returns success if the user is in the db
The one possible subscriber for account creation is IAMApiServiceImpl, which when receiving the event
* opens a database transaction
* adds the account to acl_group_account_map
* commits the database transaction
* finds the domain for the account
* opens a database transaction
* finds the domain for the account
* (auto)closes transaction
* finds the domain groups for the domain
* opens a database transaction
* finds the domain groups for the domain
* (auto)closes transaction
* for each domain group
* opens a database transaction
* adds the account to acl_group_account_map
* commits the database transaction
in other words, if there's 1 domain group and an enabled IAM thingie, this spreads out "make an account" over 6 transactions. Without IAM thingie its 2 two transactions with a no-op message bus thingie in the middle. Is that correct?
If so, I don't understand this at all. The pre-November code doesn't make that much sense to me (why query the database? If you don't trust your database its ACID guarantees...why use transactions? Why do we ever need a message bus between two java components in the same classloader?), but the new code scares me.
In the case of errors in between transactions, you can end up with accounts that are not in all the groups they should be in. I imagine I would much rather see the whole thing fail, and the complete api call fail, so that I can re-try it as a whole, than end up with a somehow half-initialized account. I.e. have everything account-management-y happen in one transaction which is rolled back on any failure.
Any thoughts?
Thanks!
Leo (who can't ask Hugo since Hugo is at apachecon/ccceu and he isn't :))
2014-11-18 13:36:33,145 ERROR [o.a.c.f.m.MessageBusBase] (qtp1734055321-25:ctx-05df2079 ctx-25ea4461 ctx-3aac3268)
NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!
com.cloud.utils.exception.CloudRuntimeException:
NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!
at org.apache.cloudstack.framework.messagebus.MessageBusBase.publish(MessageBusBase.java:167)
at com.cloud.user.AccountManagerImpl$2.doInTransaction(AccountManagerImpl.java:1052)
at com.cloud.user.AccountManagerImpl$2.doInTransaction(AccountManagerImpl.java:1027)
at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:57)
at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
at com.cloud.utils.db.Transaction.execute(Transaction.java:54)
at com.cloud.user.AccountManagerImpl.createUserAccount(AccountManagerImpl.java:1027)
at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy108.createUserAccount(Unknown Source)
at org.apache.cloudstack.api.command.admin.account.CreateAccountCmd.execute(CreateAccountCmd.java:178)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:691)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:514)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:273)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:117)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:114)
at com.cloud.api.ApiServlet.doPost(ApiServlet.java:81)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
Re: No event publish can be wrapped within db transaction...why?
Posted by Min Chen <mi...@citrix.com>.
For IAM feature design goal, you can take a look at our FS at
https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+Identity+
and+Access+Management+%28IAM%29+Plugin. Also Prachi and I presented this
work at Denver Apache CloudStack Collab conference, you can also view our
slideshare at
http://events.linuxfoundation.org/sites/events/files/slides/ApachIAM.pdf
and view the recorded presentation at
https://www.youtube.com/watch?v=iUThjMl2yl8&list=PLU2OcwpQkYCyPx_cwJxyOK0YK
SM86Mj9n&index=24.
Hope that those pointers can provide some help.
-min
On 11/18/14 11:26 AM, "Leo Simons" <LS...@schubergphilis.com> wrote:
>Hi Min,
>
>Thanks for a very clear answer!
>
>However, I'm afraid I still don't get it :-). So...
>
>...do you have any specific example or use case of an external IAM
>service to integrate with? Is there some kind of design document for me
>to understand the goals?
>
>I ask because all the ones that I'm familiar with tend to assume that the
>owner of identity information (and grouping, and possibly other kinds of
>AAA assertions) is externalized from systems like cloudstack to the
>identity system, i.e. integration is "the other way around".
>
>So i.e. you would have AD or other LDAP or an SSO server or a SAML
>implementation (or all of those...), where systems like cloudstack then
>delegate AAA questions/assertions to those systems, rather than
>propagating local identities to that central system.
>
>I imagine if you have an external identity provider, you plug in a
>different implementation of AccountManager (LDAPAccountManager? etc.),
>and then a CreateAccountCmd would fail with an error saying the server is
>configured to use <<external thing>> so account creation is unsupported.
>
>
>cheers!
>
>
>Leo
>
>
>On Nov 18, 2014, at 7:50 PM, Min Chen <mi...@citrix.com> wrote:
>> Hi Leo,
>>
>> "NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!" is along the
>> same line as "NO AGENT COMMAND CAN BE WRAPPED WITHIN DB TRANSACTION!".
>>The
>> rationale behind this is simple: event subscriber execution or agent
>> command handling at resource layer may take too long, and we don't want
>>to
>> have that long transaction window to hold DB for too long.
>> As for your questions about why we bother to use message bus to
>> communicate between two java component, there is a reason for it: loose
>> coupling. IAMApiServiceImpl is a class in IAM plugin service, which can
>>be
>> deployed as a totally different service from CloudStack management
>>server
>> and ideally with future 3rd-party authentication/authorization
>> integration, they may use a totally different database from "cloud"
>> database we are currently using just for simplicity. In this deployment
>> architecture, we have to make sure that this IAM service and CloudStack
>>MS
>> components are loosely coupled. Message bus provided us a very good
>> approach to achieve that.
>> As you said, ideally we would like to achieve a prefect transaction
>> related to account creation in both CloudStack main component and its
>> plugin services, but in reality, this may not work always and big
>> transaction will be error-prone for large scale distributed systems,
>> especially for this loosely coupled components that are crossing
>>different
>> DBs. The plugin architecture in CloudStack is designed to easily
>> enable/disable each plugin component without impacting too much on main
>> CloudStack components. So in this case, I would personally prefer that
>>we
>> should make sure of data integrity in the scope of CloudStack main
>> components first and handle potential message handling failure in plugin
>> module separately through application level logic.
>>
>> Thanks
>> -min
>>
>>
>>
>> On 11/18/14 5:52 AM, "Leo Simons" <LS...@schubergphilis.com> wrote:
>>
>>> Hi Min, hi Koushik,
>>>
>>> Cloudstack is shouting at me:
>>> NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!
>>>
>>> (full stack trace below). I've learned this is happening on our
>>> systemvm-persistent-config feature branch because it has commit
>>> ffaabdc13fde0f0f7b2667a483006e2a4b805f63 but it does not have commit
>>> f585dd266188a134a9c8b911376b066b9d3806e8 yet.
>>>
>>> I'm now trying to understand what's happening here -- the transaction /
>>> concurrency / messaging logic gave me significant headache with its
>>> triple negatives, nested transaction scoping and home-grown gates, but
>>>I
>>> think I got it now.
>>>
>>> As I understand it, in the olde world, creating an account:
>>>
>>> * opens a database transaction
>>> * creates an account in the db
>>> * creates the first user in that account in the db
>>> * publishes an event
>>> * which is listened to by 0 subscribers
>>> * commmits the database transaction
>>> * check the user is there
>>> * opens a database transaction
>>> * find the created user in the database
>>> * (auto)closes transaction
>>> * returns success if the user is in the db
>>>
>>> this, err, works, but in some other cases, apparently, there are
>>>concerns
>>> that the db transaction is open too long while message handling
>>>happens.
>>> So that's why the warning was added, and follow up on, and so now,
>>> creating an account:
>>>
>>> * opens a database transaction
>>> * creates an account in the db
>>> * creates the first user in that account in the db
>>> * commmits the database transaction
>>> * publishes an event
>>> * which is still listened to by on average 0 subscribers,
>>> but there could be an IAM subscriber
>>> * check the user is there
>>> * opens a database transaction
>>> * find the created user in the database
>>> * (auto)closes transaction
>>> * returns success if the user is in the db
>>>
>>> The one possible subscriber for account creation is IAMApiServiceImpl,
>>> which when receiving the event
>>>
>>> * opens a database transaction
>>> * adds the account to acl_group_account_map
>>> * commits the database transaction
>>> * finds the domain for the account
>>> * opens a database transaction
>>> * finds the domain for the account
>>> * (auto)closes transaction
>>> * finds the domain groups for the domain
>>> * opens a database transaction
>>> * finds the domain groups for the domain
>>> * (auto)closes transaction
>>> * for each domain group
>>> * opens a database transaction
>>> * adds the account to acl_group_account_map
>>> * commits the database transaction
>>>
>>> in other words, if there's 1 domain group and an enabled IAM thingie,
>>> this spreads out "make an account" over 6 transactions. Without IAM
>>> thingie its 2 two transactions with a no-op message bus thingie in the
>>> middle. Is that correct?
>>>
>>> If so, I don't understand this at all. The pre-November code doesn't
>>>make
>>> that much sense to me (why query the database? If you don't trust your
>>> database its ACID guarantees...why use transactions? Why do we ever
>>>need
>>> a message bus between two java components in the same classloader?),
>>>but
>>> the new code scares me.
>>>
>>> In the case of errors in between transactions, you can end up with
>>> accounts that are not in all the groups they should be in. I imagine I
>>> would much rather see the whole thing fail, and the complete api call
>>> fail, so that I can re-try it as a whole, than end up with a somehow
>>> half-initialized account. I.e. have everything account-management-y
>>> happen in one transaction which is rolled back on any failure.
>>>
>>> Any thoughts?
>>>
>>>
>>> Thanks!
>>>
>>>
>>> Leo (who can't ask Hugo since Hugo is at apachecon/ccceu and he isn't
>>>:))
>>>
>>>
>>> 2014-11-18 13:36:33,145 ERROR [o.a.c.f.m.MessageBusBase]
>>> (qtp1734055321-25:ctx-05df2079 ctx-25ea4461 ctx-3aac3268)
>>> NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!
>>> com.cloud.utils.exception.CloudRuntimeException:
>>> NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!
>>> at
>>>
>>>org.apache.cloudstack.framework.messagebus.MessageBusBase.publish(Messag
>>>eB
>>> usBase.java:167)
>>> at
>>>
>>>com.cloud.user.AccountManagerImpl$2.doInTransaction(AccountManagerImpl.j
>>>av
>>> a:1052)
>>> at
>>>
>>>com.cloud.user.AccountManagerImpl$2.doInTransaction(AccountManagerImpl.j
>>>av
>>> a:1027)
>>> at
>>>com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:57)
>>> at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
>>> at com.cloud.utils.db.Transaction.execute(Transaction.java:54)
>>> at
>>>
>>>com.cloud.user.AccountManagerImpl.createUserAccount(AccountManagerImpl.j
>>>av
>>> a:1027)
>>> at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source)
>>> at
>>>
>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>Im
>>> pl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at
>>>
>>>org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(
>>>Ao
>>> pUtils.java:317)
>>> at
>>>
>>>org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinp
>>>oi
>>> nt(ReflectiveMethodInvocation.java:183)
>>> at
>>>
>>>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Ref
>>>le
>>> ctiveMethodInvocation.java:150)
>>> at
>>>
>>>org.apache.cloudstack.network.contrail.management.EventUtils$EventInterc
>>>ep
>>> tor.invoke(EventUtils.java:106)
>>> at
>>>
>>>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Ref
>>>le
>>> ctiveMethodInvocation.java:161)
>>> at
>>>
>>>com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.jav
>>>a:
>>> 51)
>>> at
>>>
>>>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Ref
>>>le
>>> ctiveMethodInvocation.java:161)
>>> at
>>>
>>>org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(E
>>>xp
>>> oseInvocationInterceptor.java:91)
>>> at
>>>
>>>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Ref
>>>le
>>> ctiveMethodInvocation.java:172)
>>> at
>>>
>>>org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAo
>>>pP
>>> roxy.java:204)
>>> at com.sun.proxy.$Proxy108.createUserAccount(Unknown Source)
>>> at
>>>
>>>org.apache.cloudstack.api.command.admin.account.CreateAccountCmd.execute
>>>(C
>>> reateAccountCmd.java:178)
>>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
>>> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:691)
>>> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:514)
>>> at
>>>com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:273)
>>> at com.cloud.api.ApiServlet$1.run(ApiServlet.java:117)
>>> at
>>>
>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(
>>>De
>>> faultManagedContext.java:56)
>>> at
>>>
>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWit
>>>hC
>>> ontext(DefaultManagedContext.java:103)
>>> at
>>>
>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWith
>>>Co
>>> ntext(DefaultManagedContext.java:53)
>>> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:114)
>>> at com.cloud.api.ApiServlet.doPost(ApiServlet.java:81)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>
>
Re: No event publish can be wrapped within db transaction...why?
Posted by Rohit Yadav <ro...@shapeblue.com>.
I’m unable to follow the thread. I’ve gone through the slides Min shared and the talk, can Min/Prachi or anyone else working on IAM/Events take a second attempt at explaining the issue/cause/implementation.
Leo, if you’ve already figured out I would love to hear from you. Should we even allow events to be published within db transaction, why not do it after db transaction goes successful (or fails in case we want to publish failures)?
> On 19-Nov-2014, at 12:56 am, Leo Simons <LS...@schubergphilis.com> wrote:
>
> Hi Min,
>
> Thanks for a very clear answer!
>
> However, I'm afraid I still don't get it :-). So...
>
> ...do you have any specific example or use case of an external IAM service to integrate with? Is there some kind of design document for me to understand the goals?
>
> I ask because all the ones that I'm familiar with tend to assume that the owner of identity information (and grouping, and possibly other kinds of AAA assertions) is externalized from systems like cloudstack to the identity system, i.e. integration is "the other way around".
>
> So i.e. you would have AD or other LDAP or an SSO server or a SAML implementation (or all of those...), where systems like cloudstack then delegate AAA questions/assertions to those systems, rather than propagating local identities to that central system.
>
> I imagine if you have an external identity provider, you plug in a different implementation of AccountManager (LDAPAccountManager? etc.), and then a CreateAccountCmd would fail with an error saying the server is configured to use <<external thing>> so account creation is unsupported.
>
>
> cheers!
>
>
> Leo
>
>
> On Nov 18, 2014, at 7:50 PM, Min Chen <mi...@citrix.com> wrote:
>> Hi Leo,
>>
>> "NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!" is along the
>> same line as "NO AGENT COMMAND CAN BE WRAPPED WITHIN DB TRANSACTION!". The
>> rationale behind this is simple: event subscriber execution or agent
>> command handling at resource layer may take too long, and we don't want to
>> have that long transaction window to hold DB for too long.
>> As for your questions about why we bother to use message bus to
>> communicate between two java component, there is a reason for it: loose
>> coupling. IAMApiServiceImpl is a class in IAM plugin service, which can be
>> deployed as a totally different service from CloudStack management server
>> and ideally with future 3rd-party authentication/authorization
>> integration, they may use a totally different database from "cloud"
>> database we are currently using just for simplicity. In this deployment
>> architecture, we have to make sure that this IAM service and CloudStack MS
>> components are loosely coupled. Message bus provided us a very good
>> approach to achieve that.
>> As you said, ideally we would like to achieve a prefect transaction
>> related to account creation in both CloudStack main component and its
>> plugin services, but in reality, this may not work always and big
>> transaction will be error-prone for large scale distributed systems,
>> especially for this loosely coupled components that are crossing different
>> DBs. The plugin architecture in CloudStack is designed to easily
>> enable/disable each plugin component without impacting too much on main
>> CloudStack components. So in this case, I would personally prefer that we
>> should make sure of data integrity in the scope of CloudStack main
>> components first and handle potential message handling failure in plugin
>> module separately through application level logic.
>>
>> Thanks
>> -min
>>
>>
>>
>> On 11/18/14 5:52 AM, "Leo Simons" <LS...@schubergphilis.com> wrote:
>>
>>> Hi Min, hi Koushik,
>>>
>>> Cloudstack is shouting at me:
>>> NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!
>>>
>>> (full stack trace below). I've learned this is happening on our
>>> systemvm-persistent-config feature branch because it has commit
>>> ffaabdc13fde0f0f7b2667a483006e2a4b805f63 but it does not have commit
>>> f585dd266188a134a9c8b911376b066b9d3806e8 yet.
>>>
>>> I'm now trying to understand what's happening here -- the transaction /
>>> concurrency / messaging logic gave me significant headache with its
>>> triple negatives, nested transaction scoping and home-grown gates, but I
>>> think I got it now.
>>>
>>> As I understand it, in the olde world, creating an account:
>>>
>>> * opens a database transaction
>>> * creates an account in the db
>>> * creates the first user in that account in the db
>>> * publishes an event
>>> * which is listened to by 0 subscribers
>>> * commmits the database transaction
>>> * check the user is there
>>> * opens a database transaction
>>> * find the created user in the database
>>> * (auto)closes transaction
>>> * returns success if the user is in the db
>>>
>>> this, err, works, but in some other cases, apparently, there are concerns
>>> that the db transaction is open too long while message handling happens.
>>> So that's why the warning was added, and follow up on, and so now,
>>> creating an account:
>>>
>>> * opens a database transaction
>>> * creates an account in the db
>>> * creates the first user in that account in the db
>>> * commmits the database transaction
>>> * publishes an event
>>> * which is still listened to by on average 0 subscribers,
>>> but there could be an IAM subscriber
>>> * check the user is there
>>> * opens a database transaction
>>> * find the created user in the database
>>> * (auto)closes transaction
>>> * returns success if the user is in the db
>>>
>>> The one possible subscriber for account creation is IAMApiServiceImpl,
>>> which when receiving the event
>>>
>>> * opens a database transaction
>>> * adds the account to acl_group_account_map
>>> * commits the database transaction
>>> * finds the domain for the account
>>> * opens a database transaction
>>> * finds the domain for the account
>>> * (auto)closes transaction
>>> * finds the domain groups for the domain
>>> * opens a database transaction
>>> * finds the domain groups for the domain
>>> * (auto)closes transaction
>>> * for each domain group
>>> * opens a database transaction
>>> * adds the account to acl_group_account_map
>>> * commits the database transaction
>>>
>>> in other words, if there's 1 domain group and an enabled IAM thingie,
>>> this spreads out "make an account" over 6 transactions. Without IAM
>>> thingie its 2 two transactions with a no-op message bus thingie in the
>>> middle. Is that correct?
>>>
>>> If so, I don't understand this at all. The pre-November code doesn't make
>>> that much sense to me (why query the database? If you don't trust your
>>> database its ACID guarantees...why use transactions? Why do we ever need
>>> a message bus between two java components in the same classloader?), but
>>> the new code scares me.
>>>
>>> In the case of errors in between transactions, you can end up with
>>> accounts that are not in all the groups they should be in. I imagine I
>>> would much rather see the whole thing fail, and the complete api call
>>> fail, so that I can re-try it as a whole, than end up with a somehow
>>> half-initialized account. I.e. have everything account-management-y
>>> happen in one transaction which is rolled back on any failure.
>>>
>>> Any thoughts?
>>>
>>>
>>> Thanks!
>>>
>>>
>>> Leo (who can't ask Hugo since Hugo is at apachecon/ccceu and he isn't :))
>>>
>>>
>>> 2014-11-18 13:36:33,145 ERROR [o.a.c.f.m.MessageBusBase]
>>> (qtp1734055321-25:ctx-05df2079 ctx-25ea4461 ctx-3aac3268)
>>> NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!
>>> com.cloud.utils.exception.CloudRuntimeException:
>>> NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!
>>> at
>>> org.apache.cloudstack.framework.messagebus.MessageBusBase.publish(MessageB
>>> usBase.java:167)
>>> at
>>> com.cloud.user.AccountManagerImpl$2.doInTransaction(AccountManagerImpl.jav
>>> a:1052)
>>> at
>>> com.cloud.user.AccountManagerImpl$2.doInTransaction(AccountManagerImpl.jav
>>> a:1027)
>>> at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:57)
>>> at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
>>> at com.cloud.utils.db.Transaction.execute(Transaction.java:54)
>>> at
>>> com.cloud.user.AccountManagerImpl.createUserAccount(AccountManagerImpl.jav
>>> a:1027)
>>> at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>>> pl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at
>>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(Ao
>>> pUtils.java:317)
>>> at
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoi
>>> nt(ReflectiveMethodInvocation.java:183)
>>> at
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle
>>> ctiveMethodInvocation.java:150)
>>> at
>>> org.apache.cloudstack.network.contrail.management.EventUtils$EventIntercep
>>> tor.invoke(EventUtils.java:106)
>>> at
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle
>>> ctiveMethodInvocation.java:161)
>>> at
>>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:
>>> 51)
>>> at
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle
>>> ctiveMethodInvocation.java:161)
>>> at
>>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Exp
>>> oseInvocationInterceptor.java:91)
>>> at
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle
>>> ctiveMethodInvocation.java:172)
>>> at
>>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopP
>>> roxy.java:204)
>>> at com.sun.proxy.$Proxy108.createUserAccount(Unknown Source)
>>> at
>>> org.apache.cloudstack.api.command.admin.account.CreateAccountCmd.execute(C
>>> reateAccountCmd.java:178)
>>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
>>> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:691)
>>> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:514)
>>> at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:273)
>>> at com.cloud.api.ApiServlet$1.run(ApiServlet.java:117)
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(De
>>> faultManagedContext.java:56)
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithC
>>> ontext(DefaultManagedContext.java:103)
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCo
>>> ntext(DefaultManagedContext.java:53)
>>> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:114)
>>> at com.cloud.api.ApiServlet.doPost(ApiServlet.java:81)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>
>
Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. +91 88 262 30892 | rohit.yadav@shapeblue.com
Blog: bhaisaab.org | Twitter: @_bhaisaab
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: No event publish can be wrapped within db transaction...why?
Posted by Leo Simons <LS...@schubergphilis.com>.
Hi Min,
Thanks for a very clear answer!
However, I'm afraid I still don't get it :-). So...
...do you have any specific example or use case of an external IAM service to integrate with? Is there some kind of design document for me to understand the goals?
I ask because all the ones that I'm familiar with tend to assume that the owner of identity information (and grouping, and possibly other kinds of AAA assertions) is externalized from systems like cloudstack to the identity system, i.e. integration is "the other way around".
So i.e. you would have AD or other LDAP or an SSO server or a SAML implementation (or all of those...), where systems like cloudstack then delegate AAA questions/assertions to those systems, rather than propagating local identities to that central system.
I imagine if you have an external identity provider, you plug in a different implementation of AccountManager (LDAPAccountManager? etc.), and then a CreateAccountCmd would fail with an error saying the server is configured to use <<external thing>> so account creation is unsupported.
cheers!
Leo
On Nov 18, 2014, at 7:50 PM, Min Chen <mi...@citrix.com> wrote:
> Hi Leo,
>
> "NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!" is along the
> same line as "NO AGENT COMMAND CAN BE WRAPPED WITHIN DB TRANSACTION!". The
> rationale behind this is simple: event subscriber execution or agent
> command handling at resource layer may take too long, and we don't want to
> have that long transaction window to hold DB for too long.
> As for your questions about why we bother to use message bus to
> communicate between two java component, there is a reason for it: loose
> coupling. IAMApiServiceImpl is a class in IAM plugin service, which can be
> deployed as a totally different service from CloudStack management server
> and ideally with future 3rd-party authentication/authorization
> integration, they may use a totally different database from "cloud"
> database we are currently using just for simplicity. In this deployment
> architecture, we have to make sure that this IAM service and CloudStack MS
> components are loosely coupled. Message bus provided us a very good
> approach to achieve that.
> As you said, ideally we would like to achieve a prefect transaction
> related to account creation in both CloudStack main component and its
> plugin services, but in reality, this may not work always and big
> transaction will be error-prone for large scale distributed systems,
> especially for this loosely coupled components that are crossing different
> DBs. The plugin architecture in CloudStack is designed to easily
> enable/disable each plugin component without impacting too much on main
> CloudStack components. So in this case, I would personally prefer that we
> should make sure of data integrity in the scope of CloudStack main
> components first and handle potential message handling failure in plugin
> module separately through application level logic.
>
> Thanks
> -min
>
>
>
> On 11/18/14 5:52 AM, "Leo Simons" <LS...@schubergphilis.com> wrote:
>
>> Hi Min, hi Koushik,
>>
>> Cloudstack is shouting at me:
>> NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!
>>
>> (full stack trace below). I've learned this is happening on our
>> systemvm-persistent-config feature branch because it has commit
>> ffaabdc13fde0f0f7b2667a483006e2a4b805f63 but it does not have commit
>> f585dd266188a134a9c8b911376b066b9d3806e8 yet.
>>
>> I'm now trying to understand what's happening here -- the transaction /
>> concurrency / messaging logic gave me significant headache with its
>> triple negatives, nested transaction scoping and home-grown gates, but I
>> think I got it now.
>>
>> As I understand it, in the olde world, creating an account:
>>
>> * opens a database transaction
>> * creates an account in the db
>> * creates the first user in that account in the db
>> * publishes an event
>> * which is listened to by 0 subscribers
>> * commmits the database transaction
>> * check the user is there
>> * opens a database transaction
>> * find the created user in the database
>> * (auto)closes transaction
>> * returns success if the user is in the db
>>
>> this, err, works, but in some other cases, apparently, there are concerns
>> that the db transaction is open too long while message handling happens.
>> So that's why the warning was added, and follow up on, and so now,
>> creating an account:
>>
>> * opens a database transaction
>> * creates an account in the db
>> * creates the first user in that account in the db
>> * commmits the database transaction
>> * publishes an event
>> * which is still listened to by on average 0 subscribers,
>> but there could be an IAM subscriber
>> * check the user is there
>> * opens a database transaction
>> * find the created user in the database
>> * (auto)closes transaction
>> * returns success if the user is in the db
>>
>> The one possible subscriber for account creation is IAMApiServiceImpl,
>> which when receiving the event
>>
>> * opens a database transaction
>> * adds the account to acl_group_account_map
>> * commits the database transaction
>> * finds the domain for the account
>> * opens a database transaction
>> * finds the domain for the account
>> * (auto)closes transaction
>> * finds the domain groups for the domain
>> * opens a database transaction
>> * finds the domain groups for the domain
>> * (auto)closes transaction
>> * for each domain group
>> * opens a database transaction
>> * adds the account to acl_group_account_map
>> * commits the database transaction
>>
>> in other words, if there's 1 domain group and an enabled IAM thingie,
>> this spreads out "make an account" over 6 transactions. Without IAM
>> thingie its 2 two transactions with a no-op message bus thingie in the
>> middle. Is that correct?
>>
>> If so, I don't understand this at all. The pre-November code doesn't make
>> that much sense to me (why query the database? If you don't trust your
>> database its ACID guarantees...why use transactions? Why do we ever need
>> a message bus between two java components in the same classloader?), but
>> the new code scares me.
>>
>> In the case of errors in between transactions, you can end up with
>> accounts that are not in all the groups they should be in. I imagine I
>> would much rather see the whole thing fail, and the complete api call
>> fail, so that I can re-try it as a whole, than end up with a somehow
>> half-initialized account. I.e. have everything account-management-y
>> happen in one transaction which is rolled back on any failure.
>>
>> Any thoughts?
>>
>>
>> Thanks!
>>
>>
>> Leo (who can't ask Hugo since Hugo is at apachecon/ccceu and he isn't :))
>>
>>
>> 2014-11-18 13:36:33,145 ERROR [o.a.c.f.m.MessageBusBase]
>> (qtp1734055321-25:ctx-05df2079 ctx-25ea4461 ctx-3aac3268)
>> NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!
>> com.cloud.utils.exception.CloudRuntimeException:
>> NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!
>> at
>> org.apache.cloudstack.framework.messagebus.MessageBusBase.publish(MessageB
>> usBase.java:167)
>> at
>> com.cloud.user.AccountManagerImpl$2.doInTransaction(AccountManagerImpl.jav
>> a:1052)
>> at
>> com.cloud.user.AccountManagerImpl$2.doInTransaction(AccountManagerImpl.jav
>> a:1027)
>> at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:57)
>> at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
>> at com.cloud.utils.db.Transaction.execute(Transaction.java:54)
>> at
>> com.cloud.user.AccountManagerImpl.createUserAccount(AccountManagerImpl.jav
>> a:1027)
>> at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>> pl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(Ao
>> pUtils.java:317)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoi
>> nt(ReflectiveMethodInvocation.java:183)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle
>> ctiveMethodInvocation.java:150)
>> at
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventIntercep
>> tor.invoke(EventUtils.java:106)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle
>> ctiveMethodInvocation.java:161)
>> at
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:
>> 51)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle
>> ctiveMethodInvocation.java:161)
>> at
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Exp
>> oseInvocationInterceptor.java:91)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle
>> ctiveMethodInvocation.java:172)
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopP
>> roxy.java:204)
>> at com.sun.proxy.$Proxy108.createUserAccount(Unknown Source)
>> at
>> org.apache.cloudstack.api.command.admin.account.CreateAccountCmd.execute(C
>> reateAccountCmd.java:178)
>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
>> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:691)
>> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:514)
>> at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:273)
>> at com.cloud.api.ApiServlet$1.run(ApiServlet.java:117)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(De
>> faultManagedContext.java:56)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithC
>> ontext(DefaultManagedContext.java:103)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCo
>> ntext(DefaultManagedContext.java:53)
>> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:114)
>> at com.cloud.api.ApiServlet.doPost(ApiServlet.java:81)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>
Re: No event publish can be wrapped within db transaction...why?
Posted by Min Chen <mi...@citrix.com>.
Hi Leo,
"NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!" is along the
same line as "NO AGENT COMMAND CAN BE WRAPPED WITHIN DB TRANSACTION!". The
rationale behind this is simple: event subscriber execution or agent
command handling at resource layer may take too long, and we don't want to
have that long transaction window to hold DB for too long.
As for your questions about why we bother to use message bus to
communicate between two java component, there is a reason for it: loose
coupling. IAMApiServiceImpl is a class in IAM plugin service, which can be
deployed as a totally different service from CloudStack management server
and ideally with future 3rd-party authentication/authorization
integration, they may use a totally different database from "cloud"
database we are currently using just for simplicity. In this deployment
architecture, we have to make sure that this IAM service and CloudStack MS
components are loosely coupled. Message bus provided us a very good
approach to achieve that.
As you said, ideally we would like to achieve a prefect transaction
related to account creation in both CloudStack main component and its
plugin services, but in reality, this may not work always and big
transaction will be error-prone for large scale distributed systems,
especially for this loosely coupled components that are crossing different
DBs. The plugin architecture in CloudStack is designed to easily
enable/disable each plugin component without impacting too much on main
CloudStack components. So in this case, I would personally prefer that we
should make sure of data integrity in the scope of CloudStack main
components first and handle potential message handling failure in plugin
module separately through application level logic.
Thanks
-min
On 11/18/14 5:52 AM, "Leo Simons" <LS...@schubergphilis.com> wrote:
>Hi Min, hi Koushik,
>
>Cloudstack is shouting at me:
> NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!
>
>(full stack trace below). I've learned this is happening on our
>systemvm-persistent-config feature branch because it has commit
>ffaabdc13fde0f0f7b2667a483006e2a4b805f63 but it does not have commit
>f585dd266188a134a9c8b911376b066b9d3806e8 yet.
>
>I'm now trying to understand what's happening here -- the transaction /
>concurrency / messaging logic gave me significant headache with its
>triple negatives, nested transaction scoping and home-grown gates, but I
>think I got it now.
>
>As I understand it, in the olde world, creating an account:
>
> * opens a database transaction
> * creates an account in the db
> * creates the first user in that account in the db
> * publishes an event
> * which is listened to by 0 subscribers
> * commmits the database transaction
> * check the user is there
> * opens a database transaction
> * find the created user in the database
> * (auto)closes transaction
> * returns success if the user is in the db
>
>this, err, works, but in some other cases, apparently, there are concerns
>that the db transaction is open too long while message handling happens.
>So that's why the warning was added, and follow up on, and so now,
>creating an account:
>
> * opens a database transaction
> * creates an account in the db
> * creates the first user in that account in the db
> * commmits the database transaction
> * publishes an event
> * which is still listened to by on average 0 subscribers,
> but there could be an IAM subscriber
> * check the user is there
> * opens a database transaction
> * find the created user in the database
> * (auto)closes transaction
> * returns success if the user is in the db
>
>The one possible subscriber for account creation is IAMApiServiceImpl,
>which when receiving the event
>
> * opens a database transaction
> * adds the account to acl_group_account_map
> * commits the database transaction
> * finds the domain for the account
> * opens a database transaction
> * finds the domain for the account
> * (auto)closes transaction
> * finds the domain groups for the domain
> * opens a database transaction
> * finds the domain groups for the domain
> * (auto)closes transaction
> * for each domain group
> * opens a database transaction
> * adds the account to acl_group_account_map
> * commits the database transaction
>
>in other words, if there's 1 domain group and an enabled IAM thingie,
>this spreads out "make an account" over 6 transactions. Without IAM
>thingie its 2 two transactions with a no-op message bus thingie in the
>middle. Is that correct?
>
>If so, I don't understand this at all. The pre-November code doesn't make
>that much sense to me (why query the database? If you don't trust your
>database its ACID guarantees...why use transactions? Why do we ever need
>a message bus between two java components in the same classloader?), but
>the new code scares me.
>
>In the case of errors in between transactions, you can end up with
>accounts that are not in all the groups they should be in. I imagine I
>would much rather see the whole thing fail, and the complete api call
>fail, so that I can re-try it as a whole, than end up with a somehow
>half-initialized account. I.e. have everything account-management-y
>happen in one transaction which is rolled back on any failure.
>
>Any thoughts?
>
>
>Thanks!
>
>
>Leo (who can't ask Hugo since Hugo is at apachecon/ccceu and he isn't :))
>
>
>2014-11-18 13:36:33,145 ERROR [o.a.c.f.m.MessageBusBase]
>(qtp1734055321-25:ctx-05df2079 ctx-25ea4461 ctx-3aac3268)
> NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!
> com.cloud.utils.exception.CloudRuntimeException:
> NO EVENT PUBLISH CAN BE WRAPPED WITHIN DB TRANSACTION!
> at
>org.apache.cloudstack.framework.messagebus.MessageBusBase.publish(MessageB
>usBase.java:167)
> at
>com.cloud.user.AccountManagerImpl$2.doInTransaction(AccountManagerImpl.jav
>a:1052)
> at
>com.cloud.user.AccountManagerImpl$2.doInTransaction(AccountManagerImpl.jav
>a:1027)
> at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:57)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:54)
> at
>com.cloud.user.AccountManagerImpl.createUserAccount(AccountManagerImpl.jav
>a:1027)
> at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source)
> at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>pl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
>org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(Ao
>pUtils.java:317)
> at
>org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoi
>nt(ReflectiveMethodInvocation.java:183)
> at
>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle
>ctiveMethodInvocation.java:150)
> at
>org.apache.cloudstack.network.contrail.management.EventUtils$EventIntercep
>tor.invoke(EventUtils.java:106)
> at
>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle
>ctiveMethodInvocation.java:161)
> at
>com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:
>51)
> at
>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle
>ctiveMethodInvocation.java:161)
> at
>org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Exp
>oseInvocationInterceptor.java:91)
> at
>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refle
>ctiveMethodInvocation.java:172)
> at
>org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopP
>roxy.java:204)
> at com.sun.proxy.$Proxy108.createUserAccount(Unknown Source)
> at
>org.apache.cloudstack.api.command.admin.account.CreateAccountCmd.execute(C
>reateAccountCmd.java:178)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:691)
> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:514)
> at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:273)
> at com.cloud.api.ApiServlet$1.run(ApiServlet.java:117)
> at
>org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(De
>faultManagedContext.java:56)
> at
>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithC
>ontext(DefaultManagedContext.java:103)
> at
>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCo
>ntext(DefaultManagedContext.java:53)
> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:114)
> at com.cloud.api.ApiServlet.doPost(ApiServlet.java:81)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)