You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2014/01/03 11:19:54 UTC
[jira] [Commented] (CLOUDSTACK-5730)
[Automation]DirectAgentAttache's send command fails due to
ArrayIndexOutOfBoundsException; Unable to cleanup hypervisor's
datastructures as part of Delete Account Job
[ https://issues.apache.org/jira/browse/CLOUDSTACK-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13861395#comment-13861395 ]
ASF subversion and git services commented on CLOUDSTACK-5730:
-------------------------------------------------------------
Commit 6ce46e94f522f08c91db2cde33829f5b41539397 in branch refs/heads/4.3 from [~harikrishna.patnala]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=6ce46e9 ]
CLOUDSTACK-5730: [Automation]DirectAgentAttache's send command fails due to ArrayIndexOutOfBoundsException; Unable to cleanup hypervisor's datastructures as part of Delete Account Job
Signed-off-by: Koushik Das <ko...@apache.org>
> [Automation]DirectAgentAttache's send command fails due to ArrayIndexOutOfBoundsException; Unable to cleanup hypervisor's datastructures as part of Delete Account Job
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-5730
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5730
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Management Server
> Affects Versions: 4.3.0
> Reporter: Chandan Purushothama
> Assignee: Harikrishna Patnala
> Priority: Critical
> Fix For: 4.3.0
>
>
> =======================================================
> ArrayIndexOutOfBoundsException during DeleteAccount Job Execution:
> =======================================================
> 2014-01-03 07:39:16,752 DEBUG [c.c.v.VirtualMachineManagerImpl] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Destroying vm VM[User|QA-88cfa2c8-e8ef-4133-8cf1-002ea3cb0c7b]
> 2014-01-03 07:39:16,752 DEBUG [c.c.v.VirtualMachineManagerImpl] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Cleaning up NICS
> 2014-01-03 07:39:16,752 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Cleaning network for vm: 44
> 2014-01-03 07:39:16,763 DEBUG [c.c.n.NetworkModelImpl] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Service SecurityGroup is not supported in the network id=229
> 2014-01-03 07:39:16,780 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Removed nic id=81
> 2014-01-03 07:39:16,782 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Revoving nic secondary ip entry ...
> 2014-01-03 07:39:16,783 DEBUG [c.c.v.VirtualMachineManagerImpl] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Cleaning up hypervisor data structures (ex. SRs in XenServer) for managed storage
> 2014-01-03 07:39:16,796 DEBUG [c.c.a.ApiServlet] (catalina-exec-5:ctx-f044cb4e) ===START=== 10.147.38.149 -- GET signature=shTK1s%2FnQv84C1gwQYEUo%2F5WFW4%3D&apiKey=SyITk_W_BpbLs73aYJqNRBu9Sir1d7k6hqoKVUmFCMSVG-vshSHHKOdj_nVVTOQHqyvfR3ZSXhDzL2QUquSdJA&command=queryAsyncJobResult&response=json&jobid=fb908e1c-cdf8-4ae9-8c0d-09116061c3f7
> 2014-01-03 07:39:16,800 DEBUG [c.c.a.t.Request] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Seq 1-678298558: Sending { Cmd , MgmtId: 6631563722783, via: 1(Rack1Pod1Host15), Ver: v1, Flags: 100011, [] }
> 2014-01-03 07:39:16,801 DEBUG [c.c.a.t.Request] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Seq 1-678298558: Executing: { Cmd , MgmtId: 6631563722783, via: 1(Rack1Pod1Host15), Ver: v1, Flags: 100011, [] }
> 2014-01-03 07:39:16,801 WARN [c.c.a.m.AgentAttache] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Seq 1-678298558: Unable to send due to
> java.lang.ArrayIndexOutOfBoundsException: 0
> at com.cloud.agent.manager.DirectAgentAttache.send(DirectAgentAttache.java:100)
> at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:374)
> at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:398)
> at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:394)
> at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:920)
> at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:497)
> at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:448)
> at com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:1704)
> at sun.reflect.GeneratedMethodAccessor429.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy169.expunge(Unknown Source)
> at com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManagerImpl.java:629)
> at com.cloud.user.AccountManagerImpl.deleteAccount(AccountManagerImpl.java:561)
> at com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountManagerImpl.java:1308)
> at sun.reflect.GeneratedMethodAccessor478.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy82.deleteUserAccount(Unknown Source)
> at org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccount(RegionManagerImpl.java:193)
> at org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccount(RegionServiceImpl.java:118)
> at org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd.execute(DeleteAccountCmd.java:101)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
> at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
> at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2014-01-03 07:39:16,836 DEBUG [c.c.a.m.AgentAttache] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Seq 1-678298558: Cancelling.
> 2014-01-03 07:39:16,837 WARN [c.c.v.UserVmManagerImpl] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Unable to expunge VM[User|QA-88cfa2c8-e8ef-4133-8cf1-002ea3cb0c7b]
> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Problem due to other exception 0
> at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:389)
> at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:398)
> at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:394)
> at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:920)
> at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:497)
> at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:448)
> at com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:1704)
> at sun.reflect.GeneratedMethodAccessor429.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy169.expunge(Unknown Source)
> at com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManagerImpl.java:629)
> at com.cloud.user.AccountManagerImpl.deleteAccount(AccountManagerImpl.java:561)
> at com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountManagerImpl.java:1308)
> at sun.reflect.GeneratedMethodAccessor478.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy82.deleteUserAccount(Unknown Source)
> at org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccount(RegionManagerImpl.java:193)
> at org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccount(RegionServiceImpl.java:118)
> at org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd.execute(DeleteAccountCmd.java:101)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
> at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
> at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2014-01-03 07:39:16,839 ERROR [c.c.u.AccountManagerImpl] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Unable to expunge vm: 44
> 2014-01-03 07:39:16,869 INFO [c.c.u.AccountManagerImpl] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) deleteAccount: Deleted 1 network groups for account 26
> 2014-01-03 07:39:16,870 INFO [c.c.u.AccountManagerImpl] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) deleteAccount: Deleted 0 affinity groups for account 26
> 2014-01-03 07:39:16,871 DEBUG [c.c.u.AccountManagerImpl] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Deleting networks for account 26
> 2014-01-03 07:39:16,884 DEBUG [c.c.a.ApiServlet] (catalina-exec-5:ctx-f044cb4e ctx-e5d3f936 ctx-efc2b479) ===END=== 10.147.38.149 -- GET signature=shTK1s%2FnQv84C1gwQYEUo%2F5WFW4%3D&apiKey=SyITk_W_BpbLs73aYJqNRBu9Sir1d7k6hqoKVUmFCMSVG-vshSHHKOdj_nVVTOQHqyvfR3ZSXhDzL2QUquSdJA&command=queryAsyncJobResult&response=json&jobid=fb908e1c-cdf8-4ae9-8c0d-09116061c3f7
> 2014-01-03 07:39:16,894 WARN [o.a.c.e.o.NetworkOrchestrator] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Can't delete the network, not all user vms are expunged. Vm VM[User|QA-03d9e9fa-081a-4175-9bf2-2508034b6e17] is in Expunging state
> 2014-01-03 07:39:16,894 WARN [c.c.u.AccountManagerImpl] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Unable to destroy network Ntwk[229|Guest|8] as a part of account id=26 cleanup.
> 2014-01-03 07:39:16,895 DEBUG [c.c.u.AccountManagerImpl] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Deleting vpcs for account 26
> 2014-01-03 07:39:16,897 DEBUG [c.c.u.AccountManagerImpl] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Deleting site-to-site VPN customer gateways for account 26
> 2014-01-03 07:39:16,908 INFO [c.c.u.AccountManagerImpl] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) deleteAccount: Released 0 dedicated guest vlan ranges from account 26
> 2014-01-03 07:39:17,034 INFO [c.c.u.AccountManagerImpl] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Cleanup for account 26 is needed.
> 2014-01-03 07:39:17,048 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Complete async job-359, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.SuccessResponse/null/{"success":true}
> 2014-01-03 07:39:17,056 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-72:ctx-d82a5609) Done executing org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd for job-359
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)