You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Grayson Head <gr...@graysonhead.net> on 2015/04/06 22:27:13 UTC
Unable to remove or modify VPC
One of my coworkers has created a VPC that we are unable to modify or remove.
The coworker said he was testing the site-to-site VPN functionality,
and ended up having to create a seperate VPC instance to get it
working. Now we are unable to remove this VPC instance using the web
interface or the API. I suspect based on the log outputs that the
site-to-site VPN is what is causing this problem, but the VPC shows
that it doesn't have any site-to-site connections configured.
The VPC still has one external IP address allocated to it, which I
think was the one he was using as the VPN gateway. I am also unable to
remove this IP address, it fails with an "internal server" error. The
following is the management-server.log when attempting to release this
IP address.
2015-04-06 13:18:42,427 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-96:ctx-428fa98b job-6262) Add job-6262 into job
monitoring
2015-04-06 13:18:42,427 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-96:ctx-428fa98b job-6262) Executing AsyncJobVO
{id:6262, userId: 3, accountId: 3, instanceType: IpAddress,
instanceId: 9, cmd:
org.apache.cloudstack.api.command.user.address.DisassociateIPAddrCmd,
cmdInfo: {"response":"json","id":"3d2205a4-41e9-4cf7-9261-d1ae63ce863f","sessionkey":"gSf69NNJJZ0SzrIEz1fwOQJpRTo\u003d","ctxDetails":"{\"com.cloud.network.IpAddress\":\"3d2205a4-41e9-4cf7-9261-d1ae63ce863f\"}","cmdEventType":"NET.IPRELEASE","ctxUserId":"3","httpmethod":"GET","_":"1428351522388","uuid":"3d2205a4-41e9-4cf7-9261-d1ae63ce863f","ctxAccountId":"3","ctxStartEventId":"7705"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 345050016868, completeMsid: null, lastUpdated:
null, lastPolled: null, created: null}
2015-04-06 13:18:42,427 DEBUG [c.c.a.ApiServlet]
(catalina-exec-14:ctx-91e999f7 ctx-49efc96a) ===END=== 10.233.51.176
-- GET command=disassociateIpAddress&response=json&sessionkey=gSf69NNJJZ0SzrIEz1fwOQJpRTo%3D&id=3d2205a4-41e9-4cf7-9261-d1ae63ce863f&_=1428351522388
2015-04-06 13:18:42,451 DEBUG [c.c.n.IpAddressManagerImpl]
(API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Revoking all
Firewallrules as a part of public IP id=9 release...
2015-04-06 13:18:42,462 DEBUG [c.c.n.f.FirewallManagerImpl]
(API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Releasing 0
firewall rules for ip id=9
2015-04-06 13:18:42,464 DEBUG [c.c.n.f.FirewallManagerImpl]
(API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) There are no
firewall rules to apply
2015-04-06 13:18:42,466 DEBUG [c.c.n.f.FirewallManagerImpl]
(API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Successfully
released firewall rules for ip id=9 and # of rules now = 0
2015-04-06 13:18:42,475 DEBUG [c.c.n.IpAddressManagerImpl]
(API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Revoking all
PortForwarding/StaticNat rules as a part of public IP id=9 release...
2015-04-06 13:18:42,477 DEBUG [c.c.n.r.RulesManagerImpl]
(API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Releasing 0
port forwarding rules for ip id=9
2015-04-06 13:18:42,478 DEBUG [c.c.n.r.RulesManagerImpl]
(API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Releasing 0
static nat rules for ip id=9
2015-04-06 13:18:42,480 DEBUG [c.c.n.r.RulesManagerImpl]
(API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) There are no
port forwarding rules to apply for ip id=9
2015-04-06 13:18:42,482 DEBUG [c.c.n.r.RulesManagerImpl]
(API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) There are no
static nat rules to apply for ip id=9
2015-04-06 13:18:42,483 DEBUG [c.c.n.r.RulesManagerImpl]
(API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Source ip
id=Ip[66.194.167.148-1] is not one to one nat
2015-04-06 13:18:42,487 DEBUG [c.c.n.r.RulesManagerImpl]
(API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Successfully
released rules for ip id=9 and # of rules now = 0
2015-04-06 13:18:42,487 DEBUG [c.c.n.IpAddressManagerImpl]
(API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Revoking all
LoadBalancing rules as a part of public IP id=9 release...
2015-04-06 13:18:42,488 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Found 0 lb
rules to cleanup
2015-04-06 13:18:42,488 DEBUG [c.c.n.IpAddressManagerImpl]
(API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Cleaning up
remote access vpns as a part of public IP id=9 release...
2015-04-06 13:18:42,523 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-96:ctx-428fa98b job-6262) Unexpected exception while
executing org.apache.cloudstack.api.command.user.address.DisassociateIPAddrCmd
java.lang.NullPointerException
at com.cloud.network.NetworkModelImpl.getPhysicalNetworkId(NetworkModelImpl.java:1374)
at com.cloud.network.element.VirtualRouterElement.canHandle(VirtualRouterElement.java:152)
at com.cloud.network.element.VirtualRouterElement.stopVpn(VirtualRouterElement.java:443)
at com.cloud.network.vpn.RemoteAccessVpnManagerImpl.destroyRemoteAccessVpnForIp(RemoteAccessVpnManagerImpl.java:305)
at sun.reflect.GeneratedMethodAccessor670.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy231.destroyRemoteAccessVpnForIp(Unknown Source)
at com.cloud.network.IpAddressManagerImpl.cleanupIpResources(IpAddressManagerImpl.java:556)
at com.cloud.network.IpAddressManagerImpl.disassociatePublicIpAddress(IpAddressManagerImpl.java:571)
at com.cloud.network.NetworkServiceImpl.releaseIpAddressInternal(NetworkServiceImpl.java:935)
at com.cloud.network.NetworkServiceImpl.releaseIpAddress(NetworkServiceImpl.java:895)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy156.releaseIpAddress(Unknown Source)
at org.apache.cloudstack.api.command.user.address.DisassociateIPAddrCmd.execute(DisassociateIPAddrCmd.java:80)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-04-06 13:18:42,525 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-96:ctx-428fa98b job-6262) Complete async job-6262,
jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530}
2015-04-06 13:18:42,534 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-96:ctx-428fa98b job-6262) Done executing
org.apache.cloudstack.api.command.user.address.DisassociateIPAddrCmd
for job-6262
2015-04-06 13:18:42,539 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-96:ctx-428fa98b job-6262) Remove job-6262 from job
monitoring
2015-04-06 13:18:42,800 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-11:null) SeqA 9-147481: Processing Seq 9-147481:
{ Cmd , MgmtId: -1, via: 9, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":78,"_loadInfo":"{\n
\"connections\": []\n}","wait":0}}] }
2015-04-06 13:18:42,808 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-11:null) SeqA 9-147481: Sending Seq 9-147481: {
Ans: , MgmtId: 345050016868, via: 9, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}]
}
2015-04-06 13:18:45,447 DEBUG [c.c.a.ApiServlet]
(catalina-exec-5:ctx-931b2c88) ===START=== 10.233.51.176 -- GET
command=queryAsyncJobResult&jobId=61329013-6b40-43d2-ad1b-8fba0325bae1&response=json&sessionkey=gSf69NNJJZ0SzrIEz1fwOQJpRTo%3D&_=1428351525444
2015-04-06 13:18:45,478 DEBUG [c.c.a.ApiServlet]
(catalina-exec-5:ctx-931b2c88 ctx-7b984e22) ===END=== 10.233.51.176
-- GET command=queryAsyncJobResult&jobId=61329013-6b40-43d2-ad1b-8fba0325bae1&response=json&sessionkey=gSf69NNJJZ0SzrIEz1fwOQJpRTo%3D&_=1428351525444
Removing the VPC also gives an internal server error and outputs the
following to the logs.
2015-04-06 07:03:10,148 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(catalina-exec-12:ctx-1b8cdf03 ctx-97a67e82) submit async job-6257,
details: AsyncJobVO {id:6257, userId: 3, accountId: 3, instanceType:
None, instanceId: null, cmd:
org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd, cmdInfo:
{"response":"json","id":"02158f27-88f7-4ffe-83bf-60cfea333dee","sessionkey":"68X4liCjVtaZUrDpChfoXD1a2Mo\u003d","ctxDetails":"{\"com.cloud.network.vpc.Vpc\":\"02158f27-88f7-4ffe-83bf-60cfea333dee\"}","cmdEventType":"VPC.DELETE","ctxUserId":"3","httpmethod":"GET","_":"1428328990097","uuid":"02158f27-88f7-4ffe-83bf-60cfea333dee","ctxAccountId":"3","ctxStartEventId":"7627"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 345050016868, completeMsid: null, lastUpdated:
null, lastPolled: null, created: null}
2015-04-06 07:03:10,148 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-91:ctx-9fe47d69 job-6257) Add job-6257 into job
monitoring
2015-04-06 07:03:10,148 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257) Executing AsyncJobVO
{id:6257, userId: 3, accountId: 3, instanceType: None, instanceId:
null, cmd: org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd,
cmdInfo: {"response":"json","id":"02158f27-88f7-4ffe-83bf-60cfea333dee","sessionkey":"68X4liCjVtaZUrDpChfoXD1a2Mo\u003d","ctxDetails":"{\"com.cloud.network.vpc.Vpc\":\"02158f27-88f7-4ffe-83bf-60cfea333dee\"}","cmdEventType":"VPC.DELETE","ctxUserId":"3","httpmethod":"GET","_":"1428328990097","uuid":"02158f27-88f7-4ffe-83bf-60cfea333dee","ctxAccountId":"3","ctxStartEventId":"7627"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 345050016868, completeMsid: null, lastUpdated:
null, lastPolled: null, created: null}
2015-04-06 07:03:10,149 DEBUG [c.c.a.ApiServlet]
(catalina-exec-12:ctx-1b8cdf03 ctx-97a67e82) ===END=== 10.233.51.176
-- GET command=deleteVPC&response=json&sessionkey=68X4liCjVtaZUrDpChfoXD1a2Mo%3D&id=02158f27-88f7-4ffe-83bf-60cfea333dee&_=1428328990097
2015-04-06 07:03:10,164 DEBUG [c.c.n.v.VpcManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Destroying
vpc [VPC [9-VPC]
2015-04-06 07:03:10,169 DEBUG [c.c.n.v.VpcManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Shutting down
vpc [VPC [9-VPC]
2015-04-06 07:03:10,172 DEBUG [c.c.n.v.VpcManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Vpc [VPC
[9-VPC] has been shutdown succesfully
2015-04-06 07:03:10,172 DEBUG [c.c.n.v.VpcManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Cleaning up
resources for vpc id=9
2015-04-06 07:03:10,172 DEBUG [c.c.n.v.VpcManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Cleaning up
existed site to site VPN connections
2015-04-06 07:03:10,174 DEBUG [c.c.n.v.VpcManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Cleaning up
existed site to site VPN gateways
2015-04-06 07:03:10,177 DEBUG [c.c.n.v.VpcManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Releasing ips
for vpc id=9 as a part of vpc cleanup
2015-04-06 07:03:10,177 DEBUG [c.c.n.IpAddressManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Revoking all
Firewallrules as a part of public IP id=9 release...
2015-04-06 07:03:10,189 DEBUG [c.c.n.f.FirewallManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Releasing 0
firewall rules for ip id=9
2015-04-06 07:03:10,190 DEBUG [c.c.n.f.FirewallManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) There are no
firewall rules to apply
2015-04-06 07:03:10,192 DEBUG [c.c.n.f.FirewallManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Successfully
released firewall rules for ip id=9 and # of rules now = 0
2015-04-06 07:03:10,202 DEBUG [c.c.n.IpAddressManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Revoking all
PortForwarding/StaticNat rules as a part of public IP id=9 release...
2015-04-06 07:03:10,204 DEBUG [c.c.n.r.RulesManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Releasing 0
port forwarding rules for ip id=9
2015-04-06 07:03:10,206 DEBUG [c.c.n.r.RulesManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Releasing 0
static nat rules for ip id=9
2015-04-06 07:03:10,208 DEBUG [c.c.n.r.RulesManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) There are no
port forwarding rules to apply for ip id=9
2015-04-06 07:03:10,209 DEBUG [c.c.n.r.RulesManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) There are no
static nat rules to apply for ip id=9
2015-04-06 07:03:10,211 DEBUG [c.c.n.r.RulesManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Source ip
id=Ip[66.194.167.148-1] is not one to one nat
2015-04-06 07:03:10,214 DEBUG [c.c.n.r.RulesManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Successfully
released rules for ip id=9 and # of rules now = 0
2015-04-06 07:03:10,214 DEBUG [c.c.n.IpAddressManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Revoking all
LoadBalancing rules as a part of public IP id=9 release...
2015-04-06 07:03:10,216 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Found 0 lb
rules to cleanup
2015-04-06 07:03:10,216 DEBUG [c.c.n.IpAddressManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Cleaning up
remote access vpns as a part of public IP id=9 release...
2015-04-06 07:03:10,254 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-91:ctx-9fe47d69 job-6257) Unexpected exception while
executing org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd
java.lang.NullPointerException
at com.cloud.network.NetworkModelImpl.getPhysicalNetworkId(NetworkModelImpl.java:1374)
at com.cloud.network.element.VirtualRouterElement.canHandle(VirtualRouterElement.java:152)
at com.cloud.network.element.VirtualRouterElement.stopVpn(VirtualRouterElement.java:443)
at com.cloud.network.vpn.RemoteAccessVpnManagerImpl.destroyRemoteAccessVpnForIp(RemoteAccessVpnManagerImpl.java:305)
at sun.reflect.GeneratedMethodAccessor670.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy231.destroyRemoteAccessVpnForIp(Unknown Source)
at com.cloud.network.IpAddressManagerImpl.cleanupIpResources(IpAddressManagerImpl.java:556)
at com.cloud.network.IpAddressManagerImpl.disassociatePublicIpAddress(IpAddressManagerImpl.java:571)
at com.cloud.network.vpc.VpcManagerImpl.cleanupVpcResources(VpcManagerImpl.java:1429)
at com.cloud.network.vpc.VpcManagerImpl.destroyVpc(VpcManagerImpl.java:938)
at com.cloud.network.vpc.VpcManagerImpl.deleteVpc(VpcManagerImpl.java:899)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy161.deleteVpc(Unknown Source)
at org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd.execute(DeleteVPCCmd.java:76)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-04-06 07:03:10,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257) Complete async job-6257,
jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530}
2015-04-06 07:03:10,266 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-91:ctx-9fe47d69 job-6257) Done executing
org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd for job-6257
2015-04-06 07:03:10,273 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-91:ctx-9fe47d69 job-6257) Remove job-6257 from job
monitoring
2015-04-06 07:03:11,046 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-1:null) Ping from 13
2015-04-06 07:03:11,046 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) Process host VM state report from ping
process. host: 13
2015-04-06 07:03:11,066 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) Process VM state report. host: 13,
number of records in report: 9
2015-04-06 07:03:11,066 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM state report. host: 13, vm id: 959,
power state: PowerOn
2015-04-06 07:03:11,068 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM power state does not change, skip DB
writing. vm id: 959
2015-04-06 07:03:11,068 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM state report. host: 13, vm id: 968,
power state: PowerOn
2015-04-06 07:03:11,071 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM power state does not change, skip DB
writing. vm id: 968
2015-04-06 07:03:11,071 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM state report. host: 13, vm id: 975,
power state: PowerOn
2015-04-06 07:03:11,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM power state does not change, skip DB
writing. vm id: 975
2015-04-06 07:03:11,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM state report. host: 13, vm id: 974,
power state: PowerOn
2015-04-06 07:03:11,076 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM power state does not change, skip DB
writing. vm id: 974
2015-04-06 07:03:11,076 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM state report. host: 13, vm id: 871,
power state: PowerOn
2015-04-06 07:03:11,079 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM power state does not change, skip DB
writing. vm id: 871
2015-04-06 07:03:11,079 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM state report. host: 13, vm id: 78,
power state: PowerOn
2015-04-06 07:03:11,081 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM power state does not change, skip DB
writing. vm id: 78
2015-04-06 07:03:11,081 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM state report. host: 13, vm id: 893,
power state: PowerOn
2015-04-06 07:03:11,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM power state does not change, skip DB
writing. vm id: 893
2015-04-06 07:03:11,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM state report. host: 13, vm id: 914,
power state: PowerOn
2015-04-06 07:03:11,086 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM power state does not change, skip DB
writing. vm id: 914
2015-04-06 07:03:11,086 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM state report. host: 13, vm id: 912,
power state: PowerOn
2015-04-06 07:03:11,089 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) VM power state does not change, skip DB
writing. vm id: 912
2015-04-06 07:03:11,095 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-1:null) Done with process of VM state report.
host: 13
2015-04-06 07:03:13,165 DEBUG [c.c.a.ApiServlet]
(catalina-exec-11:ctx-ad36fed0) ===START=== 10.233.51.176 -- GET
command=queryAsyncJobResult&jobId=85d5c719-6114-494d-bb52-26f43372976d&response=json&sessionkey=68X4liCjVtaZUrDpChfoXD1a2Mo%3D&_=1428328993153
2015-04-06 07:03:13,191 DEBUG [c.c.a.ApiServlet]
(catalina-exec-11:ctx-ad36fed0 ctx-5af312b0) ===END=== 10.233.51.176
-- GET command=queryAsyncJobResult&jobId=85d5c719-6114-494d-bb52-26f43372976d&response=json&sessionkey=68X4liCjVtaZUrDpChfoXD1a2Mo%3D&_=1428328993153
I'm at a loss as to what could be causing this, and I don't want to
hack the database to fix this without completely understanding how all
of these components interact with each other. Any help would be
appreciated.
Re: Unable to remove or modify VPC
Posted by Grayson Head <gr...@graysonhead.net>.
Sanjeev,
I have opened a bug: https://issues.apache.org/jira/browse/CLOUDSTACK-8371
Thanks.
On Tue, Apr 7, 2015 at 4:01 AM, Sanjeev N <sa...@apache.org> wrote:
> Looks like a bug to me. Please open a bug in https://issues.apache.org/
>
> On Tue, Apr 7, 2015 at 1:57 AM, Grayson Head <gr...@graysonhead.net>
> wrote:
>
>> One of my coworkers has created a VPC that we are unable to modify or
>> remove.
>>
>> The coworker said he was testing the site-to-site VPN functionality,
>> and ended up having to create a seperate VPC instance to get it
>> working. Now we are unable to remove this VPC instance using the web
>> interface or the API. I suspect based on the log outputs that the
>> site-to-site VPN is what is causing this problem, but the VPC shows
>> that it doesn't have any site-to-site connections configured.
>>
>> The VPC still has one external IP address allocated to it, which I
>> think was the one he was using as the VPN gateway. I am also unable to
>> remove this IP address, it fails with an "internal server" error. The
>> following is the management-server.log when attempting to release this
>> IP address.
>>
>> 2015-04-06 13:18:42,427 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (API-Job-Executor-96:ctx-428fa98b job-6262) Add job-6262 into job
>> monitoring
>> 2015-04-06 13:18:42,427 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-96:ctx-428fa98b job-6262) Executing AsyncJobVO
>> {id:6262, userId: 3, accountId: 3, instanceType: IpAddress,
>> instanceId: 9, cmd:
>> org.apache.cloudstack.api.command.user.address.DisassociateIPAddrCmd,
>> cmdInfo:
>> {"response":"json","id":"3d2205a4-41e9-4cf7-9261-d1ae63ce863f","sessionkey":"gSf69NNJJZ0SzrIEz1fwOQJpRTo\u003d","ctxDetails":"{\"com.cloud.network.IpAddress\":\"3d2205a4-41e9-4cf7-9261-d1ae63ce863f\"}","cmdEventType":"NET.IPRELEASE","ctxUserId":"3","httpmethod":"GET","_":"1428351522388","uuid":"3d2205a4-41e9-4cf7-9261-d1ae63ce863f","ctxAccountId":"3","ctxStartEventId":"7705"},
>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> result: null, initMsid: 345050016868, completeMsid: null, lastUpdated:
>> null, lastPolled: null, created: null}
>> 2015-04-06 13:18:42,427 DEBUG [c.c.a.ApiServlet]
>> (catalina-exec-14:ctx-91e999f7 ctx-49efc96a) ===END=== 10.233.51.176
>> -- GET
>> command=disassociateIpAddress&response=json&sessionkey=gSf69NNJJZ0SzrIEz1fwOQJpRTo%3D&id=3d2205a4-41e9-4cf7-9261-d1ae63ce863f&_=1428351522388
>> 2015-04-06 13:18:42,451 DEBUG [c.c.n.IpAddressManagerImpl]
>> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Revoking all
>> Firewallrules as a part of public IP id=9 release...
>> 2015-04-06 13:18:42,462 DEBUG [c.c.n.f.FirewallManagerImpl]
>> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Releasing 0
>> firewall rules for ip id=9
>> 2015-04-06 13:18:42,464 DEBUG [c.c.n.f.FirewallManagerImpl]
>> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) There are no
>> firewall rules to apply
>> 2015-04-06 13:18:42,466 DEBUG [c.c.n.f.FirewallManagerImpl]
>> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Successfully
>> released firewall rules for ip id=9 and # of rules now = 0
>> 2015-04-06 13:18:42,475 DEBUG [c.c.n.IpAddressManagerImpl]
>> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Revoking all
>> PortForwarding/StaticNat rules as a part of public IP id=9 release...
>> 2015-04-06 13:18:42,477 DEBUG [c.c.n.r.RulesManagerImpl]
>> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Releasing 0
>> port forwarding rules for ip id=9
>> 2015-04-06 13:18:42,478 DEBUG [c.c.n.r.RulesManagerImpl]
>> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Releasing 0
>> static nat rules for ip id=9
>> 2015-04-06 13:18:42,480 DEBUG [c.c.n.r.RulesManagerImpl]
>> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) There are no
>> port forwarding rules to apply for ip id=9
>> 2015-04-06 13:18:42,482 DEBUG [c.c.n.r.RulesManagerImpl]
>> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) There are no
>> static nat rules to apply for ip id=9
>> 2015-04-06 13:18:42,483 DEBUG [c.c.n.r.RulesManagerImpl]
>> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Source ip
>> id=Ip[66.194.167.148-1] is not one to one nat
>> 2015-04-06 13:18:42,487 DEBUG [c.c.n.r.RulesManagerImpl]
>> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Successfully
>> released rules for ip id=9 and # of rules now = 0
>> 2015-04-06 13:18:42,487 DEBUG [c.c.n.IpAddressManagerImpl]
>> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Revoking all
>> LoadBalancing rules as a part of public IP id=9 release...
>> 2015-04-06 13:18:42,488 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
>> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Found 0 lb
>> rules to cleanup
>> 2015-04-06 13:18:42,488 DEBUG [c.c.n.IpAddressManagerImpl]
>> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Cleaning up
>> remote access vpns as a part of public IP id=9 release...
>> 2015-04-06 13:18:42,523 ERROR [c.c.a.ApiAsyncJobDispatcher]
>> (API-Job-Executor-96:ctx-428fa98b job-6262) Unexpected exception while
>> executing
>> org.apache.cloudstack.api.command.user.address.DisassociateIPAddrCmd
>> java.lang.NullPointerException
>> at
>> com.cloud.network.NetworkModelImpl.getPhysicalNetworkId(NetworkModelImpl.java:1374)
>> at
>> com.cloud.network.element.VirtualRouterElement.canHandle(VirtualRouterElement.java:152)
>> at
>> com.cloud.network.element.VirtualRouterElement.stopVpn(VirtualRouterElement.java:443)
>> at
>> com.cloud.network.vpn.RemoteAccessVpnManagerImpl.destroyRemoteAccessVpnForIp(RemoteAccessVpnManagerImpl.java:305)
>> at sun.reflect.GeneratedMethodAccessor670.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>> at
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>> at
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>> at
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>> at com.sun.proxy.$Proxy231.destroyRemoteAccessVpnForIp(Unknown Source)
>> at
>> com.cloud.network.IpAddressManagerImpl.cleanupIpResources(IpAddressManagerImpl.java:556)
>> at
>> com.cloud.network.IpAddressManagerImpl.disassociatePublicIpAddress(IpAddressManagerImpl.java:571)
>> at
>> com.cloud.network.NetworkServiceImpl.releaseIpAddressInternal(NetworkServiceImpl.java:935)
>> at
>> com.cloud.network.NetworkServiceImpl.releaseIpAddress(NetworkServiceImpl.java:895)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>> at
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>> at
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>> at
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>> at com.sun.proxy.$Proxy156.releaseIpAddress(Unknown Source)
>> at
>> org.apache.cloudstack.api.command.user.address.DisassociateIPAddrCmd.execute(DisassociateIPAddrCmd.java:80)
>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
>> at
>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2015-04-06 13:18:42,525 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-96:ctx-428fa98b job-6262) Complete async job-6262,
>> jobStatus: FAILED, resultCode: 530, result:
>>
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530}
>> 2015-04-06 13:18:42,534 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-96:ctx-428fa98b job-6262) Done executing
>> org.apache.cloudstack.api.command.user.address.DisassociateIPAddrCmd
>> for job-6262
>> 2015-04-06 13:18:42,539 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (API-Job-Executor-96:ctx-428fa98b job-6262) Remove job-6262 from job
>> monitoring
>> 2015-04-06 13:18:42,800 DEBUG [c.c.a.m.AgentManagerImpl]
>> (AgentManager-Handler-11:null) SeqA 9-147481: Processing Seq 9-147481:
>> { Cmd , MgmtId: -1, via: 9, Ver: v1, Flags: 11,
>>
>> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":78,"_loadInfo":"{\n
>> \"connections\": []\n}","wait":0}}] }
>> 2015-04-06 13:18:42,808 DEBUG [c.c.a.m.AgentManagerImpl]
>> (AgentManager-Handler-11:null) SeqA 9-147481: Sending Seq 9-147481: {
>> Ans: , MgmtId: 345050016868, via: 9, Ver: v1, Flags: 100010,
>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}]
>> }
>> 2015-04-06 13:18:45,447 DEBUG [c.c.a.ApiServlet]
>> (catalina-exec-5:ctx-931b2c88) ===START=== 10.233.51.176 -- GET
>>
>> command=queryAsyncJobResult&jobId=61329013-6b40-43d2-ad1b-8fba0325bae1&response=json&sessionkey=gSf69NNJJZ0SzrIEz1fwOQJpRTo%3D&_=1428351525444
>> 2015-04-06 13:18:45,478 DEBUG [c.c.a.ApiServlet]
>> (catalina-exec-5:ctx-931b2c88 ctx-7b984e22) ===END=== 10.233.51.176
>> -- GET
>> command=queryAsyncJobResult&jobId=61329013-6b40-43d2-ad1b-8fba0325bae1&response=json&sessionkey=gSf69NNJJZ0SzrIEz1fwOQJpRTo%3D&_=1428351525444
>>
>> Removing the VPC also gives an internal server error and outputs the
>> following to the logs.
>>
>> 2015-04-06 07:03:10,148 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (catalina-exec-12:ctx-1b8cdf03 ctx-97a67e82) submit async job-6257,
>> details: AsyncJobVO {id:6257, userId: 3, accountId: 3, instanceType:
>> None, instanceId: null, cmd:
>> org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd, cmdInfo:
>>
>> {"response":"json","id":"02158f27-88f7-4ffe-83bf-60cfea333dee","sessionkey":"68X4liCjVtaZUrDpChfoXD1a2Mo\u003d","ctxDetails":"{\"com.cloud.network.vpc.Vpc\":\"02158f27-88f7-4ffe-83bf-60cfea333dee\"}","cmdEventType":"VPC.DELETE","ctxUserId":"3","httpmethod":"GET","_":"1428328990097","uuid":"02158f27-88f7-4ffe-83bf-60cfea333dee","ctxAccountId":"3","ctxStartEventId":"7627"},
>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> result: null, initMsid: 345050016868, completeMsid: null, lastUpdated:
>> null, lastPolled: null, created: null}
>> 2015-04-06 07:03:10,148 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Add job-6257 into job
>> monitoring
>> 2015-04-06 07:03:10,148 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Executing AsyncJobVO
>> {id:6257, userId: 3, accountId: 3, instanceType: None, instanceId:
>> null, cmd: org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd,
>> cmdInfo:
>> {"response":"json","id":"02158f27-88f7-4ffe-83bf-60cfea333dee","sessionkey":"68X4liCjVtaZUrDpChfoXD1a2Mo\u003d","ctxDetails":"{\"com.cloud.network.vpc.Vpc\":\"02158f27-88f7-4ffe-83bf-60cfea333dee\"}","cmdEventType":"VPC.DELETE","ctxUserId":"3","httpmethod":"GET","_":"1428328990097","uuid":"02158f27-88f7-4ffe-83bf-60cfea333dee","ctxAccountId":"3","ctxStartEventId":"7627"},
>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> result: null, initMsid: 345050016868, completeMsid: null, lastUpdated:
>> null, lastPolled: null, created: null}
>> 2015-04-06 07:03:10,149 DEBUG [c.c.a.ApiServlet]
>> (catalina-exec-12:ctx-1b8cdf03 ctx-97a67e82) ===END=== 10.233.51.176
>> -- GET
>> command=deleteVPC&response=json&sessionkey=68X4liCjVtaZUrDpChfoXD1a2Mo%3D&id=02158f27-88f7-4ffe-83bf-60cfea333dee&_=1428328990097
>> 2015-04-06 07:03:10,164 DEBUG [c.c.n.v.VpcManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Destroying
>> vpc [VPC [9-VPC]
>> 2015-04-06 07:03:10,169 DEBUG [c.c.n.v.VpcManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Shutting down
>> vpc [VPC [9-VPC]
>> 2015-04-06 07:03:10,172 DEBUG [c.c.n.v.VpcManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Vpc [VPC
>> [9-VPC] has been shutdown succesfully
>> 2015-04-06 07:03:10,172 DEBUG [c.c.n.v.VpcManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Cleaning up
>> resources for vpc id=9
>> 2015-04-06 07:03:10,172 DEBUG [c.c.n.v.VpcManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Cleaning up
>> existed site to site VPN connections
>> 2015-04-06 07:03:10,174 DEBUG [c.c.n.v.VpcManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Cleaning up
>> existed site to site VPN gateways
>> 2015-04-06 07:03:10,177 DEBUG [c.c.n.v.VpcManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Releasing ips
>> for vpc id=9 as a part of vpc cleanup
>> 2015-04-06 07:03:10,177 DEBUG [c.c.n.IpAddressManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Revoking all
>> Firewallrules as a part of public IP id=9 release...
>> 2015-04-06 07:03:10,189 DEBUG [c.c.n.f.FirewallManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Releasing 0
>> firewall rules for ip id=9
>> 2015-04-06 07:03:10,190 DEBUG [c.c.n.f.FirewallManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) There are no
>> firewall rules to apply
>> 2015-04-06 07:03:10,192 DEBUG [c.c.n.f.FirewallManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Successfully
>> released firewall rules for ip id=9 and # of rules now = 0
>> 2015-04-06 07:03:10,202 DEBUG [c.c.n.IpAddressManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Revoking all
>> PortForwarding/StaticNat rules as a part of public IP id=9 release...
>> 2015-04-06 07:03:10,204 DEBUG [c.c.n.r.RulesManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Releasing 0
>> port forwarding rules for ip id=9
>> 2015-04-06 07:03:10,206 DEBUG [c.c.n.r.RulesManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Releasing 0
>> static nat rules for ip id=9
>> 2015-04-06 07:03:10,208 DEBUG [c.c.n.r.RulesManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) There are no
>> port forwarding rules to apply for ip id=9
>> 2015-04-06 07:03:10,209 DEBUG [c.c.n.r.RulesManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) There are no
>> static nat rules to apply for ip id=9
>> 2015-04-06 07:03:10,211 DEBUG [c.c.n.r.RulesManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Source ip
>> id=Ip[66.194.167.148-1] is not one to one nat
>> 2015-04-06 07:03:10,214 DEBUG [c.c.n.r.RulesManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Successfully
>> released rules for ip id=9 and # of rules now = 0
>> 2015-04-06 07:03:10,214 DEBUG [c.c.n.IpAddressManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Revoking all
>> LoadBalancing rules as a part of public IP id=9 release...
>> 2015-04-06 07:03:10,216 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Found 0 lb
>> rules to cleanup
>> 2015-04-06 07:03:10,216 DEBUG [c.c.n.IpAddressManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Cleaning up
>> remote access vpns as a part of public IP id=9 release...
>> 2015-04-06 07:03:10,254 ERROR [c.c.a.ApiAsyncJobDispatcher]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Unexpected exception while
>> executing org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd
>> java.lang.NullPointerException
>> at
>> com.cloud.network.NetworkModelImpl.getPhysicalNetworkId(NetworkModelImpl.java:1374)
>> at
>> com.cloud.network.element.VirtualRouterElement.canHandle(VirtualRouterElement.java:152)
>> at
>> com.cloud.network.element.VirtualRouterElement.stopVpn(VirtualRouterElement.java:443)
>> at
>> com.cloud.network.vpn.RemoteAccessVpnManagerImpl.destroyRemoteAccessVpnForIp(RemoteAccessVpnManagerImpl.java:305)
>> at sun.reflect.GeneratedMethodAccessor670.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>> at
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>> at
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>> at
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>> at com.sun.proxy.$Proxy231.destroyRemoteAccessVpnForIp(Unknown Source)
>> at
>> com.cloud.network.IpAddressManagerImpl.cleanupIpResources(IpAddressManagerImpl.java:556)
>> at
>> com.cloud.network.IpAddressManagerImpl.disassociatePublicIpAddress(IpAddressManagerImpl.java:571)
>> at
>> com.cloud.network.vpc.VpcManagerImpl.cleanupVpcResources(VpcManagerImpl.java:1429)
>> at com.cloud.network.vpc.VpcManagerImpl.destroyVpc(VpcManagerImpl.java:938)
>> at com.cloud.network.vpc.VpcManagerImpl.deleteVpc(VpcManagerImpl.java:899)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>> at
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>> at
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>> at
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>> at com.sun.proxy.$Proxy161.deleteVpc(Unknown Source)
>> at
>> org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd.execute(DeleteVPCCmd.java:76)
>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
>> at
>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2015-04-06 07:03:10,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Complete async job-6257,
>> jobStatus: FAILED, resultCode: 530, result:
>>
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530}
>> 2015-04-06 07:03:10,266 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Done executing
>> org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd for job-6257
>> 2015-04-06 07:03:10,273 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Remove job-6257 from job
>> monitoring
>> 2015-04-06 07:03:11,046 DEBUG [c.c.a.m.AgentManagerImpl]
>> (AgentManager-Handler-1:null) Ping from 13
>> 2015-04-06 07:03:11,046 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) Process host VM state report from ping
>> process. host: 13
>> 2015-04-06 07:03:11,066 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) Process VM state report. host: 13,
>> number of records in report: 9
>> 2015-04-06 07:03:11,066 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 959,
>> power state: PowerOn
>> 2015-04-06 07:03:11,068 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM power state does not change, skip DB
>> writing. vm id: 959
>> 2015-04-06 07:03:11,068 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 968,
>> power state: PowerOn
>> 2015-04-06 07:03:11,071 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM power state does not change, skip DB
>> writing. vm id: 968
>> 2015-04-06 07:03:11,071 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 975,
>> power state: PowerOn
>> 2015-04-06 07:03:11,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM power state does not change, skip DB
>> writing. vm id: 975
>> 2015-04-06 07:03:11,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 974,
>> power state: PowerOn
>> 2015-04-06 07:03:11,076 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM power state does not change, skip DB
>> writing. vm id: 974
>> 2015-04-06 07:03:11,076 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 871,
>> power state: PowerOn
>> 2015-04-06 07:03:11,079 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM power state does not change, skip DB
>> writing. vm id: 871
>> 2015-04-06 07:03:11,079 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 78,
>> power state: PowerOn
>> 2015-04-06 07:03:11,081 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM power state does not change, skip DB
>> writing. vm id: 78
>> 2015-04-06 07:03:11,081 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 893,
>> power state: PowerOn
>> 2015-04-06 07:03:11,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM power state does not change, skip DB
>> writing. vm id: 893
>> 2015-04-06 07:03:11,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 914,
>> power state: PowerOn
>> 2015-04-06 07:03:11,086 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM power state does not change, skip DB
>> writing. vm id: 914
>> 2015-04-06 07:03:11,086 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 912,
>> power state: PowerOn
>> 2015-04-06 07:03:11,089 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) VM power state does not change, skip DB
>> writing. vm id: 912
>> 2015-04-06 07:03:11,095 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (AgentManager-Handler-1:null) Done with process of VM state report.
>> host: 13
>> 2015-04-06 07:03:13,165 DEBUG [c.c.a.ApiServlet]
>> (catalina-exec-11:ctx-ad36fed0) ===START=== 10.233.51.176 -- GET
>>
>> command=queryAsyncJobResult&jobId=85d5c719-6114-494d-bb52-26f43372976d&response=json&sessionkey=68X4liCjVtaZUrDpChfoXD1a2Mo%3D&_=1428328993153
>> 2015-04-06 07:03:13,191 DEBUG [c.c.a.ApiServlet]
>> (catalina-exec-11:ctx-ad36fed0 ctx-5af312b0) ===END=== 10.233.51.176
>> -- GET
>> command=queryAsyncJobResult&jobId=85d5c719-6114-494d-bb52-26f43372976d&response=json&sessionkey=68X4liCjVtaZUrDpChfoXD1a2Mo%3D&_=1428328993153
>>
>>
>> I'm at a loss as to what could be causing this, and I don't want to
>> hack the database to fix this without completely understanding how all
>> of these components interact with each other. Any help would be
>> appreciated.
>>
Re: Unable to remove or modify VPC
Posted by Sanjeev N <sa...@apache.org>.
Looks like a bug to me. Please open a bug in https://issues.apache.org/
On Tue, Apr 7, 2015 at 1:57 AM, Grayson Head <gr...@graysonhead.net>
wrote:
> One of my coworkers has created a VPC that we are unable to modify or
> remove.
>
> The coworker said he was testing the site-to-site VPN functionality,
> and ended up having to create a seperate VPC instance to get it
> working. Now we are unable to remove this VPC instance using the web
> interface or the API. I suspect based on the log outputs that the
> site-to-site VPN is what is causing this problem, but the VPC shows
> that it doesn't have any site-to-site connections configured.
>
> The VPC still has one external IP address allocated to it, which I
> think was the one he was using as the VPN gateway. I am also unable to
> remove this IP address, it fails with an "internal server" error. The
> following is the management-server.log when attempting to release this
> IP address.
>
> 2015-04-06 13:18:42,427 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-96:ctx-428fa98b job-6262) Add job-6262 into job
> monitoring
> 2015-04-06 13:18:42,427 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-96:ctx-428fa98b job-6262) Executing AsyncJobVO
> {id:6262, userId: 3, accountId: 3, instanceType: IpAddress,
> instanceId: 9, cmd:
> org.apache.cloudstack.api.command.user.address.DisassociateIPAddrCmd,
> cmdInfo:
> {"response":"json","id":"3d2205a4-41e9-4cf7-9261-d1ae63ce863f","sessionkey":"gSf69NNJJZ0SzrIEz1fwOQJpRTo\u003d","ctxDetails":"{\"com.cloud.network.IpAddress\":\"3d2205a4-41e9-4cf7-9261-d1ae63ce863f\"}","cmdEventType":"NET.IPRELEASE","ctxUserId":"3","httpmethod":"GET","_":"1428351522388","uuid":"3d2205a4-41e9-4cf7-9261-d1ae63ce863f","ctxAccountId":"3","ctxStartEventId":"7705"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 345050016868, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: null}
> 2015-04-06 13:18:42,427 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-14:ctx-91e999f7 ctx-49efc96a) ===END=== 10.233.51.176
> -- GET
> command=disassociateIpAddress&response=json&sessionkey=gSf69NNJJZ0SzrIEz1fwOQJpRTo%3D&id=3d2205a4-41e9-4cf7-9261-d1ae63ce863f&_=1428351522388
> 2015-04-06 13:18:42,451 DEBUG [c.c.n.IpAddressManagerImpl]
> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Revoking all
> Firewallrules as a part of public IP id=9 release...
> 2015-04-06 13:18:42,462 DEBUG [c.c.n.f.FirewallManagerImpl]
> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Releasing 0
> firewall rules for ip id=9
> 2015-04-06 13:18:42,464 DEBUG [c.c.n.f.FirewallManagerImpl]
> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) There are no
> firewall rules to apply
> 2015-04-06 13:18:42,466 DEBUG [c.c.n.f.FirewallManagerImpl]
> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Successfully
> released firewall rules for ip id=9 and # of rules now = 0
> 2015-04-06 13:18:42,475 DEBUG [c.c.n.IpAddressManagerImpl]
> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Revoking all
> PortForwarding/StaticNat rules as a part of public IP id=9 release...
> 2015-04-06 13:18:42,477 DEBUG [c.c.n.r.RulesManagerImpl]
> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Releasing 0
> port forwarding rules for ip id=9
> 2015-04-06 13:18:42,478 DEBUG [c.c.n.r.RulesManagerImpl]
> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Releasing 0
> static nat rules for ip id=9
> 2015-04-06 13:18:42,480 DEBUG [c.c.n.r.RulesManagerImpl]
> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) There are no
> port forwarding rules to apply for ip id=9
> 2015-04-06 13:18:42,482 DEBUG [c.c.n.r.RulesManagerImpl]
> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) There are no
> static nat rules to apply for ip id=9
> 2015-04-06 13:18:42,483 DEBUG [c.c.n.r.RulesManagerImpl]
> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Source ip
> id=Ip[66.194.167.148-1] is not one to one nat
> 2015-04-06 13:18:42,487 DEBUG [c.c.n.r.RulesManagerImpl]
> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Successfully
> released rules for ip id=9 and # of rules now = 0
> 2015-04-06 13:18:42,487 DEBUG [c.c.n.IpAddressManagerImpl]
> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Revoking all
> LoadBalancing rules as a part of public IP id=9 release...
> 2015-04-06 13:18:42,488 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Found 0 lb
> rules to cleanup
> 2015-04-06 13:18:42,488 DEBUG [c.c.n.IpAddressManagerImpl]
> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Cleaning up
> remote access vpns as a part of public IP id=9 release...
> 2015-04-06 13:18:42,523 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-96:ctx-428fa98b job-6262) Unexpected exception while
> executing
> org.apache.cloudstack.api.command.user.address.DisassociateIPAddrCmd
> java.lang.NullPointerException
> at
> com.cloud.network.NetworkModelImpl.getPhysicalNetworkId(NetworkModelImpl.java:1374)
> at
> com.cloud.network.element.VirtualRouterElement.canHandle(VirtualRouterElement.java:152)
> at
> com.cloud.network.element.VirtualRouterElement.stopVpn(VirtualRouterElement.java:443)
> at
> com.cloud.network.vpn.RemoteAccessVpnManagerImpl.destroyRemoteAccessVpnForIp(RemoteAccessVpnManagerImpl.java:305)
> at sun.reflect.GeneratedMethodAccessor670.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy231.destroyRemoteAccessVpnForIp(Unknown Source)
> at
> com.cloud.network.IpAddressManagerImpl.cleanupIpResources(IpAddressManagerImpl.java:556)
> at
> com.cloud.network.IpAddressManagerImpl.disassociatePublicIpAddress(IpAddressManagerImpl.java:571)
> at
> com.cloud.network.NetworkServiceImpl.releaseIpAddressInternal(NetworkServiceImpl.java:935)
> at
> com.cloud.network.NetworkServiceImpl.releaseIpAddress(NetworkServiceImpl.java:895)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy156.releaseIpAddress(Unknown Source)
> at
> org.apache.cloudstack.api.command.user.address.DisassociateIPAddrCmd.execute(DisassociateIPAddrCmd.java:80)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2015-04-06 13:18:42,525 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-96:ctx-428fa98b job-6262) Complete async job-6262,
> jobStatus: FAILED, resultCode: 530, result:
>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530}
> 2015-04-06 13:18:42,534 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-96:ctx-428fa98b job-6262) Done executing
> org.apache.cloudstack.api.command.user.address.DisassociateIPAddrCmd
> for job-6262
> 2015-04-06 13:18:42,539 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-96:ctx-428fa98b job-6262) Remove job-6262 from job
> monitoring
> 2015-04-06 13:18:42,800 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-11:null) SeqA 9-147481: Processing Seq 9-147481:
> { Cmd , MgmtId: -1, via: 9, Ver: v1, Flags: 11,
>
> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":78,"_loadInfo":"{\n
> \"connections\": []\n}","wait":0}}] }
> 2015-04-06 13:18:42,808 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-11:null) SeqA 9-147481: Sending Seq 9-147481: {
> Ans: , MgmtId: 345050016868, via: 9, Ver: v1, Flags: 100010,
> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}]
> }
> 2015-04-06 13:18:45,447 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-5:ctx-931b2c88) ===START=== 10.233.51.176 -- GET
>
> command=queryAsyncJobResult&jobId=61329013-6b40-43d2-ad1b-8fba0325bae1&response=json&sessionkey=gSf69NNJJZ0SzrIEz1fwOQJpRTo%3D&_=1428351525444
> 2015-04-06 13:18:45,478 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-5:ctx-931b2c88 ctx-7b984e22) ===END=== 10.233.51.176
> -- GET
> command=queryAsyncJobResult&jobId=61329013-6b40-43d2-ad1b-8fba0325bae1&response=json&sessionkey=gSf69NNJJZ0SzrIEz1fwOQJpRTo%3D&_=1428351525444
>
> Removing the VPC also gives an internal server error and outputs the
> following to the logs.
>
> 2015-04-06 07:03:10,148 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (catalina-exec-12:ctx-1b8cdf03 ctx-97a67e82) submit async job-6257,
> details: AsyncJobVO {id:6257, userId: 3, accountId: 3, instanceType:
> None, instanceId: null, cmd:
> org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd, cmdInfo:
>
> {"response":"json","id":"02158f27-88f7-4ffe-83bf-60cfea333dee","sessionkey":"68X4liCjVtaZUrDpChfoXD1a2Mo\u003d","ctxDetails":"{\"com.cloud.network.vpc.Vpc\":\"02158f27-88f7-4ffe-83bf-60cfea333dee\"}","cmdEventType":"VPC.DELETE","ctxUserId":"3","httpmethod":"GET","_":"1428328990097","uuid":"02158f27-88f7-4ffe-83bf-60cfea333dee","ctxAccountId":"3","ctxStartEventId":"7627"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 345050016868, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: null}
> 2015-04-06 07:03:10,148 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Add job-6257 into job
> monitoring
> 2015-04-06 07:03:10,148 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Executing AsyncJobVO
> {id:6257, userId: 3, accountId: 3, instanceType: None, instanceId:
> null, cmd: org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd,
> cmdInfo:
> {"response":"json","id":"02158f27-88f7-4ffe-83bf-60cfea333dee","sessionkey":"68X4liCjVtaZUrDpChfoXD1a2Mo\u003d","ctxDetails":"{\"com.cloud.network.vpc.Vpc\":\"02158f27-88f7-4ffe-83bf-60cfea333dee\"}","cmdEventType":"VPC.DELETE","ctxUserId":"3","httpmethod":"GET","_":"1428328990097","uuid":"02158f27-88f7-4ffe-83bf-60cfea333dee","ctxAccountId":"3","ctxStartEventId":"7627"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 345050016868, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: null}
> 2015-04-06 07:03:10,149 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-12:ctx-1b8cdf03 ctx-97a67e82) ===END=== 10.233.51.176
> -- GET
> command=deleteVPC&response=json&sessionkey=68X4liCjVtaZUrDpChfoXD1a2Mo%3D&id=02158f27-88f7-4ffe-83bf-60cfea333dee&_=1428328990097
> 2015-04-06 07:03:10,164 DEBUG [c.c.n.v.VpcManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Destroying
> vpc [VPC [9-VPC]
> 2015-04-06 07:03:10,169 DEBUG [c.c.n.v.VpcManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Shutting down
> vpc [VPC [9-VPC]
> 2015-04-06 07:03:10,172 DEBUG [c.c.n.v.VpcManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Vpc [VPC
> [9-VPC] has been shutdown succesfully
> 2015-04-06 07:03:10,172 DEBUG [c.c.n.v.VpcManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Cleaning up
> resources for vpc id=9
> 2015-04-06 07:03:10,172 DEBUG [c.c.n.v.VpcManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Cleaning up
> existed site to site VPN connections
> 2015-04-06 07:03:10,174 DEBUG [c.c.n.v.VpcManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Cleaning up
> existed site to site VPN gateways
> 2015-04-06 07:03:10,177 DEBUG [c.c.n.v.VpcManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Releasing ips
> for vpc id=9 as a part of vpc cleanup
> 2015-04-06 07:03:10,177 DEBUG [c.c.n.IpAddressManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Revoking all
> Firewallrules as a part of public IP id=9 release...
> 2015-04-06 07:03:10,189 DEBUG [c.c.n.f.FirewallManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Releasing 0
> firewall rules for ip id=9
> 2015-04-06 07:03:10,190 DEBUG [c.c.n.f.FirewallManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) There are no
> firewall rules to apply
> 2015-04-06 07:03:10,192 DEBUG [c.c.n.f.FirewallManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Successfully
> released firewall rules for ip id=9 and # of rules now = 0
> 2015-04-06 07:03:10,202 DEBUG [c.c.n.IpAddressManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Revoking all
> PortForwarding/StaticNat rules as a part of public IP id=9 release...
> 2015-04-06 07:03:10,204 DEBUG [c.c.n.r.RulesManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Releasing 0
> port forwarding rules for ip id=9
> 2015-04-06 07:03:10,206 DEBUG [c.c.n.r.RulesManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Releasing 0
> static nat rules for ip id=9
> 2015-04-06 07:03:10,208 DEBUG [c.c.n.r.RulesManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) There are no
> port forwarding rules to apply for ip id=9
> 2015-04-06 07:03:10,209 DEBUG [c.c.n.r.RulesManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) There are no
> static nat rules to apply for ip id=9
> 2015-04-06 07:03:10,211 DEBUG [c.c.n.r.RulesManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Source ip
> id=Ip[66.194.167.148-1] is not one to one nat
> 2015-04-06 07:03:10,214 DEBUG [c.c.n.r.RulesManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Successfully
> released rules for ip id=9 and # of rules now = 0
> 2015-04-06 07:03:10,214 DEBUG [c.c.n.IpAddressManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Revoking all
> LoadBalancing rules as a part of public IP id=9 release...
> 2015-04-06 07:03:10,216 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Found 0 lb
> rules to cleanup
> 2015-04-06 07:03:10,216 DEBUG [c.c.n.IpAddressManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Cleaning up
> remote access vpns as a part of public IP id=9 release...
> 2015-04-06 07:03:10,254 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Unexpected exception while
> executing org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd
> java.lang.NullPointerException
> at
> com.cloud.network.NetworkModelImpl.getPhysicalNetworkId(NetworkModelImpl.java:1374)
> at
> com.cloud.network.element.VirtualRouterElement.canHandle(VirtualRouterElement.java:152)
> at
> com.cloud.network.element.VirtualRouterElement.stopVpn(VirtualRouterElement.java:443)
> at
> com.cloud.network.vpn.RemoteAccessVpnManagerImpl.destroyRemoteAccessVpnForIp(RemoteAccessVpnManagerImpl.java:305)
> at sun.reflect.GeneratedMethodAccessor670.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy231.destroyRemoteAccessVpnForIp(Unknown Source)
> at
> com.cloud.network.IpAddressManagerImpl.cleanupIpResources(IpAddressManagerImpl.java:556)
> at
> com.cloud.network.IpAddressManagerImpl.disassociatePublicIpAddress(IpAddressManagerImpl.java:571)
> at
> com.cloud.network.vpc.VpcManagerImpl.cleanupVpcResources(VpcManagerImpl.java:1429)
> at com.cloud.network.vpc.VpcManagerImpl.destroyVpc(VpcManagerImpl.java:938)
> at com.cloud.network.vpc.VpcManagerImpl.deleteVpc(VpcManagerImpl.java:899)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy161.deleteVpc(Unknown Source)
> at
> org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd.execute(DeleteVPCCmd.java:76)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2015-04-06 07:03:10,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Complete async job-6257,
> jobStatus: FAILED, resultCode: 530, result:
>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530}
> 2015-04-06 07:03:10,266 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Done executing
> org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd for job-6257
> 2015-04-06 07:03:10,273 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Remove job-6257 from job
> monitoring
> 2015-04-06 07:03:11,046 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-1:null) Ping from 13
> 2015-04-06 07:03:11,046 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) Process host VM state report from ping
> process. host: 13
> 2015-04-06 07:03:11,066 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) Process VM state report. host: 13,
> number of records in report: 9
> 2015-04-06 07:03:11,066 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 959,
> power state: PowerOn
> 2015-04-06 07:03:11,068 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM power state does not change, skip DB
> writing. vm id: 959
> 2015-04-06 07:03:11,068 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 968,
> power state: PowerOn
> 2015-04-06 07:03:11,071 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM power state does not change, skip DB
> writing. vm id: 968
> 2015-04-06 07:03:11,071 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 975,
> power state: PowerOn
> 2015-04-06 07:03:11,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM power state does not change, skip DB
> writing. vm id: 975
> 2015-04-06 07:03:11,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 974,
> power state: PowerOn
> 2015-04-06 07:03:11,076 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM power state does not change, skip DB
> writing. vm id: 974
> 2015-04-06 07:03:11,076 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 871,
> power state: PowerOn
> 2015-04-06 07:03:11,079 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM power state does not change, skip DB
> writing. vm id: 871
> 2015-04-06 07:03:11,079 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 78,
> power state: PowerOn
> 2015-04-06 07:03:11,081 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM power state does not change, skip DB
> writing. vm id: 78
> 2015-04-06 07:03:11,081 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 893,
> power state: PowerOn
> 2015-04-06 07:03:11,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM power state does not change, skip DB
> writing. vm id: 893
> 2015-04-06 07:03:11,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 914,
> power state: PowerOn
> 2015-04-06 07:03:11,086 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM power state does not change, skip DB
> writing. vm id: 914
> 2015-04-06 07:03:11,086 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 912,
> power state: PowerOn
> 2015-04-06 07:03:11,089 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) VM power state does not change, skip DB
> writing. vm id: 912
> 2015-04-06 07:03:11,095 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentManager-Handler-1:null) Done with process of VM state report.
> host: 13
> 2015-04-06 07:03:13,165 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-11:ctx-ad36fed0) ===START=== 10.233.51.176 -- GET
>
> command=queryAsyncJobResult&jobId=85d5c719-6114-494d-bb52-26f43372976d&response=json&sessionkey=68X4liCjVtaZUrDpChfoXD1a2Mo%3D&_=1428328993153
> 2015-04-06 07:03:13,191 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-11:ctx-ad36fed0 ctx-5af312b0) ===END=== 10.233.51.176
> -- GET
> command=queryAsyncJobResult&jobId=85d5c719-6114-494d-bb52-26f43372976d&response=json&sessionkey=68X4liCjVtaZUrDpChfoXD1a2Mo%3D&_=1428328993153
>
>
> I'm at a loss as to what could be causing this, and I don't want to
> hack the database to fix this without completely understanding how all
> of these components interact with each other. Any help would be
> appreciated.
>