You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Bjoern Teipel <bj...@gmail.com> on 2014/01/27 08:14:27 UTC
Urgent : Network stuck in implementing state
Hi guys,
I had trouble while I was creating a new VLAN and now I can't delete it
anymore, because it's stuck in implementing state.
I guess that happened after the VR did not came up and I restarted the
management server so I could delete the VR and all addresses.
If I would spin up a guest, the VR and the guest would come up but I sill
want to get rid of it since I have an error in the network service offering.
After I cleaned all up though the GUI I can only see an external IP for the
old VR is still associated with the VLAN but can't be deleted (I don't get
it offered)
In the logs I found this and the NPE matches the table content :
2014-01-26 13:53:02,857 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
Network id=212 is destroyed successfully, cleaning up corresponding
resources
now.
2014-01-26 13:53:02,871 DEBUG [network.guru.DirectNetworkGuru]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
Releasing ip 10.16.48.1 of placeholder nic Nic[189-null-null-10.16.48.1]
2014-01-26 13:53:02,872 DEBUG [db.Transaction.Transaction]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
Rolling back the transaction: Time = 14 Name =
-AsyncJobManagerImpl$1.run:494-Exec
utors$RunnableAdapter.call:471-FutureTask.run:262-ThreadPoolExecutor.runWorker:1145-ThreadPoolExecutor$Worker.run:615-Thread.run:744;
called by
-Transaction.rollback:897-Transaction.removeUpTo:840-Transaction.cl
ose:664-TransactionContextBuilder.interceptException:63-ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept:133-NetworkManagerImpl.destroyNetwork:3144-ComponentInstantiationPostProcessor$Intercep
torDispatcher.intercept:125-NetworkServiceImpl.deleteNetwork:1767-ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept:125-DeleteNetworkCmd.execute:70-ApiDispatcher.dispatch:158-AsyncJobManagerImp
l$1.run:531
2014-01-26 13:53:02,877 ERROR [cloud.async.AsyncJobManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
Unexpected exception while executing
org.apache.cloudstack.api.command.user.ne
twork.DeleteNetworkCmd
java.lang.NullPointerException
at
com.cloud.network.guru.DirectNetworkGuru.trash(DirectNetworkGuru.java:311)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.network.NetworkManagerImpl.destroyNetwork(NetworkManagerImpl.java:3144)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.network.NetworkServiceImpl.deleteNetwork(NetworkServiceImpl.java:1767)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd.execute(DeleteNetworkCmd.java:70)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-26 13:53:02,880 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
Complete async job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ],
jobStatus:
2, resultCode: 530, result: Error Code: 530 Error text: null
2014-01-26 13:53:02,888 DEBUG [cloud.async.SyncQueueManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Sync
queue (6) is currently empty
2014-01-26 13:53:02,889 WARN [cloud.async.AsyncJobManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Unable
to unregister active job [ 1127 ] = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24
020 ] from JMX monitoring
mysql> select * from nics where id = 189\G
*************************** 1. row ***************************
id: 189
uuid: 623d0af3-7ca5-4700-89a2-26252abdc054
instance_id: NULL
mac_address: NULL
ip4_address: 10.16.48.1
netmask: NULL
gateway: NULL
ip_type: NULL
broadcast_uri: NULL
network_id: 212
mode: NULL
state: Reserved
strategy: PlaceHolder
reserver_name: NULL
reservation_id: NULL
device_id: 0
update_time: 2014-01-24 16:22:56
isolation_uri: NULL
ip6_address: NULL
default_nic: 0
vm_type: DomainRouter
created: 2014-01-25 00:22:56
removed: NULL
ip6_gateway: NULL
ip6_cidr: NULL
secondary_ip: 0
display_nic: 1
1 row in set (0.00 sec)
So what can I do to delete this stale interface ?
Thanks,
Bjoern
Re: Urgent : Network stuck in implementing state
Posted by Bjoern Teipel <bj...@gmail.com>.
Thanks, I got one step further and the NPE is not coming up but I guess I
need to change the overall network state :?
014-01-30 15:23:16,159 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-67:job-1218 = [ 7de934e0-2b42-476b-83da-cf03109331fa ])
Network is not implemented: Ntwk[212|Guest|16]
2014-01-30 15:23:16,160 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-67:job-1218 = [ 7de934e0-2b42-476b-83da-cf03109331fa ])
Network is not not in the correct state to be destroyed: Implementing
2014-01-30 15:23:16,164 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-67:job-1218 = [ 7de934e0-2b42-476b-83da-cf03109331fa ])
Complete async job-1218 = [ 7de934e0-2b42-476b-83da-cf03109331fa ],
jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed
to delete network
On Mon, Jan 27, 2014 at 10:57 PM, Sanjeev Neelarapu <
sanjeev.neelarapu@citrix.com> wrote:
> Hi Bjoern,
>
> Since the nic's instance_id is NULL you can set the removed field with
> some timestamp (e.g. now()). Also check the external IP address state in
> user_ip_address table. If this ip address state is in allocated set it to
> NULL and try delete network again.
>
> -Sanjeev
>
> -----Original Message-----
> From: Bjoern Teipel [mailto:bjoern.teipel@gmail.com]
> Sent: Monday, January 27, 2014 12:44 PM
> To: users; dev@cloudstack.apache.org
> Subject: Urgent : Network stuck in implementing state
>
> Hi guys,
>
> I had trouble while I was creating a new VLAN and now I can't delete it
> anymore, because it's stuck in implementing state.
> I guess that happened after the VR did not came up and I restarted the
> management server so I could delete the VR and all addresses.
>
> If I would spin up a guest, the VR and the guest would come up but I sill
> want to get rid of it since I have an error in the network service offering.
> After I cleaned all up though the GUI I can only see an external IP for
> the old VR is still associated with the VLAN but can't be deleted (I don't
> get it offered)
>
> In the logs I found this and the NPE matches the table content :
>
>
>
> 2014-01-26 13:53:02,857 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
> Network id=212 is destroyed successfully, cleaning up corresponding
> resources now.
> 2014-01-26 13:53:02,871 DEBUG [network.guru.DirectNetworkGuru]
> (Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
> Releasing ip 10.16.48.1 of placeholder nic Nic[189-null-null-10.16.48.1]
> 2014-01-26 13:53:02,872 DEBUG [db.Transaction.Transaction]
> (Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
> Rolling back the transaction: Time = 14 Name =
> -AsyncJobManagerImpl$1.run:494-Exec
>
> utors$RunnableAdapter.call:471-FutureTask.run:262-ThreadPoolExecutor.runWorker:1145-ThreadPoolExecutor$Worker.run:615-Thread.run:744;
> called by
> -Transaction.rollback:897-Transaction.removeUpTo:840-Transaction.cl
>
> ose:664-TransactionContextBuilder.interceptException:63-ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept:133-NetworkManagerImpl.destroyNetwork:3144-ComponentInstantiationPostProcessor$Intercep
>
> torDispatcher.intercept:125-NetworkServiceImpl.deleteNetwork:1767-ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept:125-DeleteNetworkCmd.execute:70-ApiDispatcher.dispatch:158-AsyncJobManagerImp
> l$1.run:531
> 2014-01-26 13:53:02,877 ERROR [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
> Unexpected exception while executing
> org.apache.cloudstack.api.command.user.ne
> twork.DeleteNetworkCmd
> java.lang.NullPointerException
> at
> com.cloud.network.guru.DirectNetworkGuru.trash(DirectNetworkGuru.java:311)
> at
>
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
>
> com.cloud.network.NetworkManagerImpl.destroyNetwork(NetworkManagerImpl.java:3144)
> at
>
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
>
> com.cloud.network.NetworkServiceImpl.deleteNetwork(NetworkServiceImpl.java:1767)
> at
>
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
>
> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd.execute(DeleteNetworkCmd.java:70)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2014-01-26 13:53:02,880 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
> Complete async job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ],
> jobStatus:
> 2, resultCode: 530, result: Error Code: 530 Error text: null
> 2014-01-26 13:53:02,888 DEBUG [cloud.async.SyncQueueManagerImpl]
> (Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Sync
> queue (6) is currently empty
> 2014-01-26 13:53:02,889 WARN [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
> Unable to unregister active job [ 1127 ] = [
> f90b84ab-0d4a-45ce-9cd1-d73b63e24
> 020 ] from JMX monitoring
>
>
> mysql> select * from nics where id = 189\G
> *************************** 1. row ***************************
> id: 189
> uuid: 623d0af3-7ca5-4700-89a2-26252abdc054
> instance_id: NULL
> mac_address: NULL
> ip4_address: 10.16.48.1
> netmask: NULL
> gateway: NULL
> ip_type: NULL
> broadcast_uri: NULL
> network_id: 212
> mode: NULL
> state: Reserved
> strategy: PlaceHolder
> reserver_name: NULL
> reservation_id: NULL
> device_id: 0
> update_time: 2014-01-24 16:22:56
> isolation_uri: NULL
> ip6_address: NULL
> default_nic: 0
> vm_type: DomainRouter
> created: 2014-01-25 00:22:56
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> 1 row in set (0.00 sec)
>
>
> So what can I do to delete this stale interface ?
>
> Thanks,
> Bjoern
>
Re: Urgent : Network stuck in implementing state
Posted by Bjoern Teipel <bj...@gmail.com>.
Thanks, I got one step further and the NPE is not coming up but I guess I
need to change the overall network state :?
014-01-30 15:23:16,159 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-67:job-1218 = [ 7de934e0-2b42-476b-83da-cf03109331fa ])
Network is not implemented: Ntwk[212|Guest|16]
2014-01-30 15:23:16,160 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-67:job-1218 = [ 7de934e0-2b42-476b-83da-cf03109331fa ])
Network is not not in the correct state to be destroyed: Implementing
2014-01-30 15:23:16,164 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-67:job-1218 = [ 7de934e0-2b42-476b-83da-cf03109331fa ])
Complete async job-1218 = [ 7de934e0-2b42-476b-83da-cf03109331fa ],
jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed
to delete network
On Mon, Jan 27, 2014 at 10:57 PM, Sanjeev Neelarapu <
sanjeev.neelarapu@citrix.com> wrote:
> Hi Bjoern,
>
> Since the nic's instance_id is NULL you can set the removed field with
> some timestamp (e.g. now()). Also check the external IP address state in
> user_ip_address table. If this ip address state is in allocated set it to
> NULL and try delete network again.
>
> -Sanjeev
>
> -----Original Message-----
> From: Bjoern Teipel [mailto:bjoern.teipel@gmail.com]
> Sent: Monday, January 27, 2014 12:44 PM
> To: users; dev@cloudstack.apache.org
> Subject: Urgent : Network stuck in implementing state
>
> Hi guys,
>
> I had trouble while I was creating a new VLAN and now I can't delete it
> anymore, because it's stuck in implementing state.
> I guess that happened after the VR did not came up and I restarted the
> management server so I could delete the VR and all addresses.
>
> If I would spin up a guest, the VR and the guest would come up but I sill
> want to get rid of it since I have an error in the network service offering.
> After I cleaned all up though the GUI I can only see an external IP for
> the old VR is still associated with the VLAN but can't be deleted (I don't
> get it offered)
>
> In the logs I found this and the NPE matches the table content :
>
>
>
> 2014-01-26 13:53:02,857 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
> Network id=212 is destroyed successfully, cleaning up corresponding
> resources now.
> 2014-01-26 13:53:02,871 DEBUG [network.guru.DirectNetworkGuru]
> (Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
> Releasing ip 10.16.48.1 of placeholder nic Nic[189-null-null-10.16.48.1]
> 2014-01-26 13:53:02,872 DEBUG [db.Transaction.Transaction]
> (Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
> Rolling back the transaction: Time = 14 Name =
> -AsyncJobManagerImpl$1.run:494-Exec
>
> utors$RunnableAdapter.call:471-FutureTask.run:262-ThreadPoolExecutor.runWorker:1145-ThreadPoolExecutor$Worker.run:615-Thread.run:744;
> called by
> -Transaction.rollback:897-Transaction.removeUpTo:840-Transaction.cl
>
> ose:664-TransactionContextBuilder.interceptException:63-ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept:133-NetworkManagerImpl.destroyNetwork:3144-ComponentInstantiationPostProcessor$Intercep
>
> torDispatcher.intercept:125-NetworkServiceImpl.deleteNetwork:1767-ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept:125-DeleteNetworkCmd.execute:70-ApiDispatcher.dispatch:158-AsyncJobManagerImp
> l$1.run:531
> 2014-01-26 13:53:02,877 ERROR [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
> Unexpected exception while executing
> org.apache.cloudstack.api.command.user.ne
> twork.DeleteNetworkCmd
> java.lang.NullPointerException
> at
> com.cloud.network.guru.DirectNetworkGuru.trash(DirectNetworkGuru.java:311)
> at
>
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
>
> com.cloud.network.NetworkManagerImpl.destroyNetwork(NetworkManagerImpl.java:3144)
> at
>
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
>
> com.cloud.network.NetworkServiceImpl.deleteNetwork(NetworkServiceImpl.java:1767)
> at
>
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
>
> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd.execute(DeleteNetworkCmd.java:70)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2014-01-26 13:53:02,880 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
> Complete async job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ],
> jobStatus:
> 2, resultCode: 530, result: Error Code: 530 Error text: null
> 2014-01-26 13:53:02,888 DEBUG [cloud.async.SyncQueueManagerImpl]
> (Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Sync
> queue (6) is currently empty
> 2014-01-26 13:53:02,889 WARN [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ])
> Unable to unregister active job [ 1127 ] = [
> f90b84ab-0d4a-45ce-9cd1-d73b63e24
> 020 ] from JMX monitoring
>
>
> mysql> select * from nics where id = 189\G
> *************************** 1. row ***************************
> id: 189
> uuid: 623d0af3-7ca5-4700-89a2-26252abdc054
> instance_id: NULL
> mac_address: NULL
> ip4_address: 10.16.48.1
> netmask: NULL
> gateway: NULL
> ip_type: NULL
> broadcast_uri: NULL
> network_id: 212
> mode: NULL
> state: Reserved
> strategy: PlaceHolder
> reserver_name: NULL
> reservation_id: NULL
> device_id: 0
> update_time: 2014-01-24 16:22:56
> isolation_uri: NULL
> ip6_address: NULL
> default_nic: 0
> vm_type: DomainRouter
> created: 2014-01-25 00:22:56
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> 1 row in set (0.00 sec)
>
>
> So what can I do to delete this stale interface ?
>
> Thanks,
> Bjoern
>
RE: Urgent : Network stuck in implementing state
Posted by Sanjeev Neelarapu <sa...@citrix.com>.
Hi Bjoern,
Since the nic's instance_id is NULL you can set the removed field with some timestamp (e.g. now()). Also check the external IP address state in user_ip_address table. If this ip address state is in allocated set it to NULL and try delete network again.
-Sanjeev
-----Original Message-----
From: Bjoern Teipel [mailto:bjoern.teipel@gmail.com]
Sent: Monday, January 27, 2014 12:44 PM
To: users; dev@cloudstack.apache.org
Subject: Urgent : Network stuck in implementing state
Hi guys,
I had trouble while I was creating a new VLAN and now I can't delete it anymore, because it's stuck in implementing state.
I guess that happened after the VR did not came up and I restarted the management server so I could delete the VR and all addresses.
If I would spin up a guest, the VR and the guest would come up but I sill want to get rid of it since I have an error in the network service offering.
After I cleaned all up though the GUI I can only see an external IP for the old VR is still associated with the VLAN but can't be deleted (I don't get it offered)
In the logs I found this and the NPE matches the table content :
2014-01-26 13:53:02,857 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Network id=212 is destroyed successfully, cleaning up corresponding resources now.
2014-01-26 13:53:02,871 DEBUG [network.guru.DirectNetworkGuru]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Releasing ip 10.16.48.1 of placeholder nic Nic[189-null-null-10.16.48.1]
2014-01-26 13:53:02,872 DEBUG [db.Transaction.Transaction]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Rolling back the transaction: Time = 14 Name = -AsyncJobManagerImpl$1.run:494-Exec
utors$RunnableAdapter.call:471-FutureTask.run:262-ThreadPoolExecutor.runWorker:1145-ThreadPoolExecutor$Worker.run:615-Thread.run:744;
called by
-Transaction.rollback:897-Transaction.removeUpTo:840-Transaction.cl
ose:664-TransactionContextBuilder.interceptException:63-ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept:133-NetworkManagerImpl.destroyNetwork:3144-ComponentInstantiationPostProcessor$Intercep
torDispatcher.intercept:125-NetworkServiceImpl.deleteNetwork:1767-ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept:125-DeleteNetworkCmd.execute:70-ApiDispatcher.dispatch:158-AsyncJobManagerImp
l$1.run:531
2014-01-26 13:53:02,877 ERROR [cloud.async.AsyncJobManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Unexpected exception while executing org.apache.cloudstack.api.command.user.ne
twork.DeleteNetworkCmd
java.lang.NullPointerException
at
com.cloud.network.guru.DirectNetworkGuru.trash(DirectNetworkGuru.java:311)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.network.NetworkManagerImpl.destroyNetwork(NetworkManagerImpl.java:3144)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.network.NetworkServiceImpl.deleteNetwork(NetworkServiceImpl.java:1767)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd.execute(DeleteNetworkCmd.java:70)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-26 13:53:02,880 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Complete async job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ],
jobStatus:
2, resultCode: 530, result: Error Code: 530 Error text: null
2014-01-26 13:53:02,888 DEBUG [cloud.async.SyncQueueManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Sync queue (6) is currently empty
2014-01-26 13:53:02,889 WARN [cloud.async.AsyncJobManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Unable to unregister active job [ 1127 ] = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24
020 ] from JMX monitoring
mysql> select * from nics where id = 189\G
*************************** 1. row ***************************
id: 189
uuid: 623d0af3-7ca5-4700-89a2-26252abdc054
instance_id: NULL
mac_address: NULL
ip4_address: 10.16.48.1
netmask: NULL
gateway: NULL
ip_type: NULL
broadcast_uri: NULL
network_id: 212
mode: NULL
state: Reserved
strategy: PlaceHolder
reserver_name: NULL
reservation_id: NULL
device_id: 0
update_time: 2014-01-24 16:22:56
isolation_uri: NULL
ip6_address: NULL
default_nic: 0
vm_type: DomainRouter
created: 2014-01-25 00:22:56
removed: NULL
ip6_gateway: NULL
ip6_cidr: NULL
secondary_ip: 0
display_nic: 1
1 row in set (0.00 sec)
So what can I do to delete this stale interface ?
Thanks,
Bjoern
RE: Urgent : Network stuck in implementing state
Posted by Sanjeev Neelarapu <sa...@citrix.com>.
Hi Bjoern,
Since the nic's instance_id is NULL you can set the removed field with some timestamp (e.g. now()). Also check the external IP address state in user_ip_address table. If this ip address state is in allocated set it to NULL and try delete network again.
-Sanjeev
-----Original Message-----
From: Bjoern Teipel [mailto:bjoern.teipel@gmail.com]
Sent: Monday, January 27, 2014 12:44 PM
To: users; dev@cloudstack.apache.org
Subject: Urgent : Network stuck in implementing state
Hi guys,
I had trouble while I was creating a new VLAN and now I can't delete it anymore, because it's stuck in implementing state.
I guess that happened after the VR did not came up and I restarted the management server so I could delete the VR and all addresses.
If I would spin up a guest, the VR and the guest would come up but I sill want to get rid of it since I have an error in the network service offering.
After I cleaned all up though the GUI I can only see an external IP for the old VR is still associated with the VLAN but can't be deleted (I don't get it offered)
In the logs I found this and the NPE matches the table content :
2014-01-26 13:53:02,857 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Network id=212 is destroyed successfully, cleaning up corresponding resources now.
2014-01-26 13:53:02,871 DEBUG [network.guru.DirectNetworkGuru]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Releasing ip 10.16.48.1 of placeholder nic Nic[189-null-null-10.16.48.1]
2014-01-26 13:53:02,872 DEBUG [db.Transaction.Transaction]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Rolling back the transaction: Time = 14 Name = -AsyncJobManagerImpl$1.run:494-Exec
utors$RunnableAdapter.call:471-FutureTask.run:262-ThreadPoolExecutor.runWorker:1145-ThreadPoolExecutor$Worker.run:615-Thread.run:744;
called by
-Transaction.rollback:897-Transaction.removeUpTo:840-Transaction.cl
ose:664-TransactionContextBuilder.interceptException:63-ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept:133-NetworkManagerImpl.destroyNetwork:3144-ComponentInstantiationPostProcessor$Intercep
torDispatcher.intercept:125-NetworkServiceImpl.deleteNetwork:1767-ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept:125-DeleteNetworkCmd.execute:70-ApiDispatcher.dispatch:158-AsyncJobManagerImp
l$1.run:531
2014-01-26 13:53:02,877 ERROR [cloud.async.AsyncJobManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Unexpected exception while executing org.apache.cloudstack.api.command.user.ne
twork.DeleteNetworkCmd
java.lang.NullPointerException
at
com.cloud.network.guru.DirectNetworkGuru.trash(DirectNetworkGuru.java:311)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.network.NetworkManagerImpl.destroyNetwork(NetworkManagerImpl.java:3144)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.network.NetworkServiceImpl.deleteNetwork(NetworkServiceImpl.java:1767)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd.execute(DeleteNetworkCmd.java:70)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-01-26 13:53:02,880 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Complete async job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ],
jobStatus:
2, resultCode: 530, result: Error Code: 530 Error text: null
2014-01-26 13:53:02,888 DEBUG [cloud.async.SyncQueueManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Sync queue (6) is currently empty
2014-01-26 13:53:02,889 WARN [cloud.async.AsyncJobManagerImpl]
(Job-Executor-2:job-1127 = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24020 ]) Unable to unregister active job [ 1127 ] = [ f90b84ab-0d4a-45ce-9cd1-d73b63e24
020 ] from JMX monitoring
mysql> select * from nics where id = 189\G
*************************** 1. row ***************************
id: 189
uuid: 623d0af3-7ca5-4700-89a2-26252abdc054
instance_id: NULL
mac_address: NULL
ip4_address: 10.16.48.1
netmask: NULL
gateway: NULL
ip_type: NULL
broadcast_uri: NULL
network_id: 212
mode: NULL
state: Reserved
strategy: PlaceHolder
reserver_name: NULL
reservation_id: NULL
device_id: 0
update_time: 2014-01-24 16:22:56
isolation_uri: NULL
ip6_address: NULL
default_nic: 0
vm_type: DomainRouter
created: 2014-01-25 00:22:56
removed: NULL
ip6_gateway: NULL
ip6_cidr: NULL
secondary_ip: 0
display_nic: 1
1 row in set (0.00 sec)
So what can I do to delete this stale interface ?
Thanks,
Bjoern