You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Daan Hoogland <DH...@schubergphilis.com> on 2013/08/01 15:27:42 UTC
RE: vms hang in expunged state
Valery,
Do you still have assigned ip addresses to the vms?
Regards,
-----Original Message-----
From: Valery Ciareszka [mailto:valery.tereshko@gmail.com]
Sent: donderdag 25 juli 2013 14:54
To: users
Subject: vms hang in expunged state
Hi all.
I have faced the following problem: virtual machines remain in expunging state forever, and I see java.lang.NullPointerException in management server logs.
I use CS 4.1, CentOS 6.4, KVM.
What is the proper way to handle this issue ?
Management server log:
2013-07-25 12:30:17,674 INFO [cloud.vm.UserVmManagerImpl]
(UserVm-Scavenger-1:null) Found 2 vms to expunge.
2013-07-25 12:30:17,674 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(UserVm-Scavenger-1:null) Stopped called on VM[User|te st090703-1] but the state is Expunging
2013-07-25 12:30:17,694 DEBUG [cloud.capacity.CapacityManagerImpl]
(UserVm-Scavenger-1:null) VM state transitted from :Ex punging to Expunging with event: ExpungeOperationvm's original host id: 19 new host id: null host id before state transit
ion: null
2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(UserVm-Scavenger-1:null) Destroying vm VM[User|test09 0703-1]
2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(UserVm-Scavenger-1:null) Cleaning up NICS
2013-07-25 12:30:17,694 DEBUG [cloud.network.NetworkManagerImpl]
(UserVm-Scavenger-1:null) Cleaning network for vm: 6164
2013-07-25 12:30:17,696 DEBUG [cloud.storage.StorageManagerImpl]
(UserVm-Scavenger-1:null) Cleaning storage for vm: 6164
2013-07-25 12:30:17,697 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(UserVm-Scavenger-1:null) Expunged VM[User|test090703- 1]
2013-07-25 12:30:17,697 DEBUG [cloud.vm.UserVmManagerImpl]
(UserVm-Scavenger-1:null) Starting cleaning up vm VM[User|test 090703-1] resources...
2013-07-25 12:30:17,705 DEBUG [network.firewall.FirewallManagerImpl]
(UserVm-Scavenger-1:null) No firewall rules are foun d for vm id=6164
2013-07-25 12:30:17,706 DEBUG [cloud.vm.UserVmManagerImpl]
(UserVm-Scavenger-1:null) Firewall rules are removed successfu lly as a part of vm id=6164 expunge
2013-07-25 12:30:17,709 DEBUG [network.rules.RulesManagerImpl]
(UserVm-Scavenger-1:null) No port forwarding rules are fou nd for vm id=6164
2013-07-25 12:30:17,709 DEBUG [cloud.vm.UserVmManagerImpl]
(UserVm-Scavenger-1:null) Port forwarding rules are removed su ccessfully as a part of vm id=6164 expunge
2013-07-25 12:30:17,711 DEBUG [cloud.vm.UserVmManagerImpl]
(UserVm-Scavenger-1:null) Removed vm id=6164 from all load bal ancers as a part of expunge process
2013-07-25 12:30:17,714 DEBUG [network.rules.RulesManagerImpl]
(UserVm-Scavenger-1:null) Revoking all Firewallrules as a part of disabling static nat for public IP id=1305
2013-07-25 12:30:17,716 DEBUG [network.firewall.FirewallManagerImpl]
(UserVm-Scavenger-1:null) Releasing 3 firewall rules for ip id=1305
2013-07-25 12:30:17,718 WARN [cloud.vm.UserVmManagerImpl]
(UserVm-Scavenger-1:null) Unable to expunge VM[User|test090703-1] java.lang.NullPointerException
at com.cloud.event.EventUtils.getDomainId(EventUtils.java:116)
at com.cloud.event.EventUtils.saveStartedEvent(EventUtils.java:79)
at
com.cloud.event.ActionEventInterceptor.interceptStart(ActionEventInterceptor.java:47)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:119)
at
com.cloud.network.firewall.FirewallManagerImpl.revokeFirewallRulesForIp(FirewallManagerImpl.java:734)
at
com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1194)
at
com.cloud.vm.UserVmManagerImpl.cleanupVmResources(UserVmManagerImpl.java:1856)
at
com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:1787)
at
com.cloud.vm.UserVmManagerImpl$ExpungeTask.run(UserVmManagerImpl.java:2416)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
--
Regards,
Valery
http://protocol.by/slayer
Re: vms hang in expunged state
Posted by Daan Hoogland <da...@gmail.com>.
Thanks Valery,
I know some colleagues had the issue as well. The work around is to
delete ip-addresses first and we are using that now. I will pick it up
if it reaches the top off my prio list.
regards,
Daan
On Thu, Aug 15, 2013 at 12:14 PM, Valery Ciareszka
<va...@gmail.com> wrote:
> https://issues.apache.org/jira/browse/CLOUDSTACK-4349
>
>
> On Wed, Aug 14, 2013 at 6:35 PM, Daan Hoogland <da...@gmail.com>wrote:
>
>> Guess so,
>>
>> please enter it at issues.apache...
>>
>> thanks,
>> Daan
>>
>> On Wed, Aug 14, 2013 at 5:27 PM, Valery Ciareszka
>> <va...@gmail.com> wrote:
>> > Daan,
>> >
>> > Yes, static nat was enabled for theese vms. I disabled static nat and vms
>> > were successfully expunged.
>> > Looks like a bug.
>> >
>> > On Thu, Aug 1, 2013 at 4:27 PM, Daan Hoogland
>> > <DH...@schubergphilis.com>wrote:
>> >
>> >> Valery,
>> >>
>> >> Do you still have assigned ip addresses to the vms?
>> >>
>> >> Regards,
>> >>
>> >> -----Original Message-----
>> >> From: Valery Ciareszka [mailto:valery.tereshko@gmail.com]
>> >> Sent: donderdag 25 juli 2013 14:54
>> >> To: users
>> >> Subject: vms hang in expunged state
>> >>
>> >> Hi all.
>> >>
>> >> I have faced the following problem: virtual machines remain in expunging
>> >> state forever, and I see java.lang.NullPointerException in management
>> >> server logs.
>> >> I use CS 4.1, CentOS 6.4, KVM.
>> >>
>> >> What is the proper way to handle this issue ?
>> >>
>> >> Management server log:
>> >>
>> >> 2013-07-25 12:30:17,674 INFO [cloud.vm.UserVmManagerImpl]
>> >> (UserVm-Scavenger-1:null) Found 2 vms to expunge.
>> >> 2013-07-25 12:30:17,674 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> >> (UserVm-Scavenger-1:null) Stopped called on VM[User|te st090703-1] but
>> the
>> >> state is Expunging
>> >> 2013-07-25 12:30:17,694 DEBUG [cloud.capacity.CapacityManagerImpl]
>> >> (UserVm-Scavenger-1:null) VM state transitted from :Ex punging to
>> >> Expunging with event: ExpungeOperationvm's original host id: 19 new host
>> >> id: null host id before state transit
>> >> ion: null
>> >> 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> >> (UserVm-Scavenger-1:null) Destroying vm VM[User|test09 0703-1]
>> >> 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> >> (UserVm-Scavenger-1:null) Cleaning up NICS
>> >> 2013-07-25 12:30:17,694 DEBUG [cloud.network.NetworkManagerImpl]
>> >> (UserVm-Scavenger-1:null) Cleaning network for vm: 6164
>> >> 2013-07-25 12:30:17,696 DEBUG [cloud.storage.StorageManagerImpl]
>> >> (UserVm-Scavenger-1:null) Cleaning storage for vm: 6164
>> >> 2013-07-25 12:30:17,697 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> >> (UserVm-Scavenger-1:null) Expunged VM[User|test090703- 1]
>> >> 2013-07-25 12:30:17,697 DEBUG [cloud.vm.UserVmManagerImpl]
>> >> (UserVm-Scavenger-1:null) Starting cleaning up vm VM[User|test 090703-1]
>> >> resources...
>> >> 2013-07-25 12:30:17,705 DEBUG [network.firewall.FirewallManagerImpl]
>> >> (UserVm-Scavenger-1:null) No firewall rules are foun d for vm id=6164
>> >> 2013-07-25 12:30:17,706 DEBUG [cloud.vm.UserVmManagerImpl]
>> >> (UserVm-Scavenger-1:null) Firewall rules are removed successfu lly as a
>> >> part of vm id=6164 expunge
>> >> 2013-07-25 12:30:17,709 DEBUG [network.rules.RulesManagerImpl]
>> >> (UserVm-Scavenger-1:null) No port forwarding rules are fou nd for vm
>> >> id=6164
>> >> 2013-07-25 12:30:17,709 DEBUG [cloud.vm.UserVmManagerImpl]
>> >> (UserVm-Scavenger-1:null) Port forwarding rules are removed su
>> ccessfully
>> >> as a part of vm id=6164 expunge
>> >> 2013-07-25 12:30:17,711 DEBUG [cloud.vm.UserVmManagerImpl]
>> >> (UserVm-Scavenger-1:null) Removed vm id=6164 from all load bal ancers
>> as a
>> >> part of expunge process
>> >> 2013-07-25 12:30:17,714 DEBUG [network.rules.RulesManagerImpl]
>> >> (UserVm-Scavenger-1:null) Revoking all Firewallrules as a part of
>> >> disabling static nat for public IP id=1305
>> >> 2013-07-25 12:30:17,716 DEBUG [network.firewall.FirewallManagerImpl]
>> >> (UserVm-Scavenger-1:null) Releasing 3 firewall rules for ip id=1305
>> >> 2013-07-25 12:30:17,718 WARN [cloud.vm.UserVmManagerImpl]
>> >> (UserVm-Scavenger-1:null) Unable to expunge VM[User|test090703-1]
>> >> java.lang.NullPointerException
>> >> at com.cloud.event.EventUtils.getDomainId(EventUtils.java:116)
>> >> at
>> com.cloud.event.EventUtils.saveStartedEvent(EventUtils.java:79)
>> >> at
>> >>
>> >>
>> com.cloud.event.ActionEventInterceptor.interceptStart(ActionEventInterceptor.java:47)
>> >> at
>> >>
>> >>
>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:119)
>> >> at
>> >>
>> >>
>> com.cloud.network.firewall.FirewallManagerImpl.revokeFirewallRulesForIp(FirewallManagerImpl.java:734)
>> >> at
>> >>
>> >>
>> com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1194)
>> >> at
>> >>
>> >>
>> com.cloud.vm.UserVmManagerImpl.cleanupVmResources(UserVmManagerImpl.java:1856)
>> >> at
>> >> com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:1787)
>> >> at
>> >>
>> com.cloud.vm.UserVmManagerImpl$ExpungeTask.run(UserVmManagerImpl.java:2416)
>> >> at
>> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> >> at
>> >>
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
>> >> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >> at java.lang.Thread.run(Thread.java:679)
>> >>
>> >>
>> >> --
>> >> Regards,
>> >> Valery
>> >>
>> >> http://protocol.by/slayer
>> >>
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Valery
>> >
>> > http://protocol.by/slayer
>>
>
>
>
> --
> Regards,
> Valery
>
> http://protocol.by/slayer
Re: vms hang in expunged state
Posted by Valery Ciareszka <va...@gmail.com>.
https://issues.apache.org/jira/browse/CLOUDSTACK-4349
On Wed, Aug 14, 2013 at 6:35 PM, Daan Hoogland <da...@gmail.com>wrote:
> Guess so,
>
> please enter it at issues.apache...
>
> thanks,
> Daan
>
> On Wed, Aug 14, 2013 at 5:27 PM, Valery Ciareszka
> <va...@gmail.com> wrote:
> > Daan,
> >
> > Yes, static nat was enabled for theese vms. I disabled static nat and vms
> > were successfully expunged.
> > Looks like a bug.
> >
> > On Thu, Aug 1, 2013 at 4:27 PM, Daan Hoogland
> > <DH...@schubergphilis.com>wrote:
> >
> >> Valery,
> >>
> >> Do you still have assigned ip addresses to the vms?
> >>
> >> Regards,
> >>
> >> -----Original Message-----
> >> From: Valery Ciareszka [mailto:valery.tereshko@gmail.com]
> >> Sent: donderdag 25 juli 2013 14:54
> >> To: users
> >> Subject: vms hang in expunged state
> >>
> >> Hi all.
> >>
> >> I have faced the following problem: virtual machines remain in expunging
> >> state forever, and I see java.lang.NullPointerException in management
> >> server logs.
> >> I use CS 4.1, CentOS 6.4, KVM.
> >>
> >> What is the proper way to handle this issue ?
> >>
> >> Management server log:
> >>
> >> 2013-07-25 12:30:17,674 INFO [cloud.vm.UserVmManagerImpl]
> >> (UserVm-Scavenger-1:null) Found 2 vms to expunge.
> >> 2013-07-25 12:30:17,674 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> >> (UserVm-Scavenger-1:null) Stopped called on VM[User|te st090703-1] but
> the
> >> state is Expunging
> >> 2013-07-25 12:30:17,694 DEBUG [cloud.capacity.CapacityManagerImpl]
> >> (UserVm-Scavenger-1:null) VM state transitted from :Ex punging to
> >> Expunging with event: ExpungeOperationvm's original host id: 19 new host
> >> id: null host id before state transit
> >> ion: null
> >> 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> >> (UserVm-Scavenger-1:null) Destroying vm VM[User|test09 0703-1]
> >> 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> >> (UserVm-Scavenger-1:null) Cleaning up NICS
> >> 2013-07-25 12:30:17,694 DEBUG [cloud.network.NetworkManagerImpl]
> >> (UserVm-Scavenger-1:null) Cleaning network for vm: 6164
> >> 2013-07-25 12:30:17,696 DEBUG [cloud.storage.StorageManagerImpl]
> >> (UserVm-Scavenger-1:null) Cleaning storage for vm: 6164
> >> 2013-07-25 12:30:17,697 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> >> (UserVm-Scavenger-1:null) Expunged VM[User|test090703- 1]
> >> 2013-07-25 12:30:17,697 DEBUG [cloud.vm.UserVmManagerImpl]
> >> (UserVm-Scavenger-1:null) Starting cleaning up vm VM[User|test 090703-1]
> >> resources...
> >> 2013-07-25 12:30:17,705 DEBUG [network.firewall.FirewallManagerImpl]
> >> (UserVm-Scavenger-1:null) No firewall rules are foun d for vm id=6164
> >> 2013-07-25 12:30:17,706 DEBUG [cloud.vm.UserVmManagerImpl]
> >> (UserVm-Scavenger-1:null) Firewall rules are removed successfu lly as a
> >> part of vm id=6164 expunge
> >> 2013-07-25 12:30:17,709 DEBUG [network.rules.RulesManagerImpl]
> >> (UserVm-Scavenger-1:null) No port forwarding rules are fou nd for vm
> >> id=6164
> >> 2013-07-25 12:30:17,709 DEBUG [cloud.vm.UserVmManagerImpl]
> >> (UserVm-Scavenger-1:null) Port forwarding rules are removed su
> ccessfully
> >> as a part of vm id=6164 expunge
> >> 2013-07-25 12:30:17,711 DEBUG [cloud.vm.UserVmManagerImpl]
> >> (UserVm-Scavenger-1:null) Removed vm id=6164 from all load bal ancers
> as a
> >> part of expunge process
> >> 2013-07-25 12:30:17,714 DEBUG [network.rules.RulesManagerImpl]
> >> (UserVm-Scavenger-1:null) Revoking all Firewallrules as a part of
> >> disabling static nat for public IP id=1305
> >> 2013-07-25 12:30:17,716 DEBUG [network.firewall.FirewallManagerImpl]
> >> (UserVm-Scavenger-1:null) Releasing 3 firewall rules for ip id=1305
> >> 2013-07-25 12:30:17,718 WARN [cloud.vm.UserVmManagerImpl]
> >> (UserVm-Scavenger-1:null) Unable to expunge VM[User|test090703-1]
> >> java.lang.NullPointerException
> >> at com.cloud.event.EventUtils.getDomainId(EventUtils.java:116)
> >> at
> com.cloud.event.EventUtils.saveStartedEvent(EventUtils.java:79)
> >> at
> >>
> >>
> com.cloud.event.ActionEventInterceptor.interceptStart(ActionEventInterceptor.java:47)
> >> at
> >>
> >>
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:119)
> >> at
> >>
> >>
> com.cloud.network.firewall.FirewallManagerImpl.revokeFirewallRulesForIp(FirewallManagerImpl.java:734)
> >> at
> >>
> >>
> com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1194)
> >> at
> >>
> >>
> com.cloud.vm.UserVmManagerImpl.cleanupVmResources(UserVmManagerImpl.java:1856)
> >> at
> >> com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:1787)
> >> at
> >>
> com.cloud.vm.UserVmManagerImpl$ExpungeTask.run(UserVmManagerImpl.java:2416)
> >> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >> at
> >>
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> >> at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> >> at
> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> >> at
> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> >> at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> >> at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >> at java.lang.Thread.run(Thread.java:679)
> >>
> >>
> >> --
> >> Regards,
> >> Valery
> >>
> >> http://protocol.by/slayer
> >>
> >
> >
> >
> > --
> > Regards,
> > Valery
> >
> > http://protocol.by/slayer
>
--
Regards,
Valery
http://protocol.by/slayer
Re: vms hang in expunged state
Posted by Daan Hoogland <da...@gmail.com>.
Guess so,
please enter it at issues.apache...
thanks,
Daan
On Wed, Aug 14, 2013 at 5:27 PM, Valery Ciareszka
<va...@gmail.com> wrote:
> Daan,
>
> Yes, static nat was enabled for theese vms. I disabled static nat and vms
> were successfully expunged.
> Looks like a bug.
>
> On Thu, Aug 1, 2013 at 4:27 PM, Daan Hoogland
> <DH...@schubergphilis.com>wrote:
>
>> Valery,
>>
>> Do you still have assigned ip addresses to the vms?
>>
>> Regards,
>>
>> -----Original Message-----
>> From: Valery Ciareszka [mailto:valery.tereshko@gmail.com]
>> Sent: donderdag 25 juli 2013 14:54
>> To: users
>> Subject: vms hang in expunged state
>>
>> Hi all.
>>
>> I have faced the following problem: virtual machines remain in expunging
>> state forever, and I see java.lang.NullPointerException in management
>> server logs.
>> I use CS 4.1, CentOS 6.4, KVM.
>>
>> What is the proper way to handle this issue ?
>>
>> Management server log:
>>
>> 2013-07-25 12:30:17,674 INFO [cloud.vm.UserVmManagerImpl]
>> (UserVm-Scavenger-1:null) Found 2 vms to expunge.
>> 2013-07-25 12:30:17,674 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> (UserVm-Scavenger-1:null) Stopped called on VM[User|te st090703-1] but the
>> state is Expunging
>> 2013-07-25 12:30:17,694 DEBUG [cloud.capacity.CapacityManagerImpl]
>> (UserVm-Scavenger-1:null) VM state transitted from :Ex punging to
>> Expunging with event: ExpungeOperationvm's original host id: 19 new host
>> id: null host id before state transit
>> ion: null
>> 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> (UserVm-Scavenger-1:null) Destroying vm VM[User|test09 0703-1]
>> 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> (UserVm-Scavenger-1:null) Cleaning up NICS
>> 2013-07-25 12:30:17,694 DEBUG [cloud.network.NetworkManagerImpl]
>> (UserVm-Scavenger-1:null) Cleaning network for vm: 6164
>> 2013-07-25 12:30:17,696 DEBUG [cloud.storage.StorageManagerImpl]
>> (UserVm-Scavenger-1:null) Cleaning storage for vm: 6164
>> 2013-07-25 12:30:17,697 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> (UserVm-Scavenger-1:null) Expunged VM[User|test090703- 1]
>> 2013-07-25 12:30:17,697 DEBUG [cloud.vm.UserVmManagerImpl]
>> (UserVm-Scavenger-1:null) Starting cleaning up vm VM[User|test 090703-1]
>> resources...
>> 2013-07-25 12:30:17,705 DEBUG [network.firewall.FirewallManagerImpl]
>> (UserVm-Scavenger-1:null) No firewall rules are foun d for vm id=6164
>> 2013-07-25 12:30:17,706 DEBUG [cloud.vm.UserVmManagerImpl]
>> (UserVm-Scavenger-1:null) Firewall rules are removed successfu lly as a
>> part of vm id=6164 expunge
>> 2013-07-25 12:30:17,709 DEBUG [network.rules.RulesManagerImpl]
>> (UserVm-Scavenger-1:null) No port forwarding rules are fou nd for vm
>> id=6164
>> 2013-07-25 12:30:17,709 DEBUG [cloud.vm.UserVmManagerImpl]
>> (UserVm-Scavenger-1:null) Port forwarding rules are removed su ccessfully
>> as a part of vm id=6164 expunge
>> 2013-07-25 12:30:17,711 DEBUG [cloud.vm.UserVmManagerImpl]
>> (UserVm-Scavenger-1:null) Removed vm id=6164 from all load bal ancers as a
>> part of expunge process
>> 2013-07-25 12:30:17,714 DEBUG [network.rules.RulesManagerImpl]
>> (UserVm-Scavenger-1:null) Revoking all Firewallrules as a part of
>> disabling static nat for public IP id=1305
>> 2013-07-25 12:30:17,716 DEBUG [network.firewall.FirewallManagerImpl]
>> (UserVm-Scavenger-1:null) Releasing 3 firewall rules for ip id=1305
>> 2013-07-25 12:30:17,718 WARN [cloud.vm.UserVmManagerImpl]
>> (UserVm-Scavenger-1:null) Unable to expunge VM[User|test090703-1]
>> java.lang.NullPointerException
>> at com.cloud.event.EventUtils.getDomainId(EventUtils.java:116)
>> at com.cloud.event.EventUtils.saveStartedEvent(EventUtils.java:79)
>> at
>>
>> com.cloud.event.ActionEventInterceptor.interceptStart(ActionEventInterceptor.java:47)
>> at
>>
>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:119)
>> at
>>
>> com.cloud.network.firewall.FirewallManagerImpl.revokeFirewallRulesForIp(FirewallManagerImpl.java:734)
>> at
>>
>> com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1194)
>> at
>>
>> com.cloud.vm.UserVmManagerImpl.cleanupVmResources(UserVmManagerImpl.java:1856)
>> at
>> com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:1787)
>> at
>> com.cloud.vm.UserVmManagerImpl$ExpungeTask.run(UserVmManagerImpl.java:2416)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:679)
>>
>>
>> --
>> Regards,
>> Valery
>>
>> http://protocol.by/slayer
>>
>
>
>
> --
> Regards,
> Valery
>
> http://protocol.by/slayer
Re: vms hang in expunged state
Posted by Valery Ciareszka <va...@gmail.com>.
Daan,
Yes, static nat was enabled for theese vms. I disabled static nat and vms
were successfully expunged.
Looks like a bug.
On Thu, Aug 1, 2013 at 4:27 PM, Daan Hoogland
<DH...@schubergphilis.com>wrote:
> Valery,
>
> Do you still have assigned ip addresses to the vms?
>
> Regards,
>
> -----Original Message-----
> From: Valery Ciareszka [mailto:valery.tereshko@gmail.com]
> Sent: donderdag 25 juli 2013 14:54
> To: users
> Subject: vms hang in expunged state
>
> Hi all.
>
> I have faced the following problem: virtual machines remain in expunging
> state forever, and I see java.lang.NullPointerException in management
> server logs.
> I use CS 4.1, CentOS 6.4, KVM.
>
> What is the proper way to handle this issue ?
>
> Management server log:
>
> 2013-07-25 12:30:17,674 INFO [cloud.vm.UserVmManagerImpl]
> (UserVm-Scavenger-1:null) Found 2 vms to expunge.
> 2013-07-25 12:30:17,674 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (UserVm-Scavenger-1:null) Stopped called on VM[User|te st090703-1] but the
> state is Expunging
> 2013-07-25 12:30:17,694 DEBUG [cloud.capacity.CapacityManagerImpl]
> (UserVm-Scavenger-1:null) VM state transitted from :Ex punging to
> Expunging with event: ExpungeOperationvm's original host id: 19 new host
> id: null host id before state transit
> ion: null
> 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (UserVm-Scavenger-1:null) Destroying vm VM[User|test09 0703-1]
> 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (UserVm-Scavenger-1:null) Cleaning up NICS
> 2013-07-25 12:30:17,694 DEBUG [cloud.network.NetworkManagerImpl]
> (UserVm-Scavenger-1:null) Cleaning network for vm: 6164
> 2013-07-25 12:30:17,696 DEBUG [cloud.storage.StorageManagerImpl]
> (UserVm-Scavenger-1:null) Cleaning storage for vm: 6164
> 2013-07-25 12:30:17,697 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (UserVm-Scavenger-1:null) Expunged VM[User|test090703- 1]
> 2013-07-25 12:30:17,697 DEBUG [cloud.vm.UserVmManagerImpl]
> (UserVm-Scavenger-1:null) Starting cleaning up vm VM[User|test 090703-1]
> resources...
> 2013-07-25 12:30:17,705 DEBUG [network.firewall.FirewallManagerImpl]
> (UserVm-Scavenger-1:null) No firewall rules are foun d for vm id=6164
> 2013-07-25 12:30:17,706 DEBUG [cloud.vm.UserVmManagerImpl]
> (UserVm-Scavenger-1:null) Firewall rules are removed successfu lly as a
> part of vm id=6164 expunge
> 2013-07-25 12:30:17,709 DEBUG [network.rules.RulesManagerImpl]
> (UserVm-Scavenger-1:null) No port forwarding rules are fou nd for vm
> id=6164
> 2013-07-25 12:30:17,709 DEBUG [cloud.vm.UserVmManagerImpl]
> (UserVm-Scavenger-1:null) Port forwarding rules are removed su ccessfully
> as a part of vm id=6164 expunge
> 2013-07-25 12:30:17,711 DEBUG [cloud.vm.UserVmManagerImpl]
> (UserVm-Scavenger-1:null) Removed vm id=6164 from all load bal ancers as a
> part of expunge process
> 2013-07-25 12:30:17,714 DEBUG [network.rules.RulesManagerImpl]
> (UserVm-Scavenger-1:null) Revoking all Firewallrules as a part of
> disabling static nat for public IP id=1305
> 2013-07-25 12:30:17,716 DEBUG [network.firewall.FirewallManagerImpl]
> (UserVm-Scavenger-1:null) Releasing 3 firewall rules for ip id=1305
> 2013-07-25 12:30:17,718 WARN [cloud.vm.UserVmManagerImpl]
> (UserVm-Scavenger-1:null) Unable to expunge VM[User|test090703-1]
> java.lang.NullPointerException
> at com.cloud.event.EventUtils.getDomainId(EventUtils.java:116)
> at com.cloud.event.EventUtils.saveStartedEvent(EventUtils.java:79)
> at
>
> com.cloud.event.ActionEventInterceptor.interceptStart(ActionEventInterceptor.java:47)
> at
>
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:119)
> at
>
> com.cloud.network.firewall.FirewallManagerImpl.revokeFirewallRulesForIp(FirewallManagerImpl.java:734)
> at
>
> com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1194)
> at
>
> com.cloud.vm.UserVmManagerImpl.cleanupVmResources(UserVmManagerImpl.java:1856)
> at
> com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:1787)
> at
> com.cloud.vm.UserVmManagerImpl$ExpungeTask.run(UserVmManagerImpl.java:2416)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
>
>
> --
> Regards,
> Valery
>
> http://protocol.by/slayer
>
--
Regards,
Valery
http://protocol.by/slayer