You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Patrick Chevalley <pa...@b-i.com> on 2015/07/08 09:53:53 UTC
RE: ACS does shutdown some VMs
-----Original Message-----
From: Jochim, Ingo [mailto:Ingo.Jochim@bautzen-it.de]
Sent: jeudi 25 juin 2015 08:15
To: 'users@cloudstack.apache.org'
Subject: AW: ACS does shutdown some VMs
Hello all,
any idea how to cleanup the job table?
Thanks and regards,
Ingo
-----Ursprüngliche Nachricht-----
Von: Jochim, Ingo [mailto:Ingo.Jochim@bautzen-it.de]
Gesendet: Dienstag, 23. Juni 2015 14:59
An: 'users@cloudstack.apache.org'
Betreff: AW: ACS does shutdown some VMs
Hello Remi,
ACS has version 4.4.1.
This is the worker output. We never configured HA. How can we get rid of this?
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
| VM | Type | VM state | Created (-2H) | HAworker step taken | Step | Hypervisor | Mgt server |
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
| hts001 | Migration | Running | 2015-06-22 14:40:23 | None | Scheduled | hv01 | None |
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
Note: Found 1 HA workers.
The job queue is also quite long. What happens there?
Regards,
Ingo
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
| username | account_name | instance_name | vm_state | job_cmd | job_dispatcher | Job created | Mgt Server | Job ID | Related job ID |
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
| user01 | admin | None | None | com.cloud.vm.VmWorkReboot | VmWorkJobDispatcher | 2015-06-22 11:04:30 | None | 3791 | 3790 |
| user01 | admin | None | None | com.cloud.vm.VmWorkRemoveNicFromVm | VmWorkJobDispatcher | 2015-06-23 10:30:01 | None | 3860 | 3859 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-22 12:26:21 | None | 3817 | 3816 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-22 13:38:06 | None | 3832 | 3831 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-23 06:55:58 | None | 3857 | 3856 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStop | VmWorkJobDispatcher | 2015-06-22 09:21:34 | None | 3778 | 3777 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStop | VmWorkJobDispatcher | 2015-06-22 10:55:04 | None | 3789 | 3788 |
| user01 | admin | None | None | org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-23 10:30:01 | None | 3859 | |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:43 | None | 3601 | 3590 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:45 | None | 3602 | 3567 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:47 | None | 3604 | 3559 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:03:07 | None | 3616 | 3612 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 11:53:36 | None | 3704 | 3609 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:21 | None | 3718 | 3610 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:21 | None | 3722 | 3611 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:49 | None | 3733 | 3609 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 13:01:36 | None | 3743 | 3610 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 13:01:44 | None | 3749 | 3606 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:08:26 | None | 3796 | 3698 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:08:27 | None | 3798 | 3607 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:09:32 | None | 3809 | 3615 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 13:37:15 | None | 3825 | 3819 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3836 | 3818 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3837 | 3556 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3839 | 3559 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3841 | 3570 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3842 | 3567 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3845 | 3819 |
| system | system | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-18 07:59:09 | None | 3600 | 3578 |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.RebootVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 11:04:30 | None | 3790 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 12:26:21 | None | 3816 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 13:38:06 | None | 3831 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 13:48:39 | None | 3835 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-23 06:55:58 | None | 3856 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 09:21:34 | None | 3777 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 10:55:04 | None | 3788 | |
| system | system | s-32-VM | Expunging | None | pseudoJobDispatcher | 2015-06-20 11:53:33 | None | 3698 | |
| system | system | s-33-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3609 | |
| system | system | s-34-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3606 | |
| system | system | s-35-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3611 | |
| system | system | s-36-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3610 | |
| system | system | s-37-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:03 | None | 3615 | |
| system | system | s-38-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:03 | None | 3614 | |
| system | system | s-39-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3612 | |
| system | system | s-40-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3613 | |
| system | system | s-41-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3607 | |
| system | system | s-42-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3608 | |
| system | system | s-43-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:47 | None | 3617 | |
| system | system | s-44-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:47 | None | 3618 | |
| system | system | s-45-VM | Expunging | None | pseudoJobDispatcher | 2015-06-20 12:35:04 | None | 3734 | |
| system | system | s-82-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:52:09 | None | 3578 | |
| system | system | s-83-VM | Expunging | None | pseudoJobDispatcher | 2015-06-22 13:37:12 | None | 3819 | |
| system | system | s-84-VM | Expunging | None | pseudoJobDispatcher | 2015-06-22 13:37:12 | None | 3818 | |
| system | system | s-85-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:52:09 | None | 3579 | |
| system | system | s-86-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3559 | |
| system | system | s-87-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3570 | |
| system | system | s-88-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3558 | |
| system | system | s-89-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3560 | |
| system | system | s-90-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3556 | |
| system | system | s-91-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3557 | |
| system | system | s-92-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3567 | |
| system | system | s-93-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:57:41 | None | 3590 | |
| system | system | s-94-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3561 | |
| system | system | s-95-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3569 | |
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
Note: Found 64 running jobs.
-----Ursprüngliche Nachricht-----
Von: Remi Bergsma [mailto:RBergsma@schubergphilis.com]
Gesendet: Dienstag, 23. Juni 2015 09:41
An: users@cloudstack.apache.org
Betreff: Re: ACS does shutdown some VMs
Hi,
It’d double check the HA worker table to see if there are any left-over HA workers. Also, there could still be other jobs scheduled. If the API does not return anything useful, you may have a look at the DB itself.
I wrote scripts to easily display HA workers [1] and jobs [2] which may help you as well. You can also execute the queries yourself, see [3] and [4] to get an idea.
Regards,
Remi
[1] https://github.com/remibergsma/cloudstackOps/blob/master/listHAWorkers.py
[2] https://github.com/remibergsma/cloudstackOps/blob/master/listRunningJobs.py
[3] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L76
[4] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L109
On 23 Jun 2015, at 07:27, Tomasz Zięba <t....@gmail.com>> wrote:
What version of ACS ?
Regards,
Tomasz Zięba
Twitter: @TZieba
LinkedIn: pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/<http://pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/>
<http://pl.linkedin.com/pub/tomasz-zi%C4%99ba-ph-d/3b/7a8/ab6/>
sent from mobile
22 cze 2015 23:36 "Jochim, Ingo" <In...@bautzen-it.de>> napisał(a):
Hello Andrei,
The VM is not a HA VM.
How can I check the queue?
cloudmonkey list asyncjobs ???
isdefault = False # does this mean it should be off ?
Regards,
Ingo
________________________________________
Von: Andrei Mikhailovsky [andrei@arhont.com<ma...@arhont.com>]
Gesendet: Montag, 22. Juni 2015 22:58
An: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Betreff: Re: ACS does shutdown some VMs
Hi Ingo,
It might be an HA issue (check if these vms are set to HA). Also, it might be a broken job. Check out the jobs in the queue.
Andrei
----- Original Message -----
From: "Ingo Jochim" <In...@bautzen-it.de>>
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Sent: Monday, 22 June, 2015 9:10:05 PM
Subject: ACS does shutdown some VMs
Hello all,
we have a strange problem here. There are a couple of running VMs.
As soon as ACS get's restarted it will start shutting down some of the VMs.
It happens with the same VMs always. We can start those VMs without problems and they stay running.
Is there some kind of destination state which might be wrong in the database?
Thank you in advance.
Ingo
--
This email was Virus checked by UTM 9. http://www.sophos.com
--
This email was Virus checked by UTM 9. http://www.sophos.com