You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "Jochim, Ingo" <In...@bautzen-it.de> on 2015/06/22 22:10:05 UTC
ACS does shutdown some VMs
Hello all,
we have a strange problem here. There are a couple of running VMs.
As soon as ACS get's restarted it will start shutting down some of the VMs.
It happens with the same VMs always. We can start those VMs without problems and they stay running.
Is there some kind of destination state which might be wrong in the database?
Thank you in advance.
Ingo
--
This email was Virus checked by UTM 9. http://www.sophos.com
AW: AW: ACS does shutdown some VMs
Posted by "Jochim, Ingo" <In...@bautzen-it.de>.
It is 4.4.
-----Ursprüngliche Nachricht-----
Von: Tomasz Zięba [mailto:t.a.zieba@gmail.com]
Gesendet: Dienstag, 23. Juni 2015 07:28
An: users@cloudstack.apache.org
Betreff: Re: AW: ACS does shutdown some VMs
What version of ACS ?
Regards,
Tomasz Zięba
Twitter: @TZieba
LinkedIn: pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/
<http://pl.linkedin.com/pub/tomasz-zi%C4%99ba-ph-d/3b/7a8/ab6/>
sent from mobile
22 cze 2015 23:36 "Jochim, Ingo" <In...@bautzen-it.de> napisał(a):
> Hello Andrei,
>
> The VM is not a HA VM.
>
> How can I check the queue?
>
> cloudmonkey list asyncjobs ???
>
> isdefault = False # does this mean it should be off ?
>
> Regards,
> Ingo
>
>
> ________________________________________
> Von: Andrei Mikhailovsky [andrei@arhont.com]
> Gesendet: Montag, 22. Juni 2015 22:58
> An: users@cloudstack.apache.org
> Betreff: Re: ACS does shutdown some VMs
>
> Hi Ingo,
>
> It might be an HA issue (check if these vms are set to HA). Also, it
> might be a broken job. Check out the jobs in the queue.
>
> Andrei
>
>
>
> ----- Original Message -----
>
> From: "Ingo Jochim" <In...@bautzen-it.de>
> To: users@cloudstack.apache.org
> Sent: Monday, 22 June, 2015 9:10:05 PM
> Subject: ACS does shutdown some VMs
>
> Hello all,
>
> we have a strange problem here. There are a couple of running VMs.
> As soon as ACS get's restarted it will start shutting down some of the VMs.
> It happens with the same VMs always. We can start those VMs without
> problems and they stay running.
> Is there some kind of destination state which might be wrong in the
> database?
>
> Thank you in advance.
> Ingo
> --
> This email was Virus checked by UTM 9. http://www.sophos.com
>
>
> --
> This email was Virus checked by UTM 9. http://www.sophos.com
>
--
This email was Virus checked by UTM 9. http://www.sophos.com
storage migration fails and fills secondary storage
Posted by "Jochim, Ingo" <In...@bautzen-it.de>.
Hello all,
I have opened a bug on storage migration from NFS to Ceph.
Somehow the migration starts even if there is not enough free space on the secondary storage.
After failure there is no cleanup of the broken migration.
Does anyone have the same experience?
https://issues.apache.org/jira/browse/CLOUDSTACK-8587
Thanks and regards,
Ingo
RE: ACS does shutdown some VMs
Posted by Patrick Chevalley <pa...@b-i.com>.
-----Original Message-----
From: Jochim, Ingo [mailto:Ingo.Jochim@bautzen-it.de]
Sent: jeudi 25 juin 2015 08:15
To: 'users@cloudstack.apache.org'
Subject: AW: ACS does shutdown some VMs
Hello all,
any idea how to cleanup the job table?
Thanks and regards,
Ingo
-----Ursprüngliche Nachricht-----
Von: Jochim, Ingo [mailto:Ingo.Jochim@bautzen-it.de]
Gesendet: Dienstag, 23. Juni 2015 14:59
An: 'users@cloudstack.apache.org'
Betreff: AW: ACS does shutdown some VMs
Hello Remi,
ACS has version 4.4.1.
This is the worker output. We never configured HA. How can we get rid of this?
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
| VM | Type | VM state | Created (-2H) | HAworker step taken | Step | Hypervisor | Mgt server |
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
| hts001 | Migration | Running | 2015-06-22 14:40:23 | None | Scheduled | hv01 | None |
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
Note: Found 1 HA workers.
The job queue is also quite long. What happens there?
Regards,
Ingo
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
| username | account_name | instance_name | vm_state | job_cmd | job_dispatcher | Job created | Mgt Server | Job ID | Related job ID |
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
| user01 | admin | None | None | com.cloud.vm.VmWorkReboot | VmWorkJobDispatcher | 2015-06-22 11:04:30 | None | 3791 | 3790 |
| user01 | admin | None | None | com.cloud.vm.VmWorkRemoveNicFromVm | VmWorkJobDispatcher | 2015-06-23 10:30:01 | None | 3860 | 3859 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-22 12:26:21 | None | 3817 | 3816 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-22 13:38:06 | None | 3832 | 3831 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-23 06:55:58 | None | 3857 | 3856 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStop | VmWorkJobDispatcher | 2015-06-22 09:21:34 | None | 3778 | 3777 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStop | VmWorkJobDispatcher | 2015-06-22 10:55:04 | None | 3789 | 3788 |
| user01 | admin | None | None | org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-23 10:30:01 | None | 3859 | |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:43 | None | 3601 | 3590 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:45 | None | 3602 | 3567 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:47 | None | 3604 | 3559 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:03:07 | None | 3616 | 3612 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 11:53:36 | None | 3704 | 3609 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:21 | None | 3718 | 3610 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:21 | None | 3722 | 3611 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:49 | None | 3733 | 3609 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 13:01:36 | None | 3743 | 3610 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 13:01:44 | None | 3749 | 3606 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:08:26 | None | 3796 | 3698 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:08:27 | None | 3798 | 3607 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:09:32 | None | 3809 | 3615 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 13:37:15 | None | 3825 | 3819 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3836 | 3818 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3837 | 3556 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3839 | 3559 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3841 | 3570 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3842 | 3567 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3845 | 3819 |
| system | system | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-18 07:59:09 | None | 3600 | 3578 |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.RebootVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 11:04:30 | None | 3790 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 12:26:21 | None | 3816 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 13:38:06 | None | 3831 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 13:48:39 | None | 3835 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-23 06:55:58 | None | 3856 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 09:21:34 | None | 3777 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 10:55:04 | None | 3788 | |
| system | system | s-32-VM | Expunging | None | pseudoJobDispatcher | 2015-06-20 11:53:33 | None | 3698 | |
| system | system | s-33-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3609 | |
| system | system | s-34-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3606 | |
| system | system | s-35-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3611 | |
| system | system | s-36-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3610 | |
| system | system | s-37-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:03 | None | 3615 | |
| system | system | s-38-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:03 | None | 3614 | |
| system | system | s-39-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3612 | |
| system | system | s-40-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3613 | |
| system | system | s-41-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3607 | |
| system | system | s-42-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3608 | |
| system | system | s-43-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:47 | None | 3617 | |
| system | system | s-44-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:47 | None | 3618 | |
| system | system | s-45-VM | Expunging | None | pseudoJobDispatcher | 2015-06-20 12:35:04 | None | 3734 | |
| system | system | s-82-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:52:09 | None | 3578 | |
| system | system | s-83-VM | Expunging | None | pseudoJobDispatcher | 2015-06-22 13:37:12 | None | 3819 | |
| system | system | s-84-VM | Expunging | None | pseudoJobDispatcher | 2015-06-22 13:37:12 | None | 3818 | |
| system | system | s-85-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:52:09 | None | 3579 | |
| system | system | s-86-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3559 | |
| system | system | s-87-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3570 | |
| system | system | s-88-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3558 | |
| system | system | s-89-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3560 | |
| system | system | s-90-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3556 | |
| system | system | s-91-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3557 | |
| system | system | s-92-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3567 | |
| system | system | s-93-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:57:41 | None | 3590 | |
| system | system | s-94-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3561 | |
| system | system | s-95-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3569 | |
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
Note: Found 64 running jobs.
-----Ursprüngliche Nachricht-----
Von: Remi Bergsma [mailto:RBergsma@schubergphilis.com]
Gesendet: Dienstag, 23. Juni 2015 09:41
An: users@cloudstack.apache.org
Betreff: Re: ACS does shutdown some VMs
Hi,
It’d double check the HA worker table to see if there are any left-over HA workers. Also, there could still be other jobs scheduled. If the API does not return anything useful, you may have a look at the DB itself.
I wrote scripts to easily display HA workers [1] and jobs [2] which may help you as well. You can also execute the queries yourself, see [3] and [4] to get an idea.
Regards,
Remi
[1] https://github.com/remibergsma/cloudstackOps/blob/master/listHAWorkers.py
[2] https://github.com/remibergsma/cloudstackOps/blob/master/listRunningJobs.py
[3] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L76
[4] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L109
On 23 Jun 2015, at 07:27, Tomasz Zięba <t....@gmail.com>> wrote:
What version of ACS ?
Regards,
Tomasz Zięba
Twitter: @TZieba
LinkedIn: pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/<http://pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/>
<http://pl.linkedin.com/pub/tomasz-zi%C4%99ba-ph-d/3b/7a8/ab6/>
sent from mobile
22 cze 2015 23:36 "Jochim, Ingo" <In...@bautzen-it.de>> napisał(a):
Hello Andrei,
The VM is not a HA VM.
How can I check the queue?
cloudmonkey list asyncjobs ???
isdefault = False # does this mean it should be off ?
Regards,
Ingo
________________________________________
Von: Andrei Mikhailovsky [andrei@arhont.com<ma...@arhont.com>]
Gesendet: Montag, 22. Juni 2015 22:58
An: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Betreff: Re: ACS does shutdown some VMs
Hi Ingo,
It might be an HA issue (check if these vms are set to HA). Also, it might be a broken job. Check out the jobs in the queue.
Andrei
----- Original Message -----
From: "Ingo Jochim" <In...@bautzen-it.de>>
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Sent: Monday, 22 June, 2015 9:10:05 PM
Subject: ACS does shutdown some VMs
Hello all,
we have a strange problem here. There are a couple of running VMs.
As soon as ACS get's restarted it will start shutting down some of the VMs.
It happens with the same VMs always. We can start those VMs without problems and they stay running.
Is there some kind of destination state which might be wrong in the database?
Thank you in advance.
Ingo
--
This email was Virus checked by UTM 9. http://www.sophos.com
--
This email was Virus checked by UTM 9. http://www.sophos.com
AW: ACS does shutdown some VMs
Posted by "Jochim, Ingo" <In...@bautzen-it.de>.
Hello Patrick,
good news. One more reason for upgrade.
Thanks and regards,
Ingo
-----Ursprüngliche Nachricht-----
Von: Patrick Chevalley [mailto:patrick.chevalley@b-i.com]
Gesendet: Mittwoch, 8. Juli 2015 10:01
An: users@cloudstack.apache.org
Betreff: RE: ACS does shutdown some VMs
Hello Ingo,
Wrong manipulation, sorry for the previous email.
Ran into a similar problem, where my management log was displaying:
Unexpected exception when trying to execute queue item,
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.PreparedStatement@6d36a3e5: DELETE FROM async_job WHERE async_job.id= 17813 Cannot delete or update a parent row: a foreign key constraint fails (`cloud`.`async_job_join_map`, CONSTRAINT `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES `async_job` (`id`))
I had an async job stuck that couldn't be cleaned properly and thus blocking all subsequent jobs.
This has been fixed according to:
https://issues.apache.org/jira/browse/CLOUDSTACK-7749
Hope this helps,
Patrick
-----Original Message-----
From: Jochim, Ingo [mailto:Ingo.Jochim@bautzen-it.de]
Sent: jeudi 25 juin 2015 08:15
To: 'users@cloudstack.apache.org'
Subject: AW: ACS does shutdown some VMs
Hello all,
any idea how to cleanup the job table?
Thanks and regards,
Ingo
-----Ursprüngliche Nachricht-----
Von: Jochim, Ingo [mailto:Ingo.Jochim@bautzen-it.de]
Gesendet: Dienstag, 23. Juni 2015 14:59
An: 'users@cloudstack.apache.org'
Betreff: AW: ACS does shutdown some VMs
Hello Remi,
ACS has version 4.4.1.
This is the worker output. We never configured HA. How can we get rid of this?
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
| VM | Type | VM state | Created (-2H) | HAworker step taken | Step | Hypervisor | Mgt server |
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
| hts001 | Migration | Running | 2015-06-22 14:40:23 | None | Scheduled | hv01 | None |
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
Note: Found 1 HA workers.
The job queue is also quite long. What happens there?
Regards,
Ingo
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
| username | account_name | instance_name | vm_state | job_cmd | job_dispatcher | Job created | Mgt Server | Job ID | Related job ID |
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
| user01 | admin | None | None | com.cloud.vm.VmWorkReboot | VmWorkJobDispatcher | 2015-06-22 11:04:30 | None | 3791 | 3790 |
| user01 | admin | None | None | com.cloud.vm.VmWorkRemoveNicFromVm | VmWorkJobDispatcher | 2015-06-23 10:30:01 | None | 3860 | 3859 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-22 12:26:21 | None | 3817 | 3816 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-22 13:38:06 | None | 3832 | 3831 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-23 06:55:58 | None | 3857 | 3856 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStop | VmWorkJobDispatcher | 2015-06-22 09:21:34 | None | 3778 | 3777 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStop | VmWorkJobDispatcher | 2015-06-22 10:55:04 | None | 3789 | 3788 |
| user01 | admin | None | None | org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-23 10:30:01 | None | 3859 | |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:43 | None | 3601 | 3590 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:45 | None | 3602 | 3567 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:47 | None | 3604 | 3559 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:03:07 | None | 3616 | 3612 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 11:53:36 | None | 3704 | 3609 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:21 | None | 3718 | 3610 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:21 | None | 3722 | 3611 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:49 | None | 3733 | 3609 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 13:01:36 | None | 3743 | 3610 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 13:01:44 | None | 3749 | 3606 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:08:26 | None | 3796 | 3698 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:08:27 | None | 3798 | 3607 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:09:32 | None | 3809 | 3615 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 13:37:15 | None | 3825 | 3819 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3836 | 3818 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3837 | 3556 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3839 | 3559 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3841 | 3570 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3842 | 3567 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3845 | 3819 |
| system | system | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-18 07:59:09 | None | 3600 | 3578 |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.RebootVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 11:04:30 | None | 3790 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 12:26:21 | None | 3816 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 13:38:06 | None | 3831 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 13:48:39 | None | 3835 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-23 06:55:58 | None | 3856 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 09:21:34 | None | 3777 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 10:55:04 | None | 3788 | |
| system | system | s-32-VM | Expunging | None | pseudoJobDispatcher | 2015-06-20 11:53:33 | None | 3698 | |
| system | system | s-33-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3609 | |
| system | system | s-34-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3606 | |
| system | system | s-35-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3611 | |
| system | system | s-36-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3610 | |
| system | system | s-37-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:03 | None | 3615 | |
| system | system | s-38-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:03 | None | 3614 | |
| system | system | s-39-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3612 | |
| system | system | s-40-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3613 | |
| system | system | s-41-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3607 | |
| system | system | s-42-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3608 | |
| system | system | s-43-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:47 | None | 3617 | |
| system | system | s-44-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:47 | None | 3618 | |
| system | system | s-45-VM | Expunging | None | pseudoJobDispatcher | 2015-06-20 12:35:04 | None | 3734 | |
| system | system | s-82-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:52:09 | None | 3578 | |
| system | system | s-83-VM | Expunging | None | pseudoJobDispatcher | 2015-06-22 13:37:12 | None | 3819 | |
| system | system | s-84-VM | Expunging | None | pseudoJobDispatcher | 2015-06-22 13:37:12 | None | 3818 | |
| system | system | s-85-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:52:09 | None | 3579 | |
| system | system | s-86-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3559 | |
| system | system | s-87-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3570 | |
| system | system | s-88-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3558 | |
| system | system | s-89-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3560 | |
| system | system | s-90-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3556 | |
| system | system | s-91-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3557 | |
| system | system | s-92-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3567 | |
| system | system | s-93-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:57:41 | None | 3590 | |
| system | system | s-94-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3561 | |
| system | system | s-95-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3569 | |
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
Note: Found 64 running jobs.
-----Ursprüngliche Nachricht-----
Von: Remi Bergsma [mailto:RBergsma@schubergphilis.com]
Gesendet: Dienstag, 23. Juni 2015 09:41
An: users@cloudstack.apache.org
Betreff: Re: ACS does shutdown some VMs
Hi,
It’d double check the HA worker table to see if there are any left-over HA workers. Also, there could still be other jobs scheduled. If the API does not return anything useful, you may have a look at the DB itself.
I wrote scripts to easily display HA workers [1] and jobs [2] which may help you as well. You can also execute the queries yourself, see [3] and [4] to get an idea.
Regards,
Remi
[1] https://github.com/remibergsma/cloudstackOps/blob/master/listHAWorkers.py
[2] https://github.com/remibergsma/cloudstackOps/blob/master/listRunningJobs.py
[3] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L76
[4] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L109
On 23 Jun 2015, at 07:27, Tomasz Zięba <t....@gmail.com>> wrote:
What version of ACS ?
Regards,
Tomasz Zięba
Twitter: @TZieba
LinkedIn: pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/<http://pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/>
<http://pl.linkedin.com/pub/tomasz-zi%C4%99ba-ph-d/3b/7a8/ab6/>
sent from mobile
22 cze 2015 23:36 "Jochim, Ingo" <In...@bautzen-it.de>> napisał(a):
Hello Andrei,
The VM is not a HA VM.
How can I check the queue?
cloudmonkey list asyncjobs ???
isdefault = False # does this mean it should be off ?
Regards,
Ingo
________________________________________
Von: Andrei Mikhailovsky [andrei@arhont.com<ma...@arhont.com>]
Gesendet: Montag, 22. Juni 2015 22:58
An: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Betreff: Re: ACS does shutdown some VMs
Hi Ingo,
It might be an HA issue (check if these vms are set to HA). Also, it might be a broken job. Check out the jobs in the queue.
Andrei
----- Original Message -----
From: "Ingo Jochim" <In...@bautzen-it.de>>
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Sent: Monday, 22 June, 2015 9:10:05 PM
Subject: ACS does shutdown some VMs
Hello all,
we have a strange problem here. There are a couple of running VMs.
As soon as ACS get's restarted it will start shutting down some of the VMs.
It happens with the same VMs always. We can start those VMs without problems and they stay running.
Is there some kind of destination state which might be wrong in the database?
Thank you in advance.
Ingo
--
This email was Virus checked by UTM 9. http://www.sophos.com
--
This email was Virus checked by UTM 9. http://www.sophos.com
RE: ACS does shutdown some VMs
Posted by Patrick Chevalley <pa...@b-i.com>.
Hello Ingo,
Wrong manipulation, sorry for the previous email.
Ran into a similar problem, where my management log was displaying:
Unexpected exception when trying to execute queue item,
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.PreparedStatement@6d36a3e5: DELETE FROM async_job WHERE async_job.id= 17813
Cannot delete or update a parent row: a foreign key constraint fails (`cloud`.`async_job_join_map`, CONSTRAINT `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES `async_job` (`id`))
I had an async job stuck that couldn't be cleaned properly and thus blocking all subsequent jobs.
This has been fixed according to:
https://issues.apache.org/jira/browse/CLOUDSTACK-7749
Hope this helps,
Patrick
-----Original Message-----
From: Jochim, Ingo [mailto:Ingo.Jochim@bautzen-it.de]
Sent: jeudi 25 juin 2015 08:15
To: 'users@cloudstack.apache.org'
Subject: AW: ACS does shutdown some VMs
Hello all,
any idea how to cleanup the job table?
Thanks and regards,
Ingo
-----Ursprüngliche Nachricht-----
Von: Jochim, Ingo [mailto:Ingo.Jochim@bautzen-it.de]
Gesendet: Dienstag, 23. Juni 2015 14:59
An: 'users@cloudstack.apache.org'
Betreff: AW: ACS does shutdown some VMs
Hello Remi,
ACS has version 4.4.1.
This is the worker output. We never configured HA. How can we get rid of this?
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
| VM | Type | VM state | Created (-2H) | HAworker step taken | Step | Hypervisor | Mgt server |
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
| hts001 | Migration | Running | 2015-06-22 14:40:23 | None | Scheduled | hv01 | None |
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
Note: Found 1 HA workers.
The job queue is also quite long. What happens there?
Regards,
Ingo
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
| username | account_name | instance_name | vm_state | job_cmd | job_dispatcher | Job created | Mgt Server | Job ID | Related job ID |
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
| user01 | admin | None | None | com.cloud.vm.VmWorkReboot | VmWorkJobDispatcher | 2015-06-22 11:04:30 | None | 3791 | 3790 |
| user01 | admin | None | None | com.cloud.vm.VmWorkRemoveNicFromVm | VmWorkJobDispatcher | 2015-06-23 10:30:01 | None | 3860 | 3859 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-22 12:26:21 | None | 3817 | 3816 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-22 13:38:06 | None | 3832 | 3831 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-23 06:55:58 | None | 3857 | 3856 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStop | VmWorkJobDispatcher | 2015-06-22 09:21:34 | None | 3778 | 3777 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStop | VmWorkJobDispatcher | 2015-06-22 10:55:04 | None | 3789 | 3788 |
| user01 | admin | None | None | org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-23 10:30:01 | None | 3859 | |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:43 | None | 3601 | 3590 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:45 | None | 3602 | 3567 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:47 | None | 3604 | 3559 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:03:07 | None | 3616 | 3612 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 11:53:36 | None | 3704 | 3609 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:21 | None | 3718 | 3610 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:21 | None | 3722 | 3611 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:49 | None | 3733 | 3609 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 13:01:36 | None | 3743 | 3610 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 13:01:44 | None | 3749 | 3606 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:08:26 | None | 3796 | 3698 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:08:27 | None | 3798 | 3607 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:09:32 | None | 3809 | 3615 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 13:37:15 | None | 3825 | 3819 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3836 | 3818 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3837 | 3556 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3839 | 3559 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3841 | 3570 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3842 | 3567 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3845 | 3819 |
| system | system | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-18 07:59:09 | None | 3600 | 3578 |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.RebootVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 11:04:30 | None | 3790 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 12:26:21 | None | 3816 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 13:38:06 | None | 3831 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 13:48:39 | None | 3835 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-23 06:55:58 | None | 3856 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 09:21:34 | None | 3777 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 10:55:04 | None | 3788 | |
| system | system | s-32-VM | Expunging | None | pseudoJobDispatcher | 2015-06-20 11:53:33 | None | 3698 | |
| system | system | s-33-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3609 | |
| system | system | s-34-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3606 | |
| system | system | s-35-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3611 | |
| system | system | s-36-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3610 | |
| system | system | s-37-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:03 | None | 3615 | |
| system | system | s-38-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:03 | None | 3614 | |
| system | system | s-39-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3612 | |
| system | system | s-40-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3613 | |
| system | system | s-41-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3607 | |
| system | system | s-42-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3608 | |
| system | system | s-43-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:47 | None | 3617 | |
| system | system | s-44-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:47 | None | 3618 | |
| system | system | s-45-VM | Expunging | None | pseudoJobDispatcher | 2015-06-20 12:35:04 | None | 3734 | |
| system | system | s-82-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:52:09 | None | 3578 | |
| system | system | s-83-VM | Expunging | None | pseudoJobDispatcher | 2015-06-22 13:37:12 | None | 3819 | |
| system | system | s-84-VM | Expunging | None | pseudoJobDispatcher | 2015-06-22 13:37:12 | None | 3818 | |
| system | system | s-85-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:52:09 | None | 3579 | |
| system | system | s-86-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3559 | |
| system | system | s-87-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3570 | |
| system | system | s-88-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3558 | |
| system | system | s-89-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3560 | |
| system | system | s-90-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3556 | |
| system | system | s-91-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3557 | |
| system | system | s-92-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3567 | |
| system | system | s-93-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:57:41 | None | 3590 | |
| system | system | s-94-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3561 | |
| system | system | s-95-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3569 | |
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
Note: Found 64 running jobs.
-----Ursprüngliche Nachricht-----
Von: Remi Bergsma [mailto:RBergsma@schubergphilis.com]
Gesendet: Dienstag, 23. Juni 2015 09:41
An: users@cloudstack.apache.org
Betreff: Re: ACS does shutdown some VMs
Hi,
It’d double check the HA worker table to see if there are any left-over HA workers. Also, there could still be other jobs scheduled. If the API does not return anything useful, you may have a look at the DB itself.
I wrote scripts to easily display HA workers [1] and jobs [2] which may help you as well. You can also execute the queries yourself, see [3] and [4] to get an idea.
Regards,
Remi
[1] https://github.com/remibergsma/cloudstackOps/blob/master/listHAWorkers.py
[2] https://github.com/remibergsma/cloudstackOps/blob/master/listRunningJobs.py
[3] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L76
[4] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L109
On 23 Jun 2015, at 07:27, Tomasz Zięba <t....@gmail.com>> wrote:
What version of ACS ?
Regards,
Tomasz Zięba
Twitter: @TZieba
LinkedIn: pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/<http://pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/>
<http://pl.linkedin.com/pub/tomasz-zi%C4%99ba-ph-d/3b/7a8/ab6/>
sent from mobile
22 cze 2015 23:36 "Jochim, Ingo" <In...@bautzen-it.de>> napisał(a):
Hello Andrei,
The VM is not a HA VM.
How can I check the queue?
cloudmonkey list asyncjobs ???
isdefault = False # does this mean it should be off ?
Regards,
Ingo
________________________________________
Von: Andrei Mikhailovsky [andrei@arhont.com<ma...@arhont.com>]
Gesendet: Montag, 22. Juni 2015 22:58
An: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Betreff: Re: ACS does shutdown some VMs
Hi Ingo,
It might be an HA issue (check if these vms are set to HA). Also, it might be a broken job. Check out the jobs in the queue.
Andrei
----- Original Message -----
From: "Ingo Jochim" <In...@bautzen-it.de>>
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Sent: Monday, 22 June, 2015 9:10:05 PM
Subject: ACS does shutdown some VMs
Hello all,
we have a strange problem here. There are a couple of running VMs.
As soon as ACS get's restarted it will start shutting down some of the VMs.
It happens with the same VMs always. We can start those VMs without problems and they stay running.
Is there some kind of destination state which might be wrong in the database?
Thank you in advance.
Ingo
--
This email was Virus checked by UTM 9. http://www.sophos.com
--
This email was Virus checked by UTM 9. http://www.sophos.com
AW: ACS does shutdown some VMs
Posted by "Jochim, Ingo" <In...@bautzen-it.de>.
Hello all,
any idea how to cleanup the job table?
Thanks and regards,
Ingo
-----Ursprüngliche Nachricht-----
Von: Jochim, Ingo [mailto:Ingo.Jochim@bautzen-it.de]
Gesendet: Dienstag, 23. Juni 2015 14:59
An: 'users@cloudstack.apache.org'
Betreff: AW: ACS does shutdown some VMs
Hello Remi,
ACS has version 4.4.1.
This is the worker output. We never configured HA. How can we get rid of this?
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
| VM | Type | VM state | Created (-2H) | HAworker step taken | Step | Hypervisor | Mgt server |
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
| hts001 | Migration | Running | 2015-06-22 14:40:23 | None | Scheduled | hv01 | None |
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
Note: Found 1 HA workers.
The job queue is also quite long. What happens there?
Regards,
Ingo
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
| username | account_name | instance_name | vm_state | job_cmd | job_dispatcher | Job created | Mgt Server | Job ID | Related job ID |
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
| user01 | admin | None | None | com.cloud.vm.VmWorkReboot | VmWorkJobDispatcher | 2015-06-22 11:04:30 | None | 3791 | 3790 |
| user01 | admin | None | None | com.cloud.vm.VmWorkRemoveNicFromVm | VmWorkJobDispatcher | 2015-06-23 10:30:01 | None | 3860 | 3859 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-22 12:26:21 | None | 3817 | 3816 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-22 13:38:06 | None | 3832 | 3831 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-23 06:55:58 | None | 3857 | 3856 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStop | VmWorkJobDispatcher | 2015-06-22 09:21:34 | None | 3778 | 3777 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStop | VmWorkJobDispatcher | 2015-06-22 10:55:04 | None | 3789 | 3788 |
| user01 | admin | None | None | org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-23 10:30:01 | None | 3859 | |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:43 | None | 3601 | 3590 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:45 | None | 3602 | 3567 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:47 | None | 3604 | 3559 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:03:07 | None | 3616 | 3612 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 11:53:36 | None | 3704 | 3609 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:21 | None | 3718 | 3610 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:21 | None | 3722 | 3611 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:49 | None | 3733 | 3609 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 13:01:36 | None | 3743 | 3610 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 13:01:44 | None | 3749 | 3606 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:08:26 | None | 3796 | 3698 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:08:27 | None | 3798 | 3607 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:09:32 | None | 3809 | 3615 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 13:37:15 | None | 3825 | 3819 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3836 | 3818 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3837 | 3556 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3839 | 3559 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3841 | 3570 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3842 | 3567 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3845 | 3819 |
| system | system | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-18 07:59:09 | None | 3600 | 3578 |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.RebootVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 11:04:30 | None | 3790 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 12:26:21 | None | 3816 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 13:38:06 | None | 3831 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 13:48:39 | None | 3835 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-23 06:55:58 | None | 3856 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 09:21:34 | None | 3777 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 10:55:04 | None | 3788 | |
| system | system | s-32-VM | Expunging | None | pseudoJobDispatcher | 2015-06-20 11:53:33 | None | 3698 | |
| system | system | s-33-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3609 | |
| system | system | s-34-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3606 | |
| system | system | s-35-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3611 | |
| system | system | s-36-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3610 | |
| system | system | s-37-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:03 | None | 3615 | |
| system | system | s-38-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:03 | None | 3614 | |
| system | system | s-39-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3612 | |
| system | system | s-40-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3613 | |
| system | system | s-41-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3607 | |
| system | system | s-42-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3608 | |
| system | system | s-43-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:47 | None | 3617 | |
| system | system | s-44-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:47 | None | 3618 | |
| system | system | s-45-VM | Expunging | None | pseudoJobDispatcher | 2015-06-20 12:35:04 | None | 3734 | |
| system | system | s-82-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:52:09 | None | 3578 | |
| system | system | s-83-VM | Expunging | None | pseudoJobDispatcher | 2015-06-22 13:37:12 | None | 3819 | |
| system | system | s-84-VM | Expunging | None | pseudoJobDispatcher | 2015-06-22 13:37:12 | None | 3818 | |
| system | system | s-85-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:52:09 | None | 3579 | |
| system | system | s-86-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3559 | |
| system | system | s-87-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3570 | |
| system | system | s-88-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3558 | |
| system | system | s-89-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3560 | |
| system | system | s-90-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3556 | |
| system | system | s-91-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3557 | |
| system | system | s-92-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3567 | |
| system | system | s-93-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:57:41 | None | 3590 | |
| system | system | s-94-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3561 | |
| system | system | s-95-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3569 | |
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
Note: Found 64 running jobs.
-----Ursprüngliche Nachricht-----
Von: Remi Bergsma [mailto:RBergsma@schubergphilis.com]
Gesendet: Dienstag, 23. Juni 2015 09:41
An: users@cloudstack.apache.org
Betreff: Re: ACS does shutdown some VMs
Hi,
It’d double check the HA worker table to see if there are any left-over HA workers. Also, there could still be other jobs scheduled. If the API does not return anything useful, you may have a look at the DB itself.
I wrote scripts to easily display HA workers [1] and jobs [2] which may help you as well. You can also execute the queries yourself, see [3] and [4] to get an idea.
Regards,
Remi
[1] https://github.com/remibergsma/cloudstackOps/blob/master/listHAWorkers.py
[2] https://github.com/remibergsma/cloudstackOps/blob/master/listRunningJobs.py
[3] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L76
[4] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L109
On 23 Jun 2015, at 07:27, Tomasz Zięba <t....@gmail.com>> wrote:
What version of ACS ?
Regards,
Tomasz Zięba
Twitter: @TZieba
LinkedIn: pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/<http://pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/>
<http://pl.linkedin.com/pub/tomasz-zi%C4%99ba-ph-d/3b/7a8/ab6/>
sent from mobile
22 cze 2015 23:36 "Jochim, Ingo" <In...@bautzen-it.de>> napisał(a):
Hello Andrei,
The VM is not a HA VM.
How can I check the queue?
cloudmonkey list asyncjobs ???
isdefault = False # does this mean it should be off ?
Regards,
Ingo
________________________________________
Von: Andrei Mikhailovsky [andrei@arhont.com<ma...@arhont.com>]
Gesendet: Montag, 22. Juni 2015 22:58
An: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Betreff: Re: ACS does shutdown some VMs
Hi Ingo,
It might be an HA issue (check if these vms are set to HA). Also, it might be a broken job. Check out the jobs in the queue.
Andrei
----- Original Message -----
From: "Ingo Jochim" <In...@bautzen-it.de>>
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Sent: Monday, 22 June, 2015 9:10:05 PM
Subject: ACS does shutdown some VMs
Hello all,
we have a strange problem here. There are a couple of running VMs.
As soon as ACS get's restarted it will start shutting down some of the VMs.
It happens with the same VMs always. We can start those VMs without problems and they stay running.
Is there some kind of destination state which might be wrong in the database?
Thank you in advance.
Ingo
--
This email was Virus checked by UTM 9. http://www.sophos.com
--
This email was Virus checked by UTM 9. http://www.sophos.com
AW: ACS does shutdown some VMs
Posted by "Jochim, Ingo" <In...@bautzen-it.de>.
Hello Remi,
ACS has version 4.4.1.
This is the worker output. We never configured HA. How can we get rid of this?
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
| VM | Type | VM state | Created (-2H) | HAworker step taken | Step | Hypervisor | Mgt server |
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
| hts001 | Migration | Running | 2015-06-22 14:40:23 | None | Scheduled | hv01 | None |
+--------+-----------+----------+---------------------+---------------------+-----------+------------+------------+
Note: Found 1 HA workers.
The job queue is also quite long. What happens there?
Regards,
Ingo
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
| username | account_name | instance_name | vm_state | job_cmd | job_dispatcher | Job created | Mgt Server | Job ID | Related job ID |
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
| user01 | admin | None | None | com.cloud.vm.VmWorkReboot | VmWorkJobDispatcher | 2015-06-22 11:04:30 | None | 3791 | 3790 |
| user01 | admin | None | None | com.cloud.vm.VmWorkRemoveNicFromVm | VmWorkJobDispatcher | 2015-06-23 10:30:01 | None | 3860 | 3859 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-22 12:26:21 | None | 3817 | 3816 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-22 13:38:06 | None | 3832 | 3831 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-23 06:55:58 | None | 3857 | 3856 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStop | VmWorkJobDispatcher | 2015-06-22 09:21:34 | None | 3778 | 3777 |
| user01 | admin | None | None | com.cloud.vm.VmWorkStop | VmWorkJobDispatcher | 2015-06-22 10:55:04 | None | 3789 | 3788 |
| user01 | admin | None | None | org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-23 10:30:01 | None | 3859 | |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:43 | None | 3601 | 3590 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:45 | None | 3602 | 3567 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:01:47 | None | 3604 | 3559 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-18 08:03:07 | None | 3616 | 3612 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 11:53:36 | None | 3704 | 3609 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:21 | None | 3718 | 3610 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:21 | None | 3722 | 3611 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 12:34:49 | None | 3733 | 3609 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 13:01:36 | None | 3743 | 3610 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-20 13:01:44 | None | 3749 | 3606 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:08:26 | None | 3796 | 3698 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:08:27 | None | 3798 | 3607 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 12:09:32 | None | 3809 | 3615 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 13:37:15 | None | 3825 | 3819 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3836 | 3818 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3837 | 3556 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3839 | 3559 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3841 | 3570 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3842 | 3567 |
| system | system | None | None | com.cloud.vm.VmWorkMigrateAway | VmWorkJobDispatcher | 2015-06-22 14:40:24 | None | 3845 | 3819 |
| system | system | None | None | com.cloud.vm.VmWorkStart | VmWorkJobDispatcher | 2015-06-18 07:59:09 | None | 3600 | 3578 |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.RebootVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 11:04:30 | None | 3790 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 12:26:21 | None | 3816 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 13:38:06 | None | 3831 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 13:48:39 | None | 3835 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-23 06:55:58 | None | 3856 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 09:21:34 | None | 3777 | |
| user01 | admin | i-116-616-VM | Stopped | org.apache.cloudstack.api.command.admin.vm.StopVMCmdByAdmin | ApiAsyncJobDispatcher | 2015-06-22 10:55:04 | None | 3788 | |
| system | system | s-32-VM | Expunging | None | pseudoJobDispatcher | 2015-06-20 11:53:33 | None | 3698 | |
| system | system | s-33-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3609 | |
| system | system | s-34-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3606 | |
| system | system | s-35-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3611 | |
| system | system | s-36-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3610 | |
| system | system | s-37-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:03 | None | 3615 | |
| system | system | s-38-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:03 | None | 3614 | |
| system | system | s-39-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3612 | |
| system | system | s-40-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3613 | |
| system | system | s-41-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3607 | |
| system | system | s-42-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:02 | None | 3608 | |
| system | system | s-43-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:47 | None | 3617 | |
| system | system | s-44-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 08:03:47 | None | 3618 | |
| system | system | s-45-VM | Expunging | None | pseudoJobDispatcher | 2015-06-20 12:35:04 | None | 3734 | |
| system | system | s-82-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:52:09 | None | 3578 | |
| system | system | s-83-VM | Expunging | None | pseudoJobDispatcher | 2015-06-22 13:37:12 | None | 3819 | |
| system | system | s-84-VM | Expunging | None | pseudoJobDispatcher | 2015-06-22 13:37:12 | None | 3818 | |
| system | system | s-85-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:52:09 | None | 3579 | |
| system | system | s-86-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3559 | |
| system | system | s-87-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3570 | |
| system | system | s-88-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3558 | |
| system | system | s-89-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3560 | |
| system | system | s-90-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3556 | |
| system | system | s-91-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3557 | |
| system | system | s-92-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3567 | |
| system | system | s-93-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:57:41 | None | 3590 | |
| system | system | s-94-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:25 | None | 3561 | |
| system | system | s-95-VM | Expunging | None | pseudoJobDispatcher | 2015-06-18 07:51:26 | None | 3569 | |
+----------+--------------+---------------+-----------+----------------------------------------------------------------------+-----------------------+---------------------+------------+--------+----------------+
Note: Found 64 running jobs.
-----Ursprüngliche Nachricht-----
Von: Remi Bergsma [mailto:RBergsma@schubergphilis.com]
Gesendet: Dienstag, 23. Juni 2015 09:41
An: users@cloudstack.apache.org
Betreff: Re: ACS does shutdown some VMs
Hi,
It’d double check the HA worker table to see if there are any left-over HA workers. Also, there could still be other jobs scheduled. If the API does not return anything useful, you may have a look at the DB itself.
I wrote scripts to easily display HA workers [1] and jobs [2] which may help you as well. You can also execute the queries yourself, see [3] and [4] to get an idea.
Regards,
Remi
[1] https://github.com/remibergsma/cloudstackOps/blob/master/listHAWorkers.py
[2] https://github.com/remibergsma/cloudstackOps/blob/master/listRunningJobs.py
[3] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L76
[4] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L109
On 23 Jun 2015, at 07:27, Tomasz Zięba <t....@gmail.com>> wrote:
What version of ACS ?
Regards,
Tomasz Zięba
Twitter: @TZieba
LinkedIn: pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/<http://pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/>
<http://pl.linkedin.com/pub/tomasz-zi%C4%99ba-ph-d/3b/7a8/ab6/>
sent from mobile
22 cze 2015 23:36 "Jochim, Ingo" <In...@bautzen-it.de>> napisał(a):
Hello Andrei,
The VM is not a HA VM.
How can I check the queue?
cloudmonkey list asyncjobs ???
isdefault = False # does this mean it should be off ?
Regards,
Ingo
________________________________________
Von: Andrei Mikhailovsky [andrei@arhont.com<ma...@arhont.com>]
Gesendet: Montag, 22. Juni 2015 22:58
An: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Betreff: Re: ACS does shutdown some VMs
Hi Ingo,
It might be an HA issue (check if these vms are set to HA). Also, it might be a broken job. Check out the jobs in the queue.
Andrei
----- Original Message -----
From: "Ingo Jochim" <In...@bautzen-it.de>>
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Sent: Monday, 22 June, 2015 9:10:05 PM
Subject: ACS does shutdown some VMs
Hello all,
we have a strange problem here. There are a couple of running VMs.
As soon as ACS get's restarted it will start shutting down some of the VMs.
It happens with the same VMs always. We can start those VMs without problems and they stay running.
Is there some kind of destination state which might be wrong in the database?
Thank you in advance.
Ingo
--
This email was Virus checked by UTM 9. http://www.sophos.com
--
This email was Virus checked by UTM 9. http://www.sophos.com
Re: ACS does shutdown some VMs
Posted by Remi Bergsma <RB...@schubergphilis.com>.
Hi,
It’d double check the HA worker table to see if there are any left-over HA workers. Also, there could still be other jobs scheduled. If the API does not return anything useful, you may have a look at the DB itself.
I wrote scripts to easily display HA workers [1] and jobs [2] which may help you as well. You can also execute the queries yourself, see [3] and [4] to get an idea.
Regards,
Remi
[1] https://github.com/remibergsma/cloudstackOps/blob/master/listHAWorkers.py
[2] https://github.com/remibergsma/cloudstackOps/blob/master/listRunningJobs.py
[3] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L76
[4] https://github.com/remibergsma/cloudstackOps/blob/master/cloudstackops/cloudstacksql.py#L109
On 23 Jun 2015, at 07:27, Tomasz Zięba <t....@gmail.com>> wrote:
What version of ACS ?
Regards,
Tomasz Zięba
Twitter: @TZieba
LinkedIn: pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/<http://pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/>
<http://pl.linkedin.com/pub/tomasz-zi%C4%99ba-ph-d/3b/7a8/ab6/>
sent from mobile
22 cze 2015 23:36 "Jochim, Ingo" <In...@bautzen-it.de>> napisał(a):
Hello Andrei,
The VM is not a HA VM.
How can I check the queue?
cloudmonkey list asyncjobs ???
isdefault = False # does this mean it should be off ?
Regards,
Ingo
________________________________________
Von: Andrei Mikhailovsky [andrei@arhont.com<ma...@arhont.com>]
Gesendet: Montag, 22. Juni 2015 22:58
An: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Betreff: Re: ACS does shutdown some VMs
Hi Ingo,
It might be an HA issue (check if these vms are set to HA). Also, it might
be a broken job. Check out the jobs in the queue.
Andrei
----- Original Message -----
From: "Ingo Jochim" <In...@bautzen-it.de>>
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Sent: Monday, 22 June, 2015 9:10:05 PM
Subject: ACS does shutdown some VMs
Hello all,
we have a strange problem here. There are a couple of running VMs.
As soon as ACS get's restarted it will start shutting down some of the VMs.
It happens with the same VMs always. We can start those VMs without
problems and they stay running.
Is there some kind of destination state which might be wrong in the
database?
Thank you in advance.
Ingo
--
This email was Virus checked by UTM 9. http://www.sophos.com
--
This email was Virus checked by UTM 9. http://www.sophos.com
Re: AW: ACS does shutdown some VMs
Posted by Tomasz Zięba <t....@gmail.com>.
What version of ACS ?
Regards,
Tomasz Zięba
Twitter: @TZieba
LinkedIn: pl.linkedin.com/pub/tomasz-zięba-ph-d/3b/7a8/ab6/
<http://pl.linkedin.com/pub/tomasz-zi%C4%99ba-ph-d/3b/7a8/ab6/>
sent from mobile
22 cze 2015 23:36 "Jochim, Ingo" <In...@bautzen-it.de> napisał(a):
> Hello Andrei,
>
> The VM is not a HA VM.
>
> How can I check the queue?
>
> cloudmonkey list asyncjobs ???
>
> isdefault = False # does this mean it should be off ?
>
> Regards,
> Ingo
>
>
> ________________________________________
> Von: Andrei Mikhailovsky [andrei@arhont.com]
> Gesendet: Montag, 22. Juni 2015 22:58
> An: users@cloudstack.apache.org
> Betreff: Re: ACS does shutdown some VMs
>
> Hi Ingo,
>
> It might be an HA issue (check if these vms are set to HA). Also, it might
> be a broken job. Check out the jobs in the queue.
>
> Andrei
>
>
>
> ----- Original Message -----
>
> From: "Ingo Jochim" <In...@bautzen-it.de>
> To: users@cloudstack.apache.org
> Sent: Monday, 22 June, 2015 9:10:05 PM
> Subject: ACS does shutdown some VMs
>
> Hello all,
>
> we have a strange problem here. There are a couple of running VMs.
> As soon as ACS get's restarted it will start shutting down some of the VMs.
> It happens with the same VMs always. We can start those VMs without
> problems and they stay running.
> Is there some kind of destination state which might be wrong in the
> database?
>
> Thank you in advance.
> Ingo
> --
> This email was Virus checked by UTM 9. http://www.sophos.com
>
>
> --
> This email was Virus checked by UTM 9. http://www.sophos.com
>
AW: ACS does shutdown some VMs
Posted by "Jochim, Ingo" <In...@bautzen-it.de>.
Hello Andrei,
The VM is not a HA VM.
How can I check the queue?
cloudmonkey list asyncjobs ???
isdefault = False # does this mean it should be off ?
Regards,
Ingo
________________________________________
Von: Andrei Mikhailovsky [andrei@arhont.com]
Gesendet: Montag, 22. Juni 2015 22:58
An: users@cloudstack.apache.org
Betreff: Re: ACS does shutdown some VMs
Hi Ingo,
It might be an HA issue (check if these vms are set to HA). Also, it might be a broken job. Check out the jobs in the queue.
Andrei
----- Original Message -----
From: "Ingo Jochim" <In...@bautzen-it.de>
To: users@cloudstack.apache.org
Sent: Monday, 22 June, 2015 9:10:05 PM
Subject: ACS does shutdown some VMs
Hello all,
we have a strange problem here. There are a couple of running VMs.
As soon as ACS get's restarted it will start shutting down some of the VMs.
It happens with the same VMs always. We can start those VMs without problems and they stay running.
Is there some kind of destination state which might be wrong in the database?
Thank you in advance.
Ingo
--
This email was Virus checked by UTM 9. http://www.sophos.com
--
This email was Virus checked by UTM 9. http://www.sophos.com
Re: ACS does shutdown some VMs
Posted by Andrei Mikhailovsky <an...@arhont.com>.
Hi Ingo,
It might be an HA issue (check if these vms are set to HA). Also, it might be a broken job. Check out the jobs in the queue.
Andrei
----- Original Message -----
From: "Ingo Jochim" <In...@bautzen-it.de>
To: users@cloudstack.apache.org
Sent: Monday, 22 June, 2015 9:10:05 PM
Subject: ACS does shutdown some VMs
Hello all,
we have a strange problem here. There are a couple of running VMs.
As soon as ACS get's restarted it will start shutting down some of the VMs.
It happens with the same VMs always. We can start those VMs without problems and they stay running.
Is there some kind of destination state which might be wrong in the database?
Thank you in advance.
Ingo
--
This email was Virus checked by UTM 9. http://www.sophos.com