You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Carlos Reategui <cr...@gmail.com> on 2013/10/29 19:53:28 UTC
Can't destroy system VMs
I am trying to recover my CloudStack installation (CS 4.1.1 on ubuntu + XS
6.0.2) that I shutdown and cannot bring back to life.
I tried to destroy the CPVM and the VR and all I am seeing in the logs is
the following and they are filling up fast (management log is almost 1GB).
2013-10-29 11:51:08,291 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-14:null) Seq 1-201919186: Forwarding Seq 1-201919186:
{ Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
[{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
} to 233845174730255
2013-10-29 11:51:08,293 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-4:null) Seq 2-1168706613: Forwarding Seq
2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
100111,
[{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
} to 233845174730255
2013-10-29 11:51:08,293 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-2:null) Seq 1-201919186: Forwarding Seq 1-201919186:
{ Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
[{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
} to 233845174730255
2013-10-29 11:51:08,294 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-8:null) Seq 2-1168706613: Forwarding Seq
2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
100111,
[{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
} to 233845174730255
2013-10-29 11:51:08,294 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-9:null) Seq 1-201919186: Forwarding Seq 1-201919186:
{ Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
[{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
} to 233845174730255
2013-10-29 11:51:08,296 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-6:null) Seq 2-1168706613: Forwarding Seq
2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
100111,
[{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
} to 233845174730255
2013-10-29 11:51:08,296 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-15:null) Seq 1-201919186: Forwarding Seq 1-201919186:
{ Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
[{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
} to 233845174730255
2013-10-29 11:51:08,297 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-13:null) Seq 2-1168706613: Forwarding Seq
2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
100111,
[{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
} to 233845174730255
2013-10-29 11:51:08,297 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-10:null) Seq 1-201919186: Forwarding Seq 1-201919186:
{ Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
[{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
} to 233845174730255
2013-10-29 11:51:08,299 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-3:null) Seq 2-1168706613: Forwarding Seq
2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
100111,
[{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
} to 233845174730255
Re: Can't destroy system VMs
Posted by Daan Hoogland <da...@gmail.com>.
Sorry Carlos,
family and $dayjob kept me from looking at this anymore. Glad you got
it working.
Daan
On Wed, Nov 6, 2013 at 8:08 PM, Carlos Reategui <ca...@reategui.com> wrote:
> Since I didn't hear back I bit the bullet and decided to make the DB
> changes and I am back up and running. Woohoo. I'll post the specifics in
> the other thread I started about the mshost id.
>
>
> On Tue, Nov 5, 2013 at 11:36 AM, Carlos Reategui <ca...@reategui.com>wrote:
>
>> Hi Daan,
>>
>> The mshost thing I mentioned previously has been bugging me and as I look
>> through the db I think I have a problem here. I am guessing I have a
>> botched 4.1.0 to 4.1.1 upgrade that I did not notice when I shut things
>> down.
>>
>> The mshost table is showing 2 management servers that are supposedly on
>> the same machine listening on the same port which is not really possible.
>> So really only one of the 2 is up (I believe that would be the 4.1.1
>> version). The logs do seem to indicate something along those lines where
>> the 2 management servers are failing to communicate with each other.
>>
>> If I look at the host table I see my 2 hosts but they say they are on
>> version 4.1.0 and are pointing to management server id of the 4.1.0 line in
>> the mshost table.
>>
>> I am wondering if this is the problem and why their is no communication to
>> my hosts from the management server.
>>
>> Anyone know the schema well enough to help me update some stuff and
>> recover my installation? Would an upgrade to 4.2 possibly cleanup my DB?
>>
>> thanks,
>> Carlos
>>
>>
>>
>>
>> On Tue, Nov 5, 2013 at 10:59 AM, Carlos Reategui <ca...@reategui.com>wrote:
>>
>>> Hi Daan,
>>>
>>> I have been looking through my logs and can't figure out why the ssvm
>>> does not want to start.
>>>
>>> Here is a link to my latest logs:
>>> http://reategui.com/cloudstack/management-server-131105.log
>>>
>>> Maybe someone else can spot something.
>>>
>>> There is nothing on either the XenServer hosts logs.
>>>
>>> thanks,
>>> Carlos
>>>
>>>
>>> On Tue, Nov 5, 2013 at 12:55 AM, Daan Hoogland <da...@gmail.com>wrote:
>>>
>>>> Carlos,
>>>>
>>>> It makes no sense trying to start the vms if not the ssvm is started
>>>> first.
>>>>
>>>> I would say stop the system, clear the logs and start again. Then
>>>> analyse the logs to see what is wrong with the secondary storage vm.
>>>>
>>>> On Tue, Nov 5, 2013 at 2:08 AM, Carlos Reategui <ca...@reategui.com>
>>>> wrote:
>>>> > Hi Daan,
>>>> > There is no trace of CS communicating with either of my XS hosts in the
>>>> > SMLog file or the xensource.log files when I try to start one of my
>>>> stopped
>>>> > instances.
>>>> >
>>>> > thanks,
>>>> > Carlos
>>>> >
>>>> >
>>>> >
>>>> > On Sat, Nov 2, 2013 at 4:50 AM, Daan Hoogland <daan.hoogland@gmail.com
>>>> >wrote:
>>>> >
>>>> >> H Carlos,
>>>> >>
>>>> >> I wouldn't know about the mshost and mshost_peer but they seem
>>>> >> alright. The expunging ssvm are a real problem. I think Ahmed
>>>> >> mentioned that as well. You are having problems getting your ssvm in
>>>> >> the air. All other things are consequential. This is the first thing
>>>> >> to look at. Check the logs and the SMLog on the hypervisor to see what
>>>> >> happens when cs tries to start a ssvm. By your db you can see that it
>>>> >> tried to start it several times. Those are the expunging vm_instance
>>>> >> entries.
>>>> >>
>>>> >> regards,
>>>> >> Daan
>>>> >>
>>>> >> On Fri, Nov 1, 2013 at 9:10 PM, Carlos Reategui <ca...@reategui.com>
>>>> >> wrote:
>>>> >> > Hi Daan :)
>>>> >> >
>>>> >> > Primary and Secondary storage are on the MS. Primary is on a
>>>> different
>>>> >> > subnet and it is currently mounted (and visible as an NFS SR from
>>>> >> > XenCenter) and working fine from the xen machines. I have also
>>>> been able
>>>> >> > to mount Secondary without a problem.
>>>> >> >
>>>> >> > Here are my current system VMs:
>>>> >> > mysql> select
>>>> >> > id,name,instance_name,state,host_id,type,vm_type,desired_state from
>>>> >> > vm_instance where type!='User';
>>>> >> >
>>>> >>
>>>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
>>>> >> > | id | name | instance_name | state | host_id | type
>>>> >> |
>>>> >> > vm_type | desired_state |
>>>> >> >
>>>> >>
>>>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
>>>> >> > | 2 | v-2-VM | v-2-VM | Stopped | 1 | ConsoleProxy
>>>> >> |
>>>> >> > ConsoleProxy | NULL |
>>>> >> > | 5 | r-5-VM | r-5-VM | Stopped | 2 | DomainRouter
>>>> >> |
>>>> >> > DomainRouter | NULL |
>>>> >> > | 1 | s-1-VM | s-1-VM | Expunging | NULL |
>>>> >> SecondaryStorageVm |
>>>> >> > SecondaryStorageVm | NULL |
>>>> >> > | 3 | s-3-VM | s-3-VM | Expunging | NULL |
>>>> >> SecondaryStorageVm |
>>>> >> > SecondaryStorageVm | NULL |
>>>> >> > | 52 | s-52-VM | s-52-VM | Expunging | NULL |
>>>> >> SecondaryStorageVm |
>>>> >> > SecondaryStorageVm | NULL |
>>>> >> > | 53 | s-53-VM | s-53-VM | Expunging | NULL |
>>>> >> SecondaryStorageVm |
>>>> >> > SecondaryStorageVm | NULL |
>>>> >> > | 54 | s-54-VM | s-54-VM | Expunging | NULL |
>>>> >> SecondaryStorageVm |
>>>> >> > SecondaryStorageVm | NULL |
>>>> >> > | 55 | s-55-VM | s-55-VM | Expunging | NULL |
>>>> >> SecondaryStorageVm |
>>>> >> > SecondaryStorageVm | NULL |
>>>> >> > | 56 | s-56-VM | s-56-VM | Expunging | NULL |
>>>> >> SecondaryStorageVm |
>>>> >> > SecondaryStorageVm | NULL |
>>>> >> >
>>>> >>
>>>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
>>>> >> >
>>>> >> >
>>>> >> > I have been looking through the DB to see if anything sticks out. I
>>>> >> notice
>>>> >> > that the mshost and mshost_peer table have 2 entries but they are
>>>> the
>>>> >> same
>>>> >> > server. Is that ok? Looks like one is from the install of 4.1.0
>>>> an the
>>>> >> > other from after I upgraded to 4.1.1:
>>>> >> >
>>>> >> > mysql> select * from mshost;
>>>> >> >
>>>> >>
>>>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
>>>> >> > | id | msid | runid | name | state |
>>>> version |
>>>> >> > service_ip | service_port | last_update | removed |
>>>> alert_count
>>>> >> |
>>>> >> >
>>>> >>
>>>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
>>>> >> > | 1 | 233845174730255 | 1375383759777 | isbustor01 | Up | 4.1.0
>>>> |
>>>> >> > 172.30.45.2 | 9090 | 2013-10-24 21:13:10 | NULL |
>>>> 0
>>>> >> |
>>>> >> > | 2 | 233845174730253 | 1383331508626 | isbustor01 | Up | 4.1.1
>>>> |
>>>> >> > 172.30.45.2 | 9090 | 2013-11-01 20:00:04 | NULL |
>>>> 0
>>>> >> |
>>>> >> >
>>>> >>
>>>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
>>>> >> > 2 rows in set (0.00 sec)
>>>> >> >
>>>> >> > mysql> select * from mshost_peer;
>>>> >> >
>>>> >>
>>>> +----+--------------+-------------+---------------+------------+---------------------+
>>>> >> > | id | owner_mshost | peer_mshost | peer_runid | peer_state |
>>>> >> > last_update |
>>>> >> >
>>>> >>
>>>> +----+--------------+-------------+---------------+------------+---------------------+
>>>> >> > | 17 | 1 | 1 | 1375383759777 | Up |
>>>> >> 2013-08-01
>>>> >> > 19:02:49 |
>>>> >> > | 34 | 2 | 2 | 1383331508626 | Up |
>>>> >> 2013-11-01
>>>> >> > 18:45:19 |
>>>> >> >
>>>> >>
>>>> +----+--------------+-------------+---------------+------------+---------------------+
>>>> >> >
>>>> >> >
>>>> >> > thanks,
>>>> >> > Carlos
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > On Fri, Nov 1, 2013 at 12:17 PM, Daan Hoogland <
>>>> daan.hoogland@gmail.com
>>>> >> >wrote:
>>>> >> >
>>>> >> >> h Carlos,
>>>> >> >>
>>>> >> >> the states of virtual machines, especially the entries for the
>>>> ssvm.
>>>> >> >> if it is not Stopped. set it to stopped and check if your storage,
>>>> >> >> both primary and secondary, are reachable botjh from the ms and the
>>>> >> >> hosts.
>>>> >> >>
>>>> >> >> Daan (double a!)
>>>> >> >>
>>>> >> >> On Fri, Nov 1, 2013 at 7:52 PM, Carlos Reategui <
>>>> creategui@gmail.com>
>>>> >> >> wrote:
>>>> >> >> > Hi Dan,
>>>> >> >> > The host value is correct and I am able to connect on ports
>>>> 22,80 and
>>>> >> >> 8080
>>>> >> >> > from the Xen hosts to that IP (same subnet). Port 8787 is not
>>>> >> listening
>>>> >> >> on
>>>> >> >> > my MS.
>>>> >> >> >
>>>> >> >> > What can I look for in the DB?
>>>> >> >> >
>>>> >> >> > thanks
>>>> >> >> > Carlos
>>>> >> >> >
>>>> >> >> >
>>>> >> >> >
>>>> >> >> >
>>>> >> >> > On Fri, Nov 1, 2013 at 2:47 AM, Daan Hoogland <
>>>> >> daan.hoogland@gmail.com>
>>>> >> >> > wrote:
>>>> >> >> >>
>>>> >> >> >> H Carlox,
>>>> >> >> >>
>>>> >> >> >> It still seems to me you have a communication problem albeit
>>>> not at
>>>> >> >> >> the network level. ( I agree this doesn't seem likely) But even
>>>> so
>>>> >> the
>>>> >> >> >> ms can't reach the xen hosts. The next thing to look at would be
>>>> >> >> >> configuration. I presume you did check the ip addresses for the
>>>> >> hosts.
>>>> >> >> >> check the value 'host' in the global vars and try to telnet it
>>>> on
>>>> >> port
>>>> >> >> >> 22 80 8080 8787 or what else from the xen hosts . I am sure the
>>>> >> >> >> problem is hidden somewhere in the mysql database.
>>>> >> >> >>
>>>> >> >> >> regards,
>>>> >> >> >> Daan
>>>> >> >> >>
>>>> >> >> >> On Fri, Nov 1, 2013 at 6:23 AM, Carlos Reategui <
>>>> carlos@reategui.com
>>>> >> >
>>>> >> >> >> wrote:
>>>> >> >> >> > Hi Dan,
>>>> >> >> >> > Finally getting around to trying this. I am not able to
>>>> start any
>>>> >> vm
>>>> >> >> >> > (mine
>>>> >> >> >> > or system).
>>>> >> >> >> >
>>>> >> >> >> > I am in the process of trying Ahmad's suggestion of removing
>>>> one of
>>>> >> >> the
>>>> >> >> >> > hosts, reinstalling it and then adding it back in to see if
>>>> that
>>>> >> >> helps.
>>>> >> >> >> > However I am not able to put it in maintenance mode. I am
>>>> getting
>>>> >> the
>>>> >> >> >> > following when I try:
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> > 2013-10-31 15:19:59,120 DEBUG [cloud.api.ApiServlet]
>>>> >> >> >> > (catalina-exec-5:null)
>>>> >> >> >> > ===START=== 172.30.40.135 -- GET command=prepareHostForMa
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >>
>>>> >>
>>>> intenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
>>>> >> >> >> > 2013-10-31 15:19:59,181 DEBUG
>>>> [cloud.async.AsyncJobManagerImpl]
>>>> >> >> >> > (catalina-exec-5:null) submit async job-167, details:
>>>> AsyncJobVO
>>>> >> >> {id:16
>>>> >> >> >> > 7, userId: 2, accountId: 2, sessionKey: null, instanceType:
>>>> Host,
>>>> >> >> >> > instanceId: 1, cmd:
>>>> >> org.apache.cloudstack.api.command.admin.host.Prep
>>>> >> >> >> > areForMaintenanceCmd, cmdOriginator: null, cmdInfo:
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >>
>>>> >>
>>>> {"id":"c2f3c416-f4ba-4f73-b414-6fd6efc734e3","response":"json","sessionkey":"Yjye5q
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >>
>>>> >>
>>>> mcLtUBW0/P4XCAcx2ZTAI\u003d","ctxUserId":"2","_":"1383258001952","ctxAccountId":"2","ctxStartEventId":"699"},
>>>> >> >> >> > cmdVersion: 0, callbackTy
>>>> >> >> >> > pe: 0, callbackAddress: null, status: 0, processStatus: 0,
>>>> >> resultCode:
>>>> >> >> >> > 0,
>>>> >> >> >> > result: null, initMsid: 233845174730253, completeMsid: null,
>>>> >> >> >> > lastUpdated: null, lastPolled: null, created: null}
>>>> >> >> >> > 2013-10-31 15:19:59,185 DEBUG
>>>> [cloud.async.AsyncJobManagerImpl]
>>>> >> >> >> > (Job-Executor-1:job-167) Executing
>>>> >> >> org.apache.cloudstack.api.command.ad
>>>> >> >> >> > min.host.PrepareForMaintenanceCmd for job-167
>>>> >> >> >> > 2013-10-31 15:19:59,188 DEBUG [cloud.api.ApiServlet]
>>>> >> >> >> > (catalina-exec-5:null)
>>>> >> >> >> > ===END=== 172.30.40.135 -- GET command=prepareHostForMain
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >>
>>>> >>
>>>> tenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
>>>> >> >> >> > 2013-10-31 15:19:59,207 DEBUG
>>>> [cloud.cluster.ClusterManagerImpl]
>>>> >> >> >> > (Job-Executor-1:job-167) Propagating agent change request
>>>> >> event:AdminA
>>>> >> >> >> > skMaintenace to agent:1
>>>> >> >> >> > 2013-10-31 15:19:59,208 DEBUG
>>>> [cloud.cluster.ClusterManagerImpl]
>>>> >> >> >> > (Job-Executor-1:job-167) 233845174730253 -> 233845174730255.1
>>>> >> [{"Propa
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >>
>>>> >>
>>>> gateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
>>>> >> >> >> > 2013-10-31 15:19:59,213 INFO
>>>> >> >> [cloud.cluster.ClusterServiceServletImpl]
>>>> >> >> >> > (Cluster-Worker-1:null) Setup cluster service servlet. service
>>>> >> >> >> > url: http://172.30.45.2:9090/clusterservice, request
>>>> timeout: 300
>>>> >> >> >> > seconds
>>>> >> >> >> > 2013-10-31 15:19:59,214 DEBUG
>>>> [cloud.cluster.ClusterManagerImpl]
>>>> >> >> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 ->
>>>> >> 233845174730255
>>>> >> >> >> > . agent: 1, pdu seq: 1, pdu ack seq: 0, json:
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >>
>>>> >>
>>>> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{
>>>> >> >> >> > },"wait":0}}]
>>>> >> >> >> > 2013-10-31 15:19:59,343 DEBUG
>>>> [cloud.cluster.ClusterManagerImpl]
>>>> >> >> >> > (Cluster-Worker-7:null) Dispatch ->1, json:
>>>> >> [{"PropagateResourceEventC
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >>
>>>> >>
>>>> ommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
>>>> >> >> >> > 2013-10-31 15:19:59,347 DEBUG
>>>> [cloud.cluster.ClusterManagerImpl]
>>>> >> >> >> > (Cluster-Worker-7:null) Intercepting command to propagate
>>>> event
>>>> >> AdminA
>>>> >> >> >> > skMaintenace for host 1
>>>> >> >> >> > 2013-10-31 15:19:59,351 DEBUG
>>>> >> >> [cloud.cluster.ClusterServiceServletImpl]
>>>> >> >> >> > (Cluster-Worker-1:null) POST
>>>> http://172.30.45.2:9090/clusterser
>>>> >> >> >> > vice response :true, responding time: 80 ms
>>>> >> >> >> > 2013-10-31 15:19:59,351 DEBUG
>>>> [cloud.cluster.ClusterManagerImpl]
>>>> >> >> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 ->
>>>> >> 233845174730255
>>>> >> >> >> > completed. time: 137ms. agent: 1, pdu seq: 1, pdu ack seq: 0,
>>>> >> json:
>>>> >> >> >> >
>>>> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMai
>>>> >> >> >> > ntenace","contextMap":{},"wait":0}}]
>>>> >> >> >> > 2013-10-31 15:19:59,356 DEBUG
>>>> [agent.manager.ClusteredAgentAttache]
>>>> >> >> >> > (Cluster-Worker-7:null) Seq 1-102760483: Forwarding Seq
>>>> 1-102760483
>>>> >> >> >> > : { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>>>> 100111,
>>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>>> >> >> >> > 2013-10-31 15:19:59,360 DEBUG
>>>> [agent.manager.ClusteredAgentAttache]
>>>> >> >> >> > (AgentManager-Handler-1:null) Seq 1-102760483: Forwarding Seq
>>>> 1-102
>>>> >> >> >> > 760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>>> Flags:
>>>> >> >> >> > 100111,
>>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>>> >> >> >> > 2013-10-31 15:19:59,365 DEBUG
>>>> [agent.manager.ClusteredAgentAttache]
>>>> >> >> >> > (AgentManager-Handler-13:null) Seq 1-102760483: Forwarding
>>>> Seq 1-10
>>>> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>>> Flags:
>>>> >> >> >> > 100111,
>>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>>> >> >> >> > 2013-10-31 15:19:59,369 DEBUG
>>>> [agent.manager.ClusteredAgentAttache]
>>>> >> >> >> > (AgentManager-Handler-15:null) Seq 1-102760483: Forwarding
>>>> Seq 1-10
>>>> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>>> Flags:
>>>> >> >> >> > 100111,
>>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>>> >> >> >> > 2013-10-31 15:19:59,372 DEBUG
>>>> [agent.manager.ClusteredAgentAttache]
>>>> >> >> >> > (AgentManager-Handler-10:null) Seq 1-102760483: Forwarding
>>>> Seq 1-10
>>>> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>>> Flags:
>>>> >> >> >> > 100111,
>>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>>> >> >> >> > 2013-10-31 15:19:59,376 DEBUG
>>>> [agent.manager.ClusteredAgentAttache]
>>>> >> >> >> > (AgentManager-Handler-11:null) Seq 1-102760483: Forwarding
>>>> Seq 1-10
>>>> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>>> Flags:
>>>> >> >> >> > 100111,
>>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>>> >> >> >> >
>>>> >> >> >> > These MaintainCommand lines are repeated endlessly and very
>>>> fast
>>>> >> (1GB
>>>> >> >> of
>>>> >> >> >> > logs in about 20 mins) so I currently have the MS stopped.
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> >
>>>> >> >> >> > On Thu, Oct 31, 2013 at 3:43 PM, Carlos Reategui <
>>>> >> carlos@reategui.com
>>>> >> >> >
>>>> >> >> >> > wrote:
>>>> >> >> >> >>
>>>> >> >> >> >> Hi Dan,
>>>> >> >> >> >> Network is definitely not a problem. iptables is disabled
>>>> on MS
>>>> >> and
>>>> >> >> >> >> both
>>>> >> >> >> >> hosts. I am able to ssh from MS to both hosts. I am also
>>>> able to
>>>> >> >> >> >> connect
>>>> >> >> >> >> to port 80 from the MS.
>>>> >> >> >> >>
>>>> >> >> >> >> I need to head out now and will try what you suggested in a
>>>> couple
>>>> >> >> >> >> hours.
>>>> >> >> >> >>
>>>> >> >> >> >> thanks,
>>>> >> >> >> >> Carlos
>>>> >> >> >> >>
>>>> >> >> >> >>
>>>> >> >> >> >> On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland
>>>> >> >> >> >> <da...@gmail.com>
>>>> >> >> >> >> wrote:
>>>> >> >> >> >>>
>>>> >> >> >> >>> Carlos,
>>>> >> >> >> >>>
>>>> >> >> >> >>> can the ms reach the xs hosts?
>>>> >> >> >> >>> if so try this, Stop the ms, set the state for the
>>>> vm_instances
>>>> >> for
>>>> >> >> >> >>> cpvm and vr from expunging to stopped. and restart.
>>>> >> >> >> >>> if not you need to troubleshoot your network connectivity.
>>>> It
>>>> >> seems
>>>> >> >> >> >>> this would not be the issue as you only stopped and started
>>>> it.
>>>> >> Did
>>>> >> >> >> >>> you
>>>> >> >> >> >>>
>>>> >> >> >> >>> Daan
>>>> >> >> >> >>>
>>>> >> >> >> >>> On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui
>>>> >> >> >> >>> <ca...@reategui.com>
>>>> >> >> >> >>> wrote:
>>>> >> >> >> >>> > Hi Dan,
>>>> >> >> >> >>> >
>>>> >> >> >> >>> > Unfortunately not. Still trying to solve this. See my
>>>> last
>>>> >> email
>>>> >> >> >> >>> > to
>>>> >> >> >> >>> > Ahmad
>>>> >> >> >> >>> > in the recover cs thread about next steps I am trying to
>>>> figure
>>>> >> >> out.
>>>> >> >> >> >>> >
>>>> >> >> >> >>> > The CPVM and VR are not running. They are currently in
>>>> >> expunging
>>>> >> >> >> >>> > state.
>>>> >> >> >> >>> >
>>>> >> >> >> >>> > If I try to start an instance the MS is unable to
>>>> communicate
>>>> >> the
>>>> >> >> >> >>> > start
>>>> >> >> >> >>> > vm
>>>> >> >> >> >>> > commands to my XS hosts and then starts writing to the
>>>> logs at
>>>> >> an
>>>> >> >> >> >>> > amazingly
>>>> >> >> >> >>> > high rate. You can see one of my logs here:
>>>> >> >> >> >>> > http://reategui.com/cloudstack/management-server.log.new
>>>> >> >> >> >>> >
>>>> >> >> >> >>> > thank you
>>>> >> >> >> >>> > Carlos
>>>> >> >> >> >>> >
>>>> >> >> >> >>> >
>>>> >> >> >> >>> >
>>>> >> >> >> >>> >
>>>> >> >> >> >>> >
>>>> >> >> >> >>> > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland
>>>> >> >> >> >>> > <da...@gmail.com>
>>>> >> >> >> >>> > wrote:
>>>> >> >> >> >>> >>
>>>> >> >> >> >>> >> H Carlos,
>>>> >> >> >> >>> >>
>>>> >> >> >> >>> >> Did you solve this problem yet?
>>>> >> >> >> >>> >>
>>>> >> >> >> >>> >> You can try removing the cpvm and vr at the hypervisor
>>>> and
>>>> >> change
>>>> >> >> >> >>> >> their state to 'Stopped' in the db.
>>>> >> >> >> >>> >>
>>>> >> >> >> >>> >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui
>>>> >> >> >> >>> >> <ca...@reategui.com>
>>>> >> >> >> >>> >> wrote:
>>>> >> >> >> >>> >> > In the last 15 minutes management log file it grew
>>>> 500MB..
>>>> >> >> >> >>> >> >
>>>> >> >> >> >>> >> > Any ideas what I should look at to figure out what is
>>>> >> >> happening?
>>>> >> >> >> >>> >> >
>>>> >> >> >> >>> >> >
>>>> >> >> >> >>> >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui
>>>> >> >> >> >>> >> > <cr...@gmail.com>wrote:
>>>> >> >> >> >>> >> >
>>>> >> >> >> >>> >> >> I am trying to recover my CloudStack installation (CS
>>>> >> 4.1.1 on
>>>> >> >> >> >>> >> >> ubuntu +
>>>> >> >> >> >>> >> >> XS
>>>> >> >> >> >>> >> >> 6.0.2) that I shutdown and cannot bring back to life.
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >> I tried to destroy the CPVM and the VR and all I am
>>>> seeing
>>>> >> in
>>>> >> >> >> >>> >> >> the
>>>> >> >> >> >>> >> >> logs
>>>> >> >> >> >>> >> >> is
>>>> >> >> >> >>> >> >> the following and they are filling up fast
>>>> (management log
>>>> >> is
>>>> >> >> >> >>> >> >> almost
>>>> >> >> >> >>> >> >> 1GB).
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,291 DEBUG
>>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>>> >> >> >> >>> >> >> (AgentManager-Handler-14:null) Seq 1-201919186:
>>>> Forwarding
>>>> >> Seq
>>>> >> >> >> >>> >> >> 1-201919186:
>>>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>>> Flags:
>>>> >> >> >> >>> >> >> 100111,
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >>
>>>> >>
>>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>>> >> >> >> >>> >> >> } to 233845174730255
>>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
>>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>>> >> >> >> >>> >> >> (AgentManager-Handler-4:null) Seq 2-1168706613:
>>>> Forwarding
>>>> >> Seq
>>>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via:
>>>> 2,
>>>> >> Ver:
>>>> >> >> v1,
>>>> >> >> >> >>> >> >> Flags:
>>>> >> >> >> >>> >> >> 100111,
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >>
>>>> >>
>>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>>> >> >> >> >>> >> >> } to 233845174730255
>>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
>>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>>> >> >> >> >>> >> >> (AgentManager-Handler-2:null) Seq 1-201919186:
>>>> Forwarding
>>>> >> Seq
>>>> >> >> >> >>> >> >> 1-201919186:
>>>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>>> Flags:
>>>> >> >> >> >>> >> >> 100111,
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >>
>>>> >>
>>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>>> >> >> >> >>> >> >> } to 233845174730255
>>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
>>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>>> >> >> >> >>> >> >> (AgentManager-Handler-8:null) Seq 2-1168706613:
>>>> Forwarding
>>>> >> Seq
>>>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via:
>>>> 2,
>>>> >> Ver:
>>>> >> >> v1,
>>>> >> >> >> >>> >> >> Flags:
>>>> >> >> >> >>> >> >> 100111,
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >>
>>>> >>
>>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>>> >> >> >> >>> >> >> } to 233845174730255
>>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
>>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>>> >> >> >> >>> >> >> (AgentManager-Handler-9:null) Seq 1-201919186:
>>>> Forwarding
>>>> >> Seq
>>>> >> >> >> >>> >> >> 1-201919186:
>>>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>>> Flags:
>>>> >> >> >> >>> >> >> 100111,
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >>
>>>> >>
>>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>>> >> >> >> >>> >> >> } to 233845174730255
>>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
>>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>>> >> >> >> >>> >> >> (AgentManager-Handler-6:null) Seq 2-1168706613:
>>>> Forwarding
>>>> >> Seq
>>>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via:
>>>> 2,
>>>> >> Ver:
>>>> >> >> v1,
>>>> >> >> >> >>> >> >> Flags:
>>>> >> >> >> >>> >> >> 100111,
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >>
>>>> >>
>>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>>> >> >> >> >>> >> >> } to 233845174730255
>>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
>>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>>> >> >> >> >>> >> >> (AgentManager-Handler-15:null) Seq 1-201919186:
>>>> Forwarding
>>>> >> Seq
>>>> >> >> >> >>> >> >> 1-201919186:
>>>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>>> Flags:
>>>> >> >> >> >>> >> >> 100111,
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >>
>>>> >>
>>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>>> >> >> >> >>> >> >> } to 233845174730255
>>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
>>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>>> >> >> >> >>> >> >> (AgentManager-Handler-13:null) Seq 2-1168706613:
>>>> Forwarding
>>>> >> >> Seq
>>>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via:
>>>> 2,
>>>> >> Ver:
>>>> >> >> v1,
>>>> >> >> >> >>> >> >> Flags:
>>>> >> >> >> >>> >> >> 100111,
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >>
>>>> >>
>>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>>> >> >> >> >>> >> >> } to 233845174730255
>>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
>>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>>> >> >> >> >>> >> >> (AgentManager-Handler-10:null) Seq 1-201919186:
>>>> Forwarding
>>>> >> Seq
>>>> >> >> >> >>> >> >> 1-201919186:
>>>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>>> Flags:
>>>> >> >> >> >>> >> >> 100111,
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >>
>>>> >>
>>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>>> >> >> >> >>> >> >> } to 233845174730255
>>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,299 DEBUG
>>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>>> >> >> >> >>> >> >> (AgentManager-Handler-3:null) Seq 2-1168706613:
>>>> Forwarding
>>>> >> Seq
>>>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via:
>>>> 2,
>>>> >> Ver:
>>>> >> >> v1,
>>>> >> >> >> >>> >> >> Flags:
>>>> >> >> >> >>> >> >> 100111,
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >>
>>>> >>
>>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>>> >> >> >> >>> >> >> } to 233845174730255
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >> >>
>>>> >> >> >> >>> >
>>>> >> >> >> >>> >
>>>> >> >> >> >>
>>>> >> >> >> >>
>>>> >> >> >> >
>>>> >> >> >
>>>> >> >> >
>>>> >> >>
>>>> >>
>>>>
>>>
>>>
>>
Re: Can't destroy system VMs
Posted by Carlos Reategui <ca...@reategui.com>.
Since I didn't hear back I bit the bullet and decided to make the DB
changes and I am back up and running. Woohoo. I'll post the specifics in
the other thread I started about the mshost id.
On Tue, Nov 5, 2013 at 11:36 AM, Carlos Reategui <ca...@reategui.com>wrote:
> Hi Daan,
>
> The mshost thing I mentioned previously has been bugging me and as I look
> through the db I think I have a problem here. I am guessing I have a
> botched 4.1.0 to 4.1.1 upgrade that I did not notice when I shut things
> down.
>
> The mshost table is showing 2 management servers that are supposedly on
> the same machine listening on the same port which is not really possible.
> So really only one of the 2 is up (I believe that would be the 4.1.1
> version). The logs do seem to indicate something along those lines where
> the 2 management servers are failing to communicate with each other.
>
> If I look at the host table I see my 2 hosts but they say they are on
> version 4.1.0 and are pointing to management server id of the 4.1.0 line in
> the mshost table.
>
> I am wondering if this is the problem and why their is no communication to
> my hosts from the management server.
>
> Anyone know the schema well enough to help me update some stuff and
> recover my installation? Would an upgrade to 4.2 possibly cleanup my DB?
>
> thanks,
> Carlos
>
>
>
>
> On Tue, Nov 5, 2013 at 10:59 AM, Carlos Reategui <ca...@reategui.com>wrote:
>
>> Hi Daan,
>>
>> I have been looking through my logs and can't figure out why the ssvm
>> does not want to start.
>>
>> Here is a link to my latest logs:
>> http://reategui.com/cloudstack/management-server-131105.log
>>
>> Maybe someone else can spot something.
>>
>> There is nothing on either the XenServer hosts logs.
>>
>> thanks,
>> Carlos
>>
>>
>> On Tue, Nov 5, 2013 at 12:55 AM, Daan Hoogland <da...@gmail.com>wrote:
>>
>>> Carlos,
>>>
>>> It makes no sense trying to start the vms if not the ssvm is started
>>> first.
>>>
>>> I would say stop the system, clear the logs and start again. Then
>>> analyse the logs to see what is wrong with the secondary storage vm.
>>>
>>> On Tue, Nov 5, 2013 at 2:08 AM, Carlos Reategui <ca...@reategui.com>
>>> wrote:
>>> > Hi Daan,
>>> > There is no trace of CS communicating with either of my XS hosts in the
>>> > SMLog file or the xensource.log files when I try to start one of my
>>> stopped
>>> > instances.
>>> >
>>> > thanks,
>>> > Carlos
>>> >
>>> >
>>> >
>>> > On Sat, Nov 2, 2013 at 4:50 AM, Daan Hoogland <daan.hoogland@gmail.com
>>> >wrote:
>>> >
>>> >> H Carlos,
>>> >>
>>> >> I wouldn't know about the mshost and mshost_peer but they seem
>>> >> alright. The expunging ssvm are a real problem. I think Ahmed
>>> >> mentioned that as well. You are having problems getting your ssvm in
>>> >> the air. All other things are consequential. This is the first thing
>>> >> to look at. Check the logs and the SMLog on the hypervisor to see what
>>> >> happens when cs tries to start a ssvm. By your db you can see that it
>>> >> tried to start it several times. Those are the expunging vm_instance
>>> >> entries.
>>> >>
>>> >> regards,
>>> >> Daan
>>> >>
>>> >> On Fri, Nov 1, 2013 at 9:10 PM, Carlos Reategui <ca...@reategui.com>
>>> >> wrote:
>>> >> > Hi Daan :)
>>> >> >
>>> >> > Primary and Secondary storage are on the MS. Primary is on a
>>> different
>>> >> > subnet and it is currently mounted (and visible as an NFS SR from
>>> >> > XenCenter) and working fine from the xen machines. I have also
>>> been able
>>> >> > to mount Secondary without a problem.
>>> >> >
>>> >> > Here are my current system VMs:
>>> >> > mysql> select
>>> >> > id,name,instance_name,state,host_id,type,vm_type,desired_state from
>>> >> > vm_instance where type!='User';
>>> >> >
>>> >>
>>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
>>> >> > | id | name | instance_name | state | host_id | type
>>> >> |
>>> >> > vm_type | desired_state |
>>> >> >
>>> >>
>>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
>>> >> > | 2 | v-2-VM | v-2-VM | Stopped | 1 | ConsoleProxy
>>> >> |
>>> >> > ConsoleProxy | NULL |
>>> >> > | 5 | r-5-VM | r-5-VM | Stopped | 2 | DomainRouter
>>> >> |
>>> >> > DomainRouter | NULL |
>>> >> > | 1 | s-1-VM | s-1-VM | Expunging | NULL |
>>> >> SecondaryStorageVm |
>>> >> > SecondaryStorageVm | NULL |
>>> >> > | 3 | s-3-VM | s-3-VM | Expunging | NULL |
>>> >> SecondaryStorageVm |
>>> >> > SecondaryStorageVm | NULL |
>>> >> > | 52 | s-52-VM | s-52-VM | Expunging | NULL |
>>> >> SecondaryStorageVm |
>>> >> > SecondaryStorageVm | NULL |
>>> >> > | 53 | s-53-VM | s-53-VM | Expunging | NULL |
>>> >> SecondaryStorageVm |
>>> >> > SecondaryStorageVm | NULL |
>>> >> > | 54 | s-54-VM | s-54-VM | Expunging | NULL |
>>> >> SecondaryStorageVm |
>>> >> > SecondaryStorageVm | NULL |
>>> >> > | 55 | s-55-VM | s-55-VM | Expunging | NULL |
>>> >> SecondaryStorageVm |
>>> >> > SecondaryStorageVm | NULL |
>>> >> > | 56 | s-56-VM | s-56-VM | Expunging | NULL |
>>> >> SecondaryStorageVm |
>>> >> > SecondaryStorageVm | NULL |
>>> >> >
>>> >>
>>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
>>> >> >
>>> >> >
>>> >> > I have been looking through the DB to see if anything sticks out. I
>>> >> notice
>>> >> > that the mshost and mshost_peer table have 2 entries but they are
>>> the
>>> >> same
>>> >> > server. Is that ok? Looks like one is from the install of 4.1.0
>>> an the
>>> >> > other from after I upgraded to 4.1.1:
>>> >> >
>>> >> > mysql> select * from mshost;
>>> >> >
>>> >>
>>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
>>> >> > | id | msid | runid | name | state |
>>> version |
>>> >> > service_ip | service_port | last_update | removed |
>>> alert_count
>>> >> |
>>> >> >
>>> >>
>>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
>>> >> > | 1 | 233845174730255 | 1375383759777 | isbustor01 | Up | 4.1.0
>>> |
>>> >> > 172.30.45.2 | 9090 | 2013-10-24 21:13:10 | NULL |
>>> 0
>>> >> |
>>> >> > | 2 | 233845174730253 | 1383331508626 | isbustor01 | Up | 4.1.1
>>> |
>>> >> > 172.30.45.2 | 9090 | 2013-11-01 20:00:04 | NULL |
>>> 0
>>> >> |
>>> >> >
>>> >>
>>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
>>> >> > 2 rows in set (0.00 sec)
>>> >> >
>>> >> > mysql> select * from mshost_peer;
>>> >> >
>>> >>
>>> +----+--------------+-------------+---------------+------------+---------------------+
>>> >> > | id | owner_mshost | peer_mshost | peer_runid | peer_state |
>>> >> > last_update |
>>> >> >
>>> >>
>>> +----+--------------+-------------+---------------+------------+---------------------+
>>> >> > | 17 | 1 | 1 | 1375383759777 | Up |
>>> >> 2013-08-01
>>> >> > 19:02:49 |
>>> >> > | 34 | 2 | 2 | 1383331508626 | Up |
>>> >> 2013-11-01
>>> >> > 18:45:19 |
>>> >> >
>>> >>
>>> +----+--------------+-------------+---------------+------------+---------------------+
>>> >> >
>>> >> >
>>> >> > thanks,
>>> >> > Carlos
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Fri, Nov 1, 2013 at 12:17 PM, Daan Hoogland <
>>> daan.hoogland@gmail.com
>>> >> >wrote:
>>> >> >
>>> >> >> h Carlos,
>>> >> >>
>>> >> >> the states of virtual machines, especially the entries for the
>>> ssvm.
>>> >> >> if it is not Stopped. set it to stopped and check if your storage,
>>> >> >> both primary and secondary, are reachable botjh from the ms and the
>>> >> >> hosts.
>>> >> >>
>>> >> >> Daan (double a!)
>>> >> >>
>>> >> >> On Fri, Nov 1, 2013 at 7:52 PM, Carlos Reategui <
>>> creategui@gmail.com>
>>> >> >> wrote:
>>> >> >> > Hi Dan,
>>> >> >> > The host value is correct and I am able to connect on ports
>>> 22,80 and
>>> >> >> 8080
>>> >> >> > from the Xen hosts to that IP (same subnet). Port 8787 is not
>>> >> listening
>>> >> >> on
>>> >> >> > my MS.
>>> >> >> >
>>> >> >> > What can I look for in the DB?
>>> >> >> >
>>> >> >> > thanks
>>> >> >> > Carlos
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > On Fri, Nov 1, 2013 at 2:47 AM, Daan Hoogland <
>>> >> daan.hoogland@gmail.com>
>>> >> >> > wrote:
>>> >> >> >>
>>> >> >> >> H Carlox,
>>> >> >> >>
>>> >> >> >> It still seems to me you have a communication problem albeit
>>> not at
>>> >> >> >> the network level. ( I agree this doesn't seem likely) But even
>>> so
>>> >> the
>>> >> >> >> ms can't reach the xen hosts. The next thing to look at would be
>>> >> >> >> configuration. I presume you did check the ip addresses for the
>>> >> hosts.
>>> >> >> >> check the value 'host' in the global vars and try to telnet it
>>> on
>>> >> port
>>> >> >> >> 22 80 8080 8787 or what else from the xen hosts . I am sure the
>>> >> >> >> problem is hidden somewhere in the mysql database.
>>> >> >> >>
>>> >> >> >> regards,
>>> >> >> >> Daan
>>> >> >> >>
>>> >> >> >> On Fri, Nov 1, 2013 at 6:23 AM, Carlos Reategui <
>>> carlos@reategui.com
>>> >> >
>>> >> >> >> wrote:
>>> >> >> >> > Hi Dan,
>>> >> >> >> > Finally getting around to trying this. I am not able to
>>> start any
>>> >> vm
>>> >> >> >> > (mine
>>> >> >> >> > or system).
>>> >> >> >> >
>>> >> >> >> > I am in the process of trying Ahmad's suggestion of removing
>>> one of
>>> >> >> the
>>> >> >> >> > hosts, reinstalling it and then adding it back in to see if
>>> that
>>> >> >> helps.
>>> >> >> >> > However I am not able to put it in maintenance mode. I am
>>> getting
>>> >> the
>>> >> >> >> > following when I try:
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > 2013-10-31 15:19:59,120 DEBUG [cloud.api.ApiServlet]
>>> >> >> >> > (catalina-exec-5:null)
>>> >> >> >> > ===START=== 172.30.40.135 -- GET command=prepareHostForMa
>>> >> >> >> >
>>> >> >> >> >
>>> >> >>
>>> >>
>>> intenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
>>> >> >> >> > 2013-10-31 15:19:59,181 DEBUG
>>> [cloud.async.AsyncJobManagerImpl]
>>> >> >> >> > (catalina-exec-5:null) submit async job-167, details:
>>> AsyncJobVO
>>> >> >> {id:16
>>> >> >> >> > 7, userId: 2, accountId: 2, sessionKey: null, instanceType:
>>> Host,
>>> >> >> >> > instanceId: 1, cmd:
>>> >> org.apache.cloudstack.api.command.admin.host.Prep
>>> >> >> >> > areForMaintenanceCmd, cmdOriginator: null, cmdInfo:
>>> >> >> >> >
>>> >> >> >> >
>>> >> >>
>>> >>
>>> {"id":"c2f3c416-f4ba-4f73-b414-6fd6efc734e3","response":"json","sessionkey":"Yjye5q
>>> >> >> >> >
>>> >> >> >> >
>>> >> >>
>>> >>
>>> mcLtUBW0/P4XCAcx2ZTAI\u003d","ctxUserId":"2","_":"1383258001952","ctxAccountId":"2","ctxStartEventId":"699"},
>>> >> >> >> > cmdVersion: 0, callbackTy
>>> >> >> >> > pe: 0, callbackAddress: null, status: 0, processStatus: 0,
>>> >> resultCode:
>>> >> >> >> > 0,
>>> >> >> >> > result: null, initMsid: 233845174730253, completeMsid: null,
>>> >> >> >> > lastUpdated: null, lastPolled: null, created: null}
>>> >> >> >> > 2013-10-31 15:19:59,185 DEBUG
>>> [cloud.async.AsyncJobManagerImpl]
>>> >> >> >> > (Job-Executor-1:job-167) Executing
>>> >> >> org.apache.cloudstack.api.command.ad
>>> >> >> >> > min.host.PrepareForMaintenanceCmd for job-167
>>> >> >> >> > 2013-10-31 15:19:59,188 DEBUG [cloud.api.ApiServlet]
>>> >> >> >> > (catalina-exec-5:null)
>>> >> >> >> > ===END=== 172.30.40.135 -- GET command=prepareHostForMain
>>> >> >> >> >
>>> >> >> >> >
>>> >> >>
>>> >>
>>> tenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
>>> >> >> >> > 2013-10-31 15:19:59,207 DEBUG
>>> [cloud.cluster.ClusterManagerImpl]
>>> >> >> >> > (Job-Executor-1:job-167) Propagating agent change request
>>> >> event:AdminA
>>> >> >> >> > skMaintenace to agent:1
>>> >> >> >> > 2013-10-31 15:19:59,208 DEBUG
>>> [cloud.cluster.ClusterManagerImpl]
>>> >> >> >> > (Job-Executor-1:job-167) 233845174730253 -> 233845174730255.1
>>> >> [{"Propa
>>> >> >> >> >
>>> >> >> >> >
>>> >> >>
>>> >>
>>> gateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
>>> >> >> >> > 2013-10-31 15:19:59,213 INFO
>>> >> >> [cloud.cluster.ClusterServiceServletImpl]
>>> >> >> >> > (Cluster-Worker-1:null) Setup cluster service servlet. service
>>> >> >> >> > url: http://172.30.45.2:9090/clusterservice, request
>>> timeout: 300
>>> >> >> >> > seconds
>>> >> >> >> > 2013-10-31 15:19:59,214 DEBUG
>>> [cloud.cluster.ClusterManagerImpl]
>>> >> >> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 ->
>>> >> 233845174730255
>>> >> >> >> > . agent: 1, pdu seq: 1, pdu ack seq: 0, json:
>>> >> >> >> >
>>> >> >> >> >
>>> >> >>
>>> >>
>>> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{
>>> >> >> >> > },"wait":0}}]
>>> >> >> >> > 2013-10-31 15:19:59,343 DEBUG
>>> [cloud.cluster.ClusterManagerImpl]
>>> >> >> >> > (Cluster-Worker-7:null) Dispatch ->1, json:
>>> >> [{"PropagateResourceEventC
>>> >> >> >> >
>>> >> >> >> >
>>> >> >>
>>> >>
>>> ommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
>>> >> >> >> > 2013-10-31 15:19:59,347 DEBUG
>>> [cloud.cluster.ClusterManagerImpl]
>>> >> >> >> > (Cluster-Worker-7:null) Intercepting command to propagate
>>> event
>>> >> AdminA
>>> >> >> >> > skMaintenace for host 1
>>> >> >> >> > 2013-10-31 15:19:59,351 DEBUG
>>> >> >> [cloud.cluster.ClusterServiceServletImpl]
>>> >> >> >> > (Cluster-Worker-1:null) POST
>>> http://172.30.45.2:9090/clusterser
>>> >> >> >> > vice response :true, responding time: 80 ms
>>> >> >> >> > 2013-10-31 15:19:59,351 DEBUG
>>> [cloud.cluster.ClusterManagerImpl]
>>> >> >> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 ->
>>> >> 233845174730255
>>> >> >> >> > completed. time: 137ms. agent: 1, pdu seq: 1, pdu ack seq: 0,
>>> >> json:
>>> >> >> >> >
>>> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMai
>>> >> >> >> > ntenace","contextMap":{},"wait":0}}]
>>> >> >> >> > 2013-10-31 15:19:59,356 DEBUG
>>> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> > (Cluster-Worker-7:null) Seq 1-102760483: Forwarding Seq
>>> 1-102760483
>>> >> >> >> > : { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>>> 100111,
>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>> >> >> >> > 2013-10-31 15:19:59,360 DEBUG
>>> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> > (AgentManager-Handler-1:null) Seq 1-102760483: Forwarding Seq
>>> 1-102
>>> >> >> >> > 760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> > 100111,
>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>> >> >> >> > 2013-10-31 15:19:59,365 DEBUG
>>> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> > (AgentManager-Handler-13:null) Seq 1-102760483: Forwarding
>>> Seq 1-10
>>> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> > 100111,
>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>> >> >> >> > 2013-10-31 15:19:59,369 DEBUG
>>> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> > (AgentManager-Handler-15:null) Seq 1-102760483: Forwarding
>>> Seq 1-10
>>> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> > 100111,
>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>> >> >> >> > 2013-10-31 15:19:59,372 DEBUG
>>> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> > (AgentManager-Handler-10:null) Seq 1-102760483: Forwarding
>>> Seq 1-10
>>> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> > 100111,
>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>> >> >> >> > 2013-10-31 15:19:59,376 DEBUG
>>> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> > (AgentManager-Handler-11:null) Seq 1-102760483: Forwarding
>>> Seq 1-10
>>> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> > 100111,
>>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>>> >> >> >> >
>>> >> >> >> > These MaintainCommand lines are repeated endlessly and very
>>> fast
>>> >> (1GB
>>> >> >> of
>>> >> >> >> > logs in about 20 mins) so I currently have the MS stopped.
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > On Thu, Oct 31, 2013 at 3:43 PM, Carlos Reategui <
>>> >> carlos@reategui.com
>>> >> >> >
>>> >> >> >> > wrote:
>>> >> >> >> >>
>>> >> >> >> >> Hi Dan,
>>> >> >> >> >> Network is definitely not a problem. iptables is disabled
>>> on MS
>>> >> and
>>> >> >> >> >> both
>>> >> >> >> >> hosts. I am able to ssh from MS to both hosts. I am also
>>> able to
>>> >> >> >> >> connect
>>> >> >> >> >> to port 80 from the MS.
>>> >> >> >> >>
>>> >> >> >> >> I need to head out now and will try what you suggested in a
>>> couple
>>> >> >> >> >> hours.
>>> >> >> >> >>
>>> >> >> >> >> thanks,
>>> >> >> >> >> Carlos
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >> On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland
>>> >> >> >> >> <da...@gmail.com>
>>> >> >> >> >> wrote:
>>> >> >> >> >>>
>>> >> >> >> >>> Carlos,
>>> >> >> >> >>>
>>> >> >> >> >>> can the ms reach the xs hosts?
>>> >> >> >> >>> if so try this, Stop the ms, set the state for the
>>> vm_instances
>>> >> for
>>> >> >> >> >>> cpvm and vr from expunging to stopped. and restart.
>>> >> >> >> >>> if not you need to troubleshoot your network connectivity.
>>> It
>>> >> seems
>>> >> >> >> >>> this would not be the issue as you only stopped and started
>>> it.
>>> >> Did
>>> >> >> >> >>> you
>>> >> >> >> >>>
>>> >> >> >> >>> Daan
>>> >> >> >> >>>
>>> >> >> >> >>> On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui
>>> >> >> >> >>> <ca...@reategui.com>
>>> >> >> >> >>> wrote:
>>> >> >> >> >>> > Hi Dan,
>>> >> >> >> >>> >
>>> >> >> >> >>> > Unfortunately not. Still trying to solve this. See my
>>> last
>>> >> email
>>> >> >> >> >>> > to
>>> >> >> >> >>> > Ahmad
>>> >> >> >> >>> > in the recover cs thread about next steps I am trying to
>>> figure
>>> >> >> out.
>>> >> >> >> >>> >
>>> >> >> >> >>> > The CPVM and VR are not running. They are currently in
>>> >> expunging
>>> >> >> >> >>> > state.
>>> >> >> >> >>> >
>>> >> >> >> >>> > If I try to start an instance the MS is unable to
>>> communicate
>>> >> the
>>> >> >> >> >>> > start
>>> >> >> >> >>> > vm
>>> >> >> >> >>> > commands to my XS hosts and then starts writing to the
>>> logs at
>>> >> an
>>> >> >> >> >>> > amazingly
>>> >> >> >> >>> > high rate. You can see one of my logs here:
>>> >> >> >> >>> > http://reategui.com/cloudstack/management-server.log.new
>>> >> >> >> >>> >
>>> >> >> >> >>> > thank you
>>> >> >> >> >>> > Carlos
>>> >> >> >> >>> >
>>> >> >> >> >>> >
>>> >> >> >> >>> >
>>> >> >> >> >>> >
>>> >> >> >> >>> >
>>> >> >> >> >>> > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland
>>> >> >> >> >>> > <da...@gmail.com>
>>> >> >> >> >>> > wrote:
>>> >> >> >> >>> >>
>>> >> >> >> >>> >> H Carlos,
>>> >> >> >> >>> >>
>>> >> >> >> >>> >> Did you solve this problem yet?
>>> >> >> >> >>> >>
>>> >> >> >> >>> >> You can try removing the cpvm and vr at the hypervisor
>>> and
>>> >> change
>>> >> >> >> >>> >> their state to 'Stopped' in the db.
>>> >> >> >> >>> >>
>>> >> >> >> >>> >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui
>>> >> >> >> >>> >> <ca...@reategui.com>
>>> >> >> >> >>> >> wrote:
>>> >> >> >> >>> >> > In the last 15 minutes management log file it grew
>>> 500MB..
>>> >> >> >> >>> >> >
>>> >> >> >> >>> >> > Any ideas what I should look at to figure out what is
>>> >> >> happening?
>>> >> >> >> >>> >> >
>>> >> >> >> >>> >> >
>>> >> >> >> >>> >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui
>>> >> >> >> >>> >> > <cr...@gmail.com>wrote:
>>> >> >> >> >>> >> >
>>> >> >> >> >>> >> >> I am trying to recover my CloudStack installation (CS
>>> >> 4.1.1 on
>>> >> >> >> >>> >> >> ubuntu +
>>> >> >> >> >>> >> >> XS
>>> >> >> >> >>> >> >> 6.0.2) that I shutdown and cannot bring back to life.
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >> I tried to destroy the CPVM and the VR and all I am
>>> seeing
>>> >> in
>>> >> >> >> >>> >> >> the
>>> >> >> >> >>> >> >> logs
>>> >> >> >> >>> >> >> is
>>> >> >> >> >>> >> >> the following and they are filling up fast
>>> (management log
>>> >> is
>>> >> >> >> >>> >> >> almost
>>> >> >> >> >>> >> >> 1GB).
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,291 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-14:null) Seq 1-201919186:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 1-201919186:
>>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-4:null) Seq 2-1168706613:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via:
>>> 2,
>>> >> Ver:
>>> >> >> v1,
>>> >> >> >> >>> >> >> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-2:null) Seq 1-201919186:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 1-201919186:
>>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-8:null) Seq 2-1168706613:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via:
>>> 2,
>>> >> Ver:
>>> >> >> v1,
>>> >> >> >> >>> >> >> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-9:null) Seq 1-201919186:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 1-201919186:
>>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-6:null) Seq 2-1168706613:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via:
>>> 2,
>>> >> Ver:
>>> >> >> v1,
>>> >> >> >> >>> >> >> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-15:null) Seq 1-201919186:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 1-201919186:
>>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-13:null) Seq 2-1168706613:
>>> Forwarding
>>> >> >> Seq
>>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via:
>>> 2,
>>> >> Ver:
>>> >> >> v1,
>>> >> >> >> >>> >> >> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-10:null) Seq 1-201919186:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 1-201919186:
>>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>>> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >> 2013-10-29 11:51:08,299 DEBUG
>>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>>> >> >> >> >>> >> >> (AgentManager-Handler-3:null) Seq 2-1168706613:
>>> Forwarding
>>> >> Seq
>>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via:
>>> 2,
>>> >> Ver:
>>> >> >> v1,
>>> >> >> >> >>> >> >> Flags:
>>> >> >> >> >>> >> >> 100111,
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >>
>>> >>
>>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>>> >> >> >> >>> >> >> } to 233845174730255
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >> >>
>>> >> >> >> >>> >
>>> >> >> >> >>> >
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >>
>>> >>
>>>
>>
>>
>
Re: Can't destroy system VMs
Posted by Carlos Reategui <ca...@reategui.com>.
Hi Daan,
The mshost thing I mentioned previously has been bugging me and as I look
through the db I think I have a problem here. I am guessing I have a
botched 4.1.0 to 4.1.1 upgrade that I did not notice when I shut things
down.
The mshost table is showing 2 management servers that are supposedly on the
same machine listening on the same port which is not really possible. So
really only one of the 2 is up (I believe that would be the 4.1.1 version).
The logs do seem to indicate something along those lines where the 2
management servers are failing to communicate with each other.
If I look at the host table I see my 2 hosts but they say they are on
version 4.1.0 and are pointing to management server id of the 4.1.0 line in
the mshost table.
I am wondering if this is the problem and why their is no communication to
my hosts from the management server.
Anyone know the schema well enough to help me update some stuff and recover
my installation? Would an upgrade to 4.2 possibly cleanup my DB?
thanks,
Carlos
On Tue, Nov 5, 2013 at 10:59 AM, Carlos Reategui <ca...@reategui.com>wrote:
> Hi Daan,
>
> I have been looking through my logs and can't figure out why the ssvm does
> not want to start.
>
> Here is a link to my latest logs:
> http://reategui.com/cloudstack/management-server-131105.log
>
> Maybe someone else can spot something.
>
> There is nothing on either the XenServer hosts logs.
>
> thanks,
> Carlos
>
>
> On Tue, Nov 5, 2013 at 12:55 AM, Daan Hoogland <da...@gmail.com>wrote:
>
>> Carlos,
>>
>> It makes no sense trying to start the vms if not the ssvm is started
>> first.
>>
>> I would say stop the system, clear the logs and start again. Then
>> analyse the logs to see what is wrong with the secondary storage vm.
>>
>> On Tue, Nov 5, 2013 at 2:08 AM, Carlos Reategui <ca...@reategui.com>
>> wrote:
>> > Hi Daan,
>> > There is no trace of CS communicating with either of my XS hosts in the
>> > SMLog file or the xensource.log files when I try to start one of my
>> stopped
>> > instances.
>> >
>> > thanks,
>> > Carlos
>> >
>> >
>> >
>> > On Sat, Nov 2, 2013 at 4:50 AM, Daan Hoogland <daan.hoogland@gmail.com
>> >wrote:
>> >
>> >> H Carlos,
>> >>
>> >> I wouldn't know about the mshost and mshost_peer but they seem
>> >> alright. The expunging ssvm are a real problem. I think Ahmed
>> >> mentioned that as well. You are having problems getting your ssvm in
>> >> the air. All other things are consequential. This is the first thing
>> >> to look at. Check the logs and the SMLog on the hypervisor to see what
>> >> happens when cs tries to start a ssvm. By your db you can see that it
>> >> tried to start it several times. Those are the expunging vm_instance
>> >> entries.
>> >>
>> >> regards,
>> >> Daan
>> >>
>> >> On Fri, Nov 1, 2013 at 9:10 PM, Carlos Reategui <ca...@reategui.com>
>> >> wrote:
>> >> > Hi Daan :)
>> >> >
>> >> > Primary and Secondary storage are on the MS. Primary is on a
>> different
>> >> > subnet and it is currently mounted (and visible as an NFS SR from
>> >> > XenCenter) and working fine from the xen machines. I have also been
>> able
>> >> > to mount Secondary without a problem.
>> >> >
>> >> > Here are my current system VMs:
>> >> > mysql> select
>> >> > id,name,instance_name,state,host_id,type,vm_type,desired_state from
>> >> > vm_instance where type!='User';
>> >> >
>> >>
>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
>> >> > | id | name | instance_name | state | host_id | type
>> >> |
>> >> > vm_type | desired_state |
>> >> >
>> >>
>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
>> >> > | 2 | v-2-VM | v-2-VM | Stopped | 1 | ConsoleProxy
>> >> |
>> >> > ConsoleProxy | NULL |
>> >> > | 5 | r-5-VM | r-5-VM | Stopped | 2 | DomainRouter
>> >> |
>> >> > DomainRouter | NULL |
>> >> > | 1 | s-1-VM | s-1-VM | Expunging | NULL |
>> >> SecondaryStorageVm |
>> >> > SecondaryStorageVm | NULL |
>> >> > | 3 | s-3-VM | s-3-VM | Expunging | NULL |
>> >> SecondaryStorageVm |
>> >> > SecondaryStorageVm | NULL |
>> >> > | 52 | s-52-VM | s-52-VM | Expunging | NULL |
>> >> SecondaryStorageVm |
>> >> > SecondaryStorageVm | NULL |
>> >> > | 53 | s-53-VM | s-53-VM | Expunging | NULL |
>> >> SecondaryStorageVm |
>> >> > SecondaryStorageVm | NULL |
>> >> > | 54 | s-54-VM | s-54-VM | Expunging | NULL |
>> >> SecondaryStorageVm |
>> >> > SecondaryStorageVm | NULL |
>> >> > | 55 | s-55-VM | s-55-VM | Expunging | NULL |
>> >> SecondaryStorageVm |
>> >> > SecondaryStorageVm | NULL |
>> >> > | 56 | s-56-VM | s-56-VM | Expunging | NULL |
>> >> SecondaryStorageVm |
>> >> > SecondaryStorageVm | NULL |
>> >> >
>> >>
>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
>> >> >
>> >> >
>> >> > I have been looking through the DB to see if anything sticks out. I
>> >> notice
>> >> > that the mshost and mshost_peer table have 2 entries but they are the
>> >> same
>> >> > server. Is that ok? Looks like one is from the install of 4.1.0 an
>> the
>> >> > other from after I upgraded to 4.1.1:
>> >> >
>> >> > mysql> select * from mshost;
>> >> >
>> >>
>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
>> >> > | id | msid | runid | name | state |
>> version |
>> >> > service_ip | service_port | last_update | removed |
>> alert_count
>> >> |
>> >> >
>> >>
>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
>> >> > | 1 | 233845174730255 | 1375383759777 | isbustor01 | Up | 4.1.0
>> |
>> >> > 172.30.45.2 | 9090 | 2013-10-24 21:13:10 | NULL |
>> 0
>> >> |
>> >> > | 2 | 233845174730253 | 1383331508626 | isbustor01 | Up | 4.1.1
>> |
>> >> > 172.30.45.2 | 9090 | 2013-11-01 20:00:04 | NULL |
>> 0
>> >> |
>> >> >
>> >>
>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
>> >> > 2 rows in set (0.00 sec)
>> >> >
>> >> > mysql> select * from mshost_peer;
>> >> >
>> >>
>> +----+--------------+-------------+---------------+------------+---------------------+
>> >> > | id | owner_mshost | peer_mshost | peer_runid | peer_state |
>> >> > last_update |
>> >> >
>> >>
>> +----+--------------+-------------+---------------+------------+---------------------+
>> >> > | 17 | 1 | 1 | 1375383759777 | Up |
>> >> 2013-08-01
>> >> > 19:02:49 |
>> >> > | 34 | 2 | 2 | 1383331508626 | Up |
>> >> 2013-11-01
>> >> > 18:45:19 |
>> >> >
>> >>
>> +----+--------------+-------------+---------------+------------+---------------------+
>> >> >
>> >> >
>> >> > thanks,
>> >> > Carlos
>> >> >
>> >> >
>> >> >
>> >> > On Fri, Nov 1, 2013 at 12:17 PM, Daan Hoogland <
>> daan.hoogland@gmail.com
>> >> >wrote:
>> >> >
>> >> >> h Carlos,
>> >> >>
>> >> >> the states of virtual machines, especially the entries for the ssvm.
>> >> >> if it is not Stopped. set it to stopped and check if your storage,
>> >> >> both primary and secondary, are reachable botjh from the ms and the
>> >> >> hosts.
>> >> >>
>> >> >> Daan (double a!)
>> >> >>
>> >> >> On Fri, Nov 1, 2013 at 7:52 PM, Carlos Reategui <
>> creategui@gmail.com>
>> >> >> wrote:
>> >> >> > Hi Dan,
>> >> >> > The host value is correct and I am able to connect on ports 22,80
>> and
>> >> >> 8080
>> >> >> > from the Xen hosts to that IP (same subnet). Port 8787 is not
>> >> listening
>> >> >> on
>> >> >> > my MS.
>> >> >> >
>> >> >> > What can I look for in the DB?
>> >> >> >
>> >> >> > thanks
>> >> >> > Carlos
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Fri, Nov 1, 2013 at 2:47 AM, Daan Hoogland <
>> >> daan.hoogland@gmail.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> H Carlox,
>> >> >> >>
>> >> >> >> It still seems to me you have a communication problem albeit not
>> at
>> >> >> >> the network level. ( I agree this doesn't seem likely) But even
>> so
>> >> the
>> >> >> >> ms can't reach the xen hosts. The next thing to look at would be
>> >> >> >> configuration. I presume you did check the ip addresses for the
>> >> hosts.
>> >> >> >> check the value 'host' in the global vars and try to telnet it on
>> >> port
>> >> >> >> 22 80 8080 8787 or what else from the xen hosts . I am sure the
>> >> >> >> problem is hidden somewhere in the mysql database.
>> >> >> >>
>> >> >> >> regards,
>> >> >> >> Daan
>> >> >> >>
>> >> >> >> On Fri, Nov 1, 2013 at 6:23 AM, Carlos Reategui <
>> carlos@reategui.com
>> >> >
>> >> >> >> wrote:
>> >> >> >> > Hi Dan,
>> >> >> >> > Finally getting around to trying this. I am not able to start
>> any
>> >> vm
>> >> >> >> > (mine
>> >> >> >> > or system).
>> >> >> >> >
>> >> >> >> > I am in the process of trying Ahmad's suggestion of removing
>> one of
>> >> >> the
>> >> >> >> > hosts, reinstalling it and then adding it back in to see if
>> that
>> >> >> helps.
>> >> >> >> > However I am not able to put it in maintenance mode. I am
>> getting
>> >> the
>> >> >> >> > following when I try:
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > 2013-10-31 15:19:59,120 DEBUG [cloud.api.ApiServlet]
>> >> >> >> > (catalina-exec-5:null)
>> >> >> >> > ===START=== 172.30.40.135 -- GET command=prepareHostForMa
>> >> >> >> >
>> >> >> >> >
>> >> >>
>> >>
>> intenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
>> >> >> >> > 2013-10-31 15:19:59,181 DEBUG [cloud.async.AsyncJobManagerImpl]
>> >> >> >> > (catalina-exec-5:null) submit async job-167, details:
>> AsyncJobVO
>> >> >> {id:16
>> >> >> >> > 7, userId: 2, accountId: 2, sessionKey: null, instanceType:
>> Host,
>> >> >> >> > instanceId: 1, cmd:
>> >> org.apache.cloudstack.api.command.admin.host.Prep
>> >> >> >> > areForMaintenanceCmd, cmdOriginator: null, cmdInfo:
>> >> >> >> >
>> >> >> >> >
>> >> >>
>> >>
>> {"id":"c2f3c416-f4ba-4f73-b414-6fd6efc734e3","response":"json","sessionkey":"Yjye5q
>> >> >> >> >
>> >> >> >> >
>> >> >>
>> >>
>> mcLtUBW0/P4XCAcx2ZTAI\u003d","ctxUserId":"2","_":"1383258001952","ctxAccountId":"2","ctxStartEventId":"699"},
>> >> >> >> > cmdVersion: 0, callbackTy
>> >> >> >> > pe: 0, callbackAddress: null, status: 0, processStatus: 0,
>> >> resultCode:
>> >> >> >> > 0,
>> >> >> >> > result: null, initMsid: 233845174730253, completeMsid: null,
>> >> >> >> > lastUpdated: null, lastPolled: null, created: null}
>> >> >> >> > 2013-10-31 15:19:59,185 DEBUG [cloud.async.AsyncJobManagerImpl]
>> >> >> >> > (Job-Executor-1:job-167) Executing
>> >> >> org.apache.cloudstack.api.command.ad
>> >> >> >> > min.host.PrepareForMaintenanceCmd for job-167
>> >> >> >> > 2013-10-31 15:19:59,188 DEBUG [cloud.api.ApiServlet]
>> >> >> >> > (catalina-exec-5:null)
>> >> >> >> > ===END=== 172.30.40.135 -- GET command=prepareHostForMain
>> >> >> >> >
>> >> >> >> >
>> >> >>
>> >>
>> tenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
>> >> >> >> > 2013-10-31 15:19:59,207 DEBUG
>> [cloud.cluster.ClusterManagerImpl]
>> >> >> >> > (Job-Executor-1:job-167) Propagating agent change request
>> >> event:AdminA
>> >> >> >> > skMaintenace to agent:1
>> >> >> >> > 2013-10-31 15:19:59,208 DEBUG
>> [cloud.cluster.ClusterManagerImpl]
>> >> >> >> > (Job-Executor-1:job-167) 233845174730253 -> 233845174730255.1
>> >> [{"Propa
>> >> >> >> >
>> >> >> >> >
>> >> >>
>> >>
>> gateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
>> >> >> >> > 2013-10-31 15:19:59,213 INFO
>> >> >> [cloud.cluster.ClusterServiceServletImpl]
>> >> >> >> > (Cluster-Worker-1:null) Setup cluster service servlet. service
>> >> >> >> > url: http://172.30.45.2:9090/clusterservice, request timeout:
>> 300
>> >> >> >> > seconds
>> >> >> >> > 2013-10-31 15:19:59,214 DEBUG
>> [cloud.cluster.ClusterManagerImpl]
>> >> >> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 ->
>> >> 233845174730255
>> >> >> >> > . agent: 1, pdu seq: 1, pdu ack seq: 0, json:
>> >> >> >> >
>> >> >> >> >
>> >> >>
>> >>
>> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{
>> >> >> >> > },"wait":0}}]
>> >> >> >> > 2013-10-31 15:19:59,343 DEBUG
>> [cloud.cluster.ClusterManagerImpl]
>> >> >> >> > (Cluster-Worker-7:null) Dispatch ->1, json:
>> >> [{"PropagateResourceEventC
>> >> >> >> >
>> >> >> >> >
>> >> >>
>> >>
>> ommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
>> >> >> >> > 2013-10-31 15:19:59,347 DEBUG
>> [cloud.cluster.ClusterManagerImpl]
>> >> >> >> > (Cluster-Worker-7:null) Intercepting command to propagate event
>> >> AdminA
>> >> >> >> > skMaintenace for host 1
>> >> >> >> > 2013-10-31 15:19:59,351 DEBUG
>> >> >> [cloud.cluster.ClusterServiceServletImpl]
>> >> >> >> > (Cluster-Worker-1:null) POST
>> http://172.30.45.2:9090/clusterser
>> >> >> >> > vice response :true, responding time: 80 ms
>> >> >> >> > 2013-10-31 15:19:59,351 DEBUG
>> [cloud.cluster.ClusterManagerImpl]
>> >> >> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 ->
>> >> 233845174730255
>> >> >> >> > completed. time: 137ms. agent: 1, pdu seq: 1, pdu ack seq: 0,
>> >> json:
>> >> >> >> >
>> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMai
>> >> >> >> > ntenace","contextMap":{},"wait":0}}]
>> >> >> >> > 2013-10-31 15:19:59,356 DEBUG
>> [agent.manager.ClusteredAgentAttache]
>> >> >> >> > (Cluster-Worker-7:null) Seq 1-102760483: Forwarding Seq
>> 1-102760483
>> >> >> >> > : { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> 100111,
>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> >> >> > 2013-10-31 15:19:59,360 DEBUG
>> [agent.manager.ClusteredAgentAttache]
>> >> >> >> > (AgentManager-Handler-1:null) Seq 1-102760483: Forwarding Seq
>> 1-102
>> >> >> >> > 760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>> Flags:
>> >> >> >> > 100111,
>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> >> >> > 2013-10-31 15:19:59,365 DEBUG
>> [agent.manager.ClusteredAgentAttache]
>> >> >> >> > (AgentManager-Handler-13:null) Seq 1-102760483: Forwarding Seq
>> 1-10
>> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>> Flags:
>> >> >> >> > 100111,
>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> >> >> > 2013-10-31 15:19:59,369 DEBUG
>> [agent.manager.ClusteredAgentAttache]
>> >> >> >> > (AgentManager-Handler-15:null) Seq 1-102760483: Forwarding Seq
>> 1-10
>> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>> Flags:
>> >> >> >> > 100111,
>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> >> >> > 2013-10-31 15:19:59,372 DEBUG
>> [agent.manager.ClusteredAgentAttache]
>> >> >> >> > (AgentManager-Handler-10:null) Seq 1-102760483: Forwarding Seq
>> 1-10
>> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>> Flags:
>> >> >> >> > 100111,
>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> >> >> > 2013-10-31 15:19:59,376 DEBUG
>> [agent.manager.ClusteredAgentAttache]
>> >> >> >> > (AgentManager-Handler-11:null) Seq 1-102760483: Forwarding Seq
>> 1-10
>> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>> Flags:
>> >> >> >> > 100111,
>> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> >> >> >
>> >> >> >> > These MaintainCommand lines are repeated endlessly and very
>> fast
>> >> (1GB
>> >> >> of
>> >> >> >> > logs in about 20 mins) so I currently have the MS stopped.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Thu, Oct 31, 2013 at 3:43 PM, Carlos Reategui <
>> >> carlos@reategui.com
>> >> >> >
>> >> >> >> > wrote:
>> >> >> >> >>
>> >> >> >> >> Hi Dan,
>> >> >> >> >> Network is definitely not a problem. iptables is disabled on
>> MS
>> >> and
>> >> >> >> >> both
>> >> >> >> >> hosts. I am able to ssh from MS to both hosts. I am also
>> able to
>> >> >> >> >> connect
>> >> >> >> >> to port 80 from the MS.
>> >> >> >> >>
>> >> >> >> >> I need to head out now and will try what you suggested in a
>> couple
>> >> >> >> >> hours.
>> >> >> >> >>
>> >> >> >> >> thanks,
>> >> >> >> >> Carlos
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland
>> >> >> >> >> <da...@gmail.com>
>> >> >> >> >> wrote:
>> >> >> >> >>>
>> >> >> >> >>> Carlos,
>> >> >> >> >>>
>> >> >> >> >>> can the ms reach the xs hosts?
>> >> >> >> >>> if so try this, Stop the ms, set the state for the
>> vm_instances
>> >> for
>> >> >> >> >>> cpvm and vr from expunging to stopped. and restart.
>> >> >> >> >>> if not you need to troubleshoot your network connectivity. It
>> >> seems
>> >> >> >> >>> this would not be the issue as you only stopped and started
>> it.
>> >> Did
>> >> >> >> >>> you
>> >> >> >> >>>
>> >> >> >> >>> Daan
>> >> >> >> >>>
>> >> >> >> >>> On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui
>> >> >> >> >>> <ca...@reategui.com>
>> >> >> >> >>> wrote:
>> >> >> >> >>> > Hi Dan,
>> >> >> >> >>> >
>> >> >> >> >>> > Unfortunately not. Still trying to solve this. See my
>> last
>> >> email
>> >> >> >> >>> > to
>> >> >> >> >>> > Ahmad
>> >> >> >> >>> > in the recover cs thread about next steps I am trying to
>> figure
>> >> >> out.
>> >> >> >> >>> >
>> >> >> >> >>> > The CPVM and VR are not running. They are currently in
>> >> expunging
>> >> >> >> >>> > state.
>> >> >> >> >>> >
>> >> >> >> >>> > If I try to start an instance the MS is unable to
>> communicate
>> >> the
>> >> >> >> >>> > start
>> >> >> >> >>> > vm
>> >> >> >> >>> > commands to my XS hosts and then starts writing to the
>> logs at
>> >> an
>> >> >> >> >>> > amazingly
>> >> >> >> >>> > high rate. You can see one of my logs here:
>> >> >> >> >>> > http://reategui.com/cloudstack/management-server.log.new
>> >> >> >> >>> >
>> >> >> >> >>> > thank you
>> >> >> >> >>> > Carlos
>> >> >> >> >>> >
>> >> >> >> >>> >
>> >> >> >> >>> >
>> >> >> >> >>> >
>> >> >> >> >>> >
>> >> >> >> >>> > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland
>> >> >> >> >>> > <da...@gmail.com>
>> >> >> >> >>> > wrote:
>> >> >> >> >>> >>
>> >> >> >> >>> >> H Carlos,
>> >> >> >> >>> >>
>> >> >> >> >>> >> Did you solve this problem yet?
>> >> >> >> >>> >>
>> >> >> >> >>> >> You can try removing the cpvm and vr at the hypervisor and
>> >> change
>> >> >> >> >>> >> their state to 'Stopped' in the db.
>> >> >> >> >>> >>
>> >> >> >> >>> >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui
>> >> >> >> >>> >> <ca...@reategui.com>
>> >> >> >> >>> >> wrote:
>> >> >> >> >>> >> > In the last 15 minutes management log file it grew
>> 500MB..
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > Any ideas what I should look at to figure out what is
>> >> >> happening?
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui
>> >> >> >> >>> >> > <cr...@gmail.com>wrote:
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >> I am trying to recover my CloudStack installation (CS
>> >> 4.1.1 on
>> >> >> >> >>> >> >> ubuntu +
>> >> >> >> >>> >> >> XS
>> >> >> >> >>> >> >> 6.0.2) that I shutdown and cannot bring back to life.
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >> I tried to destroy the CPVM and the VR and all I am
>> seeing
>> >> in
>> >> >> >> >>> >> >> the
>> >> >> >> >>> >> >> logs
>> >> >> >> >>> >> >> is
>> >> >> >> >>> >> >> the following and they are filling up fast (management
>> log
>> >> is
>> >> >> >> >>> >> >> almost
>> >> >> >> >>> >> >> 1GB).
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >> 2013-10-29 11:51:08,291 DEBUG
>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >> >>> >> >> (AgentManager-Handler-14:null) Seq 1-201919186:
>> Forwarding
>> >> Seq
>> >> >> >> >>> >> >> 1-201919186:
>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>> Flags:
>> >> >> >> >>> >> >> 100111,
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >> >>> >> >> } to 233845174730255
>> >> >> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >> >>> >> >> (AgentManager-Handler-4:null) Seq 2-1168706613:
>> Forwarding
>> >> Seq
>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
>> >> Ver:
>> >> >> v1,
>> >> >> >> >>> >> >> Flags:
>> >> >> >> >>> >> >> 100111,
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >> >>> >> >> } to 233845174730255
>> >> >> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >> >>> >> >> (AgentManager-Handler-2:null) Seq 1-201919186:
>> Forwarding
>> >> Seq
>> >> >> >> >>> >> >> 1-201919186:
>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>> Flags:
>> >> >> >> >>> >> >> 100111,
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >> >>> >> >> } to 233845174730255
>> >> >> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >> >>> >> >> (AgentManager-Handler-8:null) Seq 2-1168706613:
>> Forwarding
>> >> Seq
>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
>> >> Ver:
>> >> >> v1,
>> >> >> >> >>> >> >> Flags:
>> >> >> >> >>> >> >> 100111,
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >> >>> >> >> } to 233845174730255
>> >> >> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >> >>> >> >> (AgentManager-Handler-9:null) Seq 1-201919186:
>> Forwarding
>> >> Seq
>> >> >> >> >>> >> >> 1-201919186:
>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>> Flags:
>> >> >> >> >>> >> >> 100111,
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >> >>> >> >> } to 233845174730255
>> >> >> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >> >>> >> >> (AgentManager-Handler-6:null) Seq 2-1168706613:
>> Forwarding
>> >> Seq
>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
>> >> Ver:
>> >> >> v1,
>> >> >> >> >>> >> >> Flags:
>> >> >> >> >>> >> >> 100111,
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >> >>> >> >> } to 233845174730255
>> >> >> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >> >>> >> >> (AgentManager-Handler-15:null) Seq 1-201919186:
>> Forwarding
>> >> Seq
>> >> >> >> >>> >> >> 1-201919186:
>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>> Flags:
>> >> >> >> >>> >> >> 100111,
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >> >>> >> >> } to 233845174730255
>> >> >> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >> >>> >> >> (AgentManager-Handler-13:null) Seq 2-1168706613:
>> Forwarding
>> >> >> Seq
>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
>> >> Ver:
>> >> >> v1,
>> >> >> >> >>> >> >> Flags:
>> >> >> >> >>> >> >> 100111,
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >> >>> >> >> } to 233845174730255
>> >> >> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >> >>> >> >> (AgentManager-Handler-10:null) Seq 1-201919186:
>> Forwarding
>> >> Seq
>> >> >> >> >>> >> >> 1-201919186:
>> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
>> Flags:
>> >> >> >> >>> >> >> 100111,
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >> >>> >> >> } to 233845174730255
>> >> >> >> >>> >> >> 2013-10-29 11:51:08,299 DEBUG
>> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >> >>> >> >> (AgentManager-Handler-3:null) Seq 2-1168706613:
>> Forwarding
>> >> Seq
>> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
>> >> Ver:
>> >> >> v1,
>> >> >> >> >>> >> >> Flags:
>> >> >> >> >>> >> >> 100111,
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >> >>> >> >> } to 233845174730255
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >>
>> >> >> >> >>> >
>> >> >> >> >>> >
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >
>> >> >> >
>> >> >> >
>> >> >>
>> >>
>>
>
>
Re: Can't destroy system VMs
Posted by Carlos Reategui <ca...@reategui.com>.
Hi Daan,
I have been looking through my logs and can't figure out why the ssvm does
not want to start.
Here is a link to my latest logs:
http://reategui.com/cloudstack/management-server-131105.log
Maybe someone else can spot something.
There is nothing on either the XenServer hosts logs.
thanks,
Carlos
On Tue, Nov 5, 2013 at 12:55 AM, Daan Hoogland <da...@gmail.com>wrote:
> Carlos,
>
> It makes no sense trying to start the vms if not the ssvm is started first.
>
> I would say stop the system, clear the logs and start again. Then
> analyse the logs to see what is wrong with the secondary storage vm.
>
> On Tue, Nov 5, 2013 at 2:08 AM, Carlos Reategui <ca...@reategui.com>
> wrote:
> > Hi Daan,
> > There is no trace of CS communicating with either of my XS hosts in the
> > SMLog file or the xensource.log files when I try to start one of my
> stopped
> > instances.
> >
> > thanks,
> > Carlos
> >
> >
> >
> > On Sat, Nov 2, 2013 at 4:50 AM, Daan Hoogland <daan.hoogland@gmail.com
> >wrote:
> >
> >> H Carlos,
> >>
> >> I wouldn't know about the mshost and mshost_peer but they seem
> >> alright. The expunging ssvm are a real problem. I think Ahmed
> >> mentioned that as well. You are having problems getting your ssvm in
> >> the air. All other things are consequential. This is the first thing
> >> to look at. Check the logs and the SMLog on the hypervisor to see what
> >> happens when cs tries to start a ssvm. By your db you can see that it
> >> tried to start it several times. Those are the expunging vm_instance
> >> entries.
> >>
> >> regards,
> >> Daan
> >>
> >> On Fri, Nov 1, 2013 at 9:10 PM, Carlos Reategui <ca...@reategui.com>
> >> wrote:
> >> > Hi Daan :)
> >> >
> >> > Primary and Secondary storage are on the MS. Primary is on a
> different
> >> > subnet and it is currently mounted (and visible as an NFS SR from
> >> > XenCenter) and working fine from the xen machines. I have also been
> able
> >> > to mount Secondary without a problem.
> >> >
> >> > Here are my current system VMs:
> >> > mysql> select
> >> > id,name,instance_name,state,host_id,type,vm_type,desired_state from
> >> > vm_instance where type!='User';
> >> >
> >>
> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
> >> > | id | name | instance_name | state | host_id | type
> >> |
> >> > vm_type | desired_state |
> >> >
> >>
> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
> >> > | 2 | v-2-VM | v-2-VM | Stopped | 1 | ConsoleProxy
> >> |
> >> > ConsoleProxy | NULL |
> >> > | 5 | r-5-VM | r-5-VM | Stopped | 2 | DomainRouter
> >> |
> >> > DomainRouter | NULL |
> >> > | 1 | s-1-VM | s-1-VM | Expunging | NULL |
> >> SecondaryStorageVm |
> >> > SecondaryStorageVm | NULL |
> >> > | 3 | s-3-VM | s-3-VM | Expunging | NULL |
> >> SecondaryStorageVm |
> >> > SecondaryStorageVm | NULL |
> >> > | 52 | s-52-VM | s-52-VM | Expunging | NULL |
> >> SecondaryStorageVm |
> >> > SecondaryStorageVm | NULL |
> >> > | 53 | s-53-VM | s-53-VM | Expunging | NULL |
> >> SecondaryStorageVm |
> >> > SecondaryStorageVm | NULL |
> >> > | 54 | s-54-VM | s-54-VM | Expunging | NULL |
> >> SecondaryStorageVm |
> >> > SecondaryStorageVm | NULL |
> >> > | 55 | s-55-VM | s-55-VM | Expunging | NULL |
> >> SecondaryStorageVm |
> >> > SecondaryStorageVm | NULL |
> >> > | 56 | s-56-VM | s-56-VM | Expunging | NULL |
> >> SecondaryStorageVm |
> >> > SecondaryStorageVm | NULL |
> >> >
> >>
> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
> >> >
> >> >
> >> > I have been looking through the DB to see if anything sticks out. I
> >> notice
> >> > that the mshost and mshost_peer table have 2 entries but they are the
> >> same
> >> > server. Is that ok? Looks like one is from the install of 4.1.0 an
> the
> >> > other from after I upgraded to 4.1.1:
> >> >
> >> > mysql> select * from mshost;
> >> >
> >>
> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
> >> > | id | msid | runid | name | state | version
> |
> >> > service_ip | service_port | last_update | removed |
> alert_count
> >> |
> >> >
> >>
> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
> >> > | 1 | 233845174730255 | 1375383759777 | isbustor01 | Up | 4.1.0
> |
> >> > 172.30.45.2 | 9090 | 2013-10-24 21:13:10 | NULL |
> 0
> >> |
> >> > | 2 | 233845174730253 | 1383331508626 | isbustor01 | Up | 4.1.1
> |
> >> > 172.30.45.2 | 9090 | 2013-11-01 20:00:04 | NULL |
> 0
> >> |
> >> >
> >>
> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
> >> > 2 rows in set (0.00 sec)
> >> >
> >> > mysql> select * from mshost_peer;
> >> >
> >>
> +----+--------------+-------------+---------------+------------+---------------------+
> >> > | id | owner_mshost | peer_mshost | peer_runid | peer_state |
> >> > last_update |
> >> >
> >>
> +----+--------------+-------------+---------------+------------+---------------------+
> >> > | 17 | 1 | 1 | 1375383759777 | Up |
> >> 2013-08-01
> >> > 19:02:49 |
> >> > | 34 | 2 | 2 | 1383331508626 | Up |
> >> 2013-11-01
> >> > 18:45:19 |
> >> >
> >>
> +----+--------------+-------------+---------------+------------+---------------------+
> >> >
> >> >
> >> > thanks,
> >> > Carlos
> >> >
> >> >
> >> >
> >> > On Fri, Nov 1, 2013 at 12:17 PM, Daan Hoogland <
> daan.hoogland@gmail.com
> >> >wrote:
> >> >
> >> >> h Carlos,
> >> >>
> >> >> the states of virtual machines, especially the entries for the ssvm.
> >> >> if it is not Stopped. set it to stopped and check if your storage,
> >> >> both primary and secondary, are reachable botjh from the ms and the
> >> >> hosts.
> >> >>
> >> >> Daan (double a!)
> >> >>
> >> >> On Fri, Nov 1, 2013 at 7:52 PM, Carlos Reategui <creategui@gmail.com
> >
> >> >> wrote:
> >> >> > Hi Dan,
> >> >> > The host value is correct and I am able to connect on ports 22,80
> and
> >> >> 8080
> >> >> > from the Xen hosts to that IP (same subnet). Port 8787 is not
> >> listening
> >> >> on
> >> >> > my MS.
> >> >> >
> >> >> > What can I look for in the DB?
> >> >> >
> >> >> > thanks
> >> >> > Carlos
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Fri, Nov 1, 2013 at 2:47 AM, Daan Hoogland <
> >> daan.hoogland@gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> H Carlox,
> >> >> >>
> >> >> >> It still seems to me you have a communication problem albeit not
> at
> >> >> >> the network level. ( I agree this doesn't seem likely) But even so
> >> the
> >> >> >> ms can't reach the xen hosts. The next thing to look at would be
> >> >> >> configuration. I presume you did check the ip addresses for the
> >> hosts.
> >> >> >> check the value 'host' in the global vars and try to telnet it on
> >> port
> >> >> >> 22 80 8080 8787 or what else from the xen hosts . I am sure the
> >> >> >> problem is hidden somewhere in the mysql database.
> >> >> >>
> >> >> >> regards,
> >> >> >> Daan
> >> >> >>
> >> >> >> On Fri, Nov 1, 2013 at 6:23 AM, Carlos Reategui <
> carlos@reategui.com
> >> >
> >> >> >> wrote:
> >> >> >> > Hi Dan,
> >> >> >> > Finally getting around to trying this. I am not able to start
> any
> >> vm
> >> >> >> > (mine
> >> >> >> > or system).
> >> >> >> >
> >> >> >> > I am in the process of trying Ahmad's suggestion of removing
> one of
> >> >> the
> >> >> >> > hosts, reinstalling it and then adding it back in to see if that
> >> >> helps.
> >> >> >> > However I am not able to put it in maintenance mode. I am
> getting
> >> the
> >> >> >> > following when I try:
> >> >> >> >
> >> >> >> >
> >> >> >> > 2013-10-31 15:19:59,120 DEBUG [cloud.api.ApiServlet]
> >> >> >> > (catalina-exec-5:null)
> >> >> >> > ===START=== 172.30.40.135 -- GET command=prepareHostForMa
> >> >> >> >
> >> >> >> >
> >> >>
> >>
> intenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
> >> >> >> > 2013-10-31 15:19:59,181 DEBUG [cloud.async.AsyncJobManagerImpl]
> >> >> >> > (catalina-exec-5:null) submit async job-167, details: AsyncJobVO
> >> >> {id:16
> >> >> >> > 7, userId: 2, accountId: 2, sessionKey: null, instanceType:
> Host,
> >> >> >> > instanceId: 1, cmd:
> >> org.apache.cloudstack.api.command.admin.host.Prep
> >> >> >> > areForMaintenanceCmd, cmdOriginator: null, cmdInfo:
> >> >> >> >
> >> >> >> >
> >> >>
> >>
> {"id":"c2f3c416-f4ba-4f73-b414-6fd6efc734e3","response":"json","sessionkey":"Yjye5q
> >> >> >> >
> >> >> >> >
> >> >>
> >>
> mcLtUBW0/P4XCAcx2ZTAI\u003d","ctxUserId":"2","_":"1383258001952","ctxAccountId":"2","ctxStartEventId":"699"},
> >> >> >> > cmdVersion: 0, callbackTy
> >> >> >> > pe: 0, callbackAddress: null, status: 0, processStatus: 0,
> >> resultCode:
> >> >> >> > 0,
> >> >> >> > result: null, initMsid: 233845174730253, completeMsid: null,
> >> >> >> > lastUpdated: null, lastPolled: null, created: null}
> >> >> >> > 2013-10-31 15:19:59,185 DEBUG [cloud.async.AsyncJobManagerImpl]
> >> >> >> > (Job-Executor-1:job-167) Executing
> >> >> org.apache.cloudstack.api.command.ad
> >> >> >> > min.host.PrepareForMaintenanceCmd for job-167
> >> >> >> > 2013-10-31 15:19:59,188 DEBUG [cloud.api.ApiServlet]
> >> >> >> > (catalina-exec-5:null)
> >> >> >> > ===END=== 172.30.40.135 -- GET command=prepareHostForMain
> >> >> >> >
> >> >> >> >
> >> >>
> >>
> tenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
> >> >> >> > 2013-10-31 15:19:59,207 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> >> >> > (Job-Executor-1:job-167) Propagating agent change request
> >> event:AdminA
> >> >> >> > skMaintenace to agent:1
> >> >> >> > 2013-10-31 15:19:59,208 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> >> >> > (Job-Executor-1:job-167) 233845174730253 -> 233845174730255.1
> >> [{"Propa
> >> >> >> >
> >> >> >> >
> >> >>
> >>
> gateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
> >> >> >> > 2013-10-31 15:19:59,213 INFO
> >> >> [cloud.cluster.ClusterServiceServletImpl]
> >> >> >> > (Cluster-Worker-1:null) Setup cluster service servlet. service
> >> >> >> > url: http://172.30.45.2:9090/clusterservice, request timeout:
> 300
> >> >> >> > seconds
> >> >> >> > 2013-10-31 15:19:59,214 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> >> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 ->
> >> 233845174730255
> >> >> >> > . agent: 1, pdu seq: 1, pdu ack seq: 0, json:
> >> >> >> >
> >> >> >> >
> >> >>
> >>
> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{
> >> >> >> > },"wait":0}}]
> >> >> >> > 2013-10-31 15:19:59,343 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> >> >> > (Cluster-Worker-7:null) Dispatch ->1, json:
> >> [{"PropagateResourceEventC
> >> >> >> >
> >> >> >> >
> >> >>
> >>
> ommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
> >> >> >> > 2013-10-31 15:19:59,347 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> >> >> > (Cluster-Worker-7:null) Intercepting command to propagate event
> >> AdminA
> >> >> >> > skMaintenace for host 1
> >> >> >> > 2013-10-31 15:19:59,351 DEBUG
> >> >> [cloud.cluster.ClusterServiceServletImpl]
> >> >> >> > (Cluster-Worker-1:null) POST http://172.30.45.2:9090/clusterser
> >> >> >> > vice response :true, responding time: 80 ms
> >> >> >> > 2013-10-31 15:19:59,351 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> >> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 ->
> >> 233845174730255
> >> >> >> > completed. time: 137ms. agent: 1, pdu seq: 1, pdu ack seq: 0,
> >> json:
> >> >> >> >
> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMai
> >> >> >> > ntenace","contextMap":{},"wait":0}}]
> >> >> >> > 2013-10-31 15:19:59,356 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >> >> >> > (Cluster-Worker-7:null) Seq 1-102760483: Forwarding Seq
> 1-102760483
> >> >> >> > : { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> 100111,
> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> >> >> > 2013-10-31 15:19:59,360 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >> >> >> > (AgentManager-Handler-1:null) Seq 1-102760483: Forwarding Seq
> 1-102
> >> >> >> > 760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
> Flags:
> >> >> >> > 100111,
> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> >> >> > 2013-10-31 15:19:59,365 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >> >> >> > (AgentManager-Handler-13:null) Seq 1-102760483: Forwarding Seq
> 1-10
> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
> Flags:
> >> >> >> > 100111,
> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> >> >> > 2013-10-31 15:19:59,369 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >> >> >> > (AgentManager-Handler-15:null) Seq 1-102760483: Forwarding Seq
> 1-10
> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
> Flags:
> >> >> >> > 100111,
> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> >> >> > 2013-10-31 15:19:59,372 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >> >> >> > (AgentManager-Handler-10:null) Seq 1-102760483: Forwarding Seq
> 1-10
> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
> Flags:
> >> >> >> > 100111,
> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> >> >> > 2013-10-31 15:19:59,376 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >> >> >> > (AgentManager-Handler-11:null) Seq 1-102760483: Forwarding Seq
> 1-10
> >> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
> Flags:
> >> >> >> > 100111,
> >> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> >> >> >
> >> >> >> > These MaintainCommand lines are repeated endlessly and very fast
> >> (1GB
> >> >> of
> >> >> >> > logs in about 20 mins) so I currently have the MS stopped.
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On Thu, Oct 31, 2013 at 3:43 PM, Carlos Reategui <
> >> carlos@reategui.com
> >> >> >
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> Hi Dan,
> >> >> >> >> Network is definitely not a problem. iptables is disabled on
> MS
> >> and
> >> >> >> >> both
> >> >> >> >> hosts. I am able to ssh from MS to both hosts. I am also
> able to
> >> >> >> >> connect
> >> >> >> >> to port 80 from the MS.
> >> >> >> >>
> >> >> >> >> I need to head out now and will try what you suggested in a
> couple
> >> >> >> >> hours.
> >> >> >> >>
> >> >> >> >> thanks,
> >> >> >> >> Carlos
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland
> >> >> >> >> <da...@gmail.com>
> >> >> >> >> wrote:
> >> >> >> >>>
> >> >> >> >>> Carlos,
> >> >> >> >>>
> >> >> >> >>> can the ms reach the xs hosts?
> >> >> >> >>> if so try this, Stop the ms, set the state for the
> vm_instances
> >> for
> >> >> >> >>> cpvm and vr from expunging to stopped. and restart.
> >> >> >> >>> if not you need to troubleshoot your network connectivity. It
> >> seems
> >> >> >> >>> this would not be the issue as you only stopped and started
> it.
> >> Did
> >> >> >> >>> you
> >> >> >> >>>
> >> >> >> >>> Daan
> >> >> >> >>>
> >> >> >> >>> On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui
> >> >> >> >>> <ca...@reategui.com>
> >> >> >> >>> wrote:
> >> >> >> >>> > Hi Dan,
> >> >> >> >>> >
> >> >> >> >>> > Unfortunately not. Still trying to solve this. See my last
> >> email
> >> >> >> >>> > to
> >> >> >> >>> > Ahmad
> >> >> >> >>> > in the recover cs thread about next steps I am trying to
> figure
> >> >> out.
> >> >> >> >>> >
> >> >> >> >>> > The CPVM and VR are not running. They are currently in
> >> expunging
> >> >> >> >>> > state.
> >> >> >> >>> >
> >> >> >> >>> > If I try to start an instance the MS is unable to
> communicate
> >> the
> >> >> >> >>> > start
> >> >> >> >>> > vm
> >> >> >> >>> > commands to my XS hosts and then starts writing to the logs
> at
> >> an
> >> >> >> >>> > amazingly
> >> >> >> >>> > high rate. You can see one of my logs here:
> >> >> >> >>> > http://reategui.com/cloudstack/management-server.log.new
> >> >> >> >>> >
> >> >> >> >>> > thank you
> >> >> >> >>> > Carlos
> >> >> >> >>> >
> >> >> >> >>> >
> >> >> >> >>> >
> >> >> >> >>> >
> >> >> >> >>> >
> >> >> >> >>> > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland
> >> >> >> >>> > <da...@gmail.com>
> >> >> >> >>> > wrote:
> >> >> >> >>> >>
> >> >> >> >>> >> H Carlos,
> >> >> >> >>> >>
> >> >> >> >>> >> Did you solve this problem yet?
> >> >> >> >>> >>
> >> >> >> >>> >> You can try removing the cpvm and vr at the hypervisor and
> >> change
> >> >> >> >>> >> their state to 'Stopped' in the db.
> >> >> >> >>> >>
> >> >> >> >>> >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui
> >> >> >> >>> >> <ca...@reategui.com>
> >> >> >> >>> >> wrote:
> >> >> >> >>> >> > In the last 15 minutes management log file it grew
> 500MB..
> >> >> >> >>> >> >
> >> >> >> >>> >> > Any ideas what I should look at to figure out what is
> >> >> happening?
> >> >> >> >>> >> >
> >> >> >> >>> >> >
> >> >> >> >>> >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui
> >> >> >> >>> >> > <cr...@gmail.com>wrote:
> >> >> >> >>> >> >
> >> >> >> >>> >> >> I am trying to recover my CloudStack installation (CS
> >> 4.1.1 on
> >> >> >> >>> >> >> ubuntu +
> >> >> >> >>> >> >> XS
> >> >> >> >>> >> >> 6.0.2) that I shutdown and cannot bring back to life.
> >> >> >> >>> >> >>
> >> >> >> >>> >> >> I tried to destroy the CPVM and the VR and all I am
> seeing
> >> in
> >> >> >> >>> >> >> the
> >> >> >> >>> >> >> logs
> >> >> >> >>> >> >> is
> >> >> >> >>> >> >> the following and they are filling up fast (management
> log
> >> is
> >> >> >> >>> >> >> almost
> >> >> >> >>> >> >> 1GB).
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >> 2013-10-29 11:51:08,291 DEBUG
> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >> >>> >> >> (AgentManager-Handler-14:null) Seq 1-201919186:
> Forwarding
> >> Seq
> >> >> >> >>> >> >> 1-201919186:
> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
> Flags:
> >> >> >> >>> >> >> 100111,
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >> >>> >> >> } to 233845174730255
> >> >> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >> >>> >> >> (AgentManager-Handler-4:null) Seq 2-1168706613:
> Forwarding
> >> Seq
> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
> >> Ver:
> >> >> v1,
> >> >> >> >>> >> >> Flags:
> >> >> >> >>> >> >> 100111,
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >> >>> >> >> } to 233845174730255
> >> >> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >> >>> >> >> (AgentManager-Handler-2:null) Seq 1-201919186:
> Forwarding
> >> Seq
> >> >> >> >>> >> >> 1-201919186:
> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
> Flags:
> >> >> >> >>> >> >> 100111,
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >> >>> >> >> } to 233845174730255
> >> >> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >> >>> >> >> (AgentManager-Handler-8:null) Seq 2-1168706613:
> Forwarding
> >> Seq
> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
> >> Ver:
> >> >> v1,
> >> >> >> >>> >> >> Flags:
> >> >> >> >>> >> >> 100111,
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >> >>> >> >> } to 233845174730255
> >> >> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >> >>> >> >> (AgentManager-Handler-9:null) Seq 1-201919186:
> Forwarding
> >> Seq
> >> >> >> >>> >> >> 1-201919186:
> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
> Flags:
> >> >> >> >>> >> >> 100111,
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >> >>> >> >> } to 233845174730255
> >> >> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >> >>> >> >> (AgentManager-Handler-6:null) Seq 2-1168706613:
> Forwarding
> >> Seq
> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
> >> Ver:
> >> >> v1,
> >> >> >> >>> >> >> Flags:
> >> >> >> >>> >> >> 100111,
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >> >>> >> >> } to 233845174730255
> >> >> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >> >>> >> >> (AgentManager-Handler-15:null) Seq 1-201919186:
> Forwarding
> >> Seq
> >> >> >> >>> >> >> 1-201919186:
> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
> Flags:
> >> >> >> >>> >> >> 100111,
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >> >>> >> >> } to 233845174730255
> >> >> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >> >>> >> >> (AgentManager-Handler-13:null) Seq 2-1168706613:
> Forwarding
> >> >> Seq
> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
> >> Ver:
> >> >> v1,
> >> >> >> >>> >> >> Flags:
> >> >> >> >>> >> >> 100111,
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >> >>> >> >> } to 233845174730255
> >> >> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >> >>> >> >> (AgentManager-Handler-10:null) Seq 1-201919186:
> Forwarding
> >> Seq
> >> >> >> >>> >> >> 1-201919186:
> >> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1,
> Flags:
> >> >> >> >>> >> >> 100111,
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >> >>> >> >> } to 233845174730255
> >> >> >> >>> >> >> 2013-10-29 11:51:08,299 DEBUG
> >> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >> >>> >> >> (AgentManager-Handler-3:null) Seq 2-1168706613:
> Forwarding
> >> Seq
> >> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
> >> Ver:
> >> >> v1,
> >> >> >> >>> >> >> Flags:
> >> >> >> >>> >> >> 100111,
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >> >>> >> >> } to 233845174730255
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >> >>
> >> >> >> >>> >
> >> >> >> >>> >
> >> >> >> >>
> >> >> >> >>
> >> >> >> >
> >> >> >
> >> >> >
> >> >>
> >>
>
Re: Can't destroy system VMs
Posted by Daan Hoogland <da...@gmail.com>.
Carlos,
It makes no sense trying to start the vms if not the ssvm is started first.
I would say stop the system, clear the logs and start again. Then
analyse the logs to see what is wrong with the secondary storage vm.
On Tue, Nov 5, 2013 at 2:08 AM, Carlos Reategui <ca...@reategui.com> wrote:
> Hi Daan,
> There is no trace of CS communicating with either of my XS hosts in the
> SMLog file or the xensource.log files when I try to start one of my stopped
> instances.
>
> thanks,
> Carlos
>
>
>
> On Sat, Nov 2, 2013 at 4:50 AM, Daan Hoogland <da...@gmail.com>wrote:
>
>> H Carlos,
>>
>> I wouldn't know about the mshost and mshost_peer but they seem
>> alright. The expunging ssvm are a real problem. I think Ahmed
>> mentioned that as well. You are having problems getting your ssvm in
>> the air. All other things are consequential. This is the first thing
>> to look at. Check the logs and the SMLog on the hypervisor to see what
>> happens when cs tries to start a ssvm. By your db you can see that it
>> tried to start it several times. Those are the expunging vm_instance
>> entries.
>>
>> regards,
>> Daan
>>
>> On Fri, Nov 1, 2013 at 9:10 PM, Carlos Reategui <ca...@reategui.com>
>> wrote:
>> > Hi Daan :)
>> >
>> > Primary and Secondary storage are on the MS. Primary is on a different
>> > subnet and it is currently mounted (and visible as an NFS SR from
>> > XenCenter) and working fine from the xen machines. I have also been able
>> > to mount Secondary without a problem.
>> >
>> > Here are my current system VMs:
>> > mysql> select
>> > id,name,instance_name,state,host_id,type,vm_type,desired_state from
>> > vm_instance where type!='User';
>> >
>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
>> > | id | name | instance_name | state | host_id | type
>> |
>> > vm_type | desired_state |
>> >
>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
>> > | 2 | v-2-VM | v-2-VM | Stopped | 1 | ConsoleProxy
>> |
>> > ConsoleProxy | NULL |
>> > | 5 | r-5-VM | r-5-VM | Stopped | 2 | DomainRouter
>> |
>> > DomainRouter | NULL |
>> > | 1 | s-1-VM | s-1-VM | Expunging | NULL |
>> SecondaryStorageVm |
>> > SecondaryStorageVm | NULL |
>> > | 3 | s-3-VM | s-3-VM | Expunging | NULL |
>> SecondaryStorageVm |
>> > SecondaryStorageVm | NULL |
>> > | 52 | s-52-VM | s-52-VM | Expunging | NULL |
>> SecondaryStorageVm |
>> > SecondaryStorageVm | NULL |
>> > | 53 | s-53-VM | s-53-VM | Expunging | NULL |
>> SecondaryStorageVm |
>> > SecondaryStorageVm | NULL |
>> > | 54 | s-54-VM | s-54-VM | Expunging | NULL |
>> SecondaryStorageVm |
>> > SecondaryStorageVm | NULL |
>> > | 55 | s-55-VM | s-55-VM | Expunging | NULL |
>> SecondaryStorageVm |
>> > SecondaryStorageVm | NULL |
>> > | 56 | s-56-VM | s-56-VM | Expunging | NULL |
>> SecondaryStorageVm |
>> > SecondaryStorageVm | NULL |
>> >
>> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
>> >
>> >
>> > I have been looking through the DB to see if anything sticks out. I
>> notice
>> > that the mshost and mshost_peer table have 2 entries but they are the
>> same
>> > server. Is that ok? Looks like one is from the install of 4.1.0 an the
>> > other from after I upgraded to 4.1.1:
>> >
>> > mysql> select * from mshost;
>> >
>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
>> > | id | msid | runid | name | state | version |
>> > service_ip | service_port | last_update | removed | alert_count
>> |
>> >
>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
>> > | 1 | 233845174730255 | 1375383759777 | isbustor01 | Up | 4.1.0 |
>> > 172.30.45.2 | 9090 | 2013-10-24 21:13:10 | NULL | 0
>> |
>> > | 2 | 233845174730253 | 1383331508626 | isbustor01 | Up | 4.1.1 |
>> > 172.30.45.2 | 9090 | 2013-11-01 20:00:04 | NULL | 0
>> |
>> >
>> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
>> > 2 rows in set (0.00 sec)
>> >
>> > mysql> select * from mshost_peer;
>> >
>> +----+--------------+-------------+---------------+------------+---------------------+
>> > | id | owner_mshost | peer_mshost | peer_runid | peer_state |
>> > last_update |
>> >
>> +----+--------------+-------------+---------------+------------+---------------------+
>> > | 17 | 1 | 1 | 1375383759777 | Up |
>> 2013-08-01
>> > 19:02:49 |
>> > | 34 | 2 | 2 | 1383331508626 | Up |
>> 2013-11-01
>> > 18:45:19 |
>> >
>> +----+--------------+-------------+---------------+------------+---------------------+
>> >
>> >
>> > thanks,
>> > Carlos
>> >
>> >
>> >
>> > On Fri, Nov 1, 2013 at 12:17 PM, Daan Hoogland <daan.hoogland@gmail.com
>> >wrote:
>> >
>> >> h Carlos,
>> >>
>> >> the states of virtual machines, especially the entries for the ssvm.
>> >> if it is not Stopped. set it to stopped and check if your storage,
>> >> both primary and secondary, are reachable botjh from the ms and the
>> >> hosts.
>> >>
>> >> Daan (double a!)
>> >>
>> >> On Fri, Nov 1, 2013 at 7:52 PM, Carlos Reategui <cr...@gmail.com>
>> >> wrote:
>> >> > Hi Dan,
>> >> > The host value is correct and I am able to connect on ports 22,80 and
>> >> 8080
>> >> > from the Xen hosts to that IP (same subnet). Port 8787 is not
>> listening
>> >> on
>> >> > my MS.
>> >> >
>> >> > What can I look for in the DB?
>> >> >
>> >> > thanks
>> >> > Carlos
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Fri, Nov 1, 2013 at 2:47 AM, Daan Hoogland <
>> daan.hoogland@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> H Carlox,
>> >> >>
>> >> >> It still seems to me you have a communication problem albeit not at
>> >> >> the network level. ( I agree this doesn't seem likely) But even so
>> the
>> >> >> ms can't reach the xen hosts. The next thing to look at would be
>> >> >> configuration. I presume you did check the ip addresses for the
>> hosts.
>> >> >> check the value 'host' in the global vars and try to telnet it on
>> port
>> >> >> 22 80 8080 8787 or what else from the xen hosts . I am sure the
>> >> >> problem is hidden somewhere in the mysql database.
>> >> >>
>> >> >> regards,
>> >> >> Daan
>> >> >>
>> >> >> On Fri, Nov 1, 2013 at 6:23 AM, Carlos Reategui <carlos@reategui.com
>> >
>> >> >> wrote:
>> >> >> > Hi Dan,
>> >> >> > Finally getting around to trying this. I am not able to start any
>> vm
>> >> >> > (mine
>> >> >> > or system).
>> >> >> >
>> >> >> > I am in the process of trying Ahmad's suggestion of removing one of
>> >> the
>> >> >> > hosts, reinstalling it and then adding it back in to see if that
>> >> helps.
>> >> >> > However I am not able to put it in maintenance mode. I am getting
>> the
>> >> >> > following when I try:
>> >> >> >
>> >> >> >
>> >> >> > 2013-10-31 15:19:59,120 DEBUG [cloud.api.ApiServlet]
>> >> >> > (catalina-exec-5:null)
>> >> >> > ===START=== 172.30.40.135 -- GET command=prepareHostForMa
>> >> >> >
>> >> >> >
>> >>
>> intenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
>> >> >> > 2013-10-31 15:19:59,181 DEBUG [cloud.async.AsyncJobManagerImpl]
>> >> >> > (catalina-exec-5:null) submit async job-167, details: AsyncJobVO
>> >> {id:16
>> >> >> > 7, userId: 2, accountId: 2, sessionKey: null, instanceType: Host,
>> >> >> > instanceId: 1, cmd:
>> org.apache.cloudstack.api.command.admin.host.Prep
>> >> >> > areForMaintenanceCmd, cmdOriginator: null, cmdInfo:
>> >> >> >
>> >> >> >
>> >>
>> {"id":"c2f3c416-f4ba-4f73-b414-6fd6efc734e3","response":"json","sessionkey":"Yjye5q
>> >> >> >
>> >> >> >
>> >>
>> mcLtUBW0/P4XCAcx2ZTAI\u003d","ctxUserId":"2","_":"1383258001952","ctxAccountId":"2","ctxStartEventId":"699"},
>> >> >> > cmdVersion: 0, callbackTy
>> >> >> > pe: 0, callbackAddress: null, status: 0, processStatus: 0,
>> resultCode:
>> >> >> > 0,
>> >> >> > result: null, initMsid: 233845174730253, completeMsid: null,
>> >> >> > lastUpdated: null, lastPolled: null, created: null}
>> >> >> > 2013-10-31 15:19:59,185 DEBUG [cloud.async.AsyncJobManagerImpl]
>> >> >> > (Job-Executor-1:job-167) Executing
>> >> org.apache.cloudstack.api.command.ad
>> >> >> > min.host.PrepareForMaintenanceCmd for job-167
>> >> >> > 2013-10-31 15:19:59,188 DEBUG [cloud.api.ApiServlet]
>> >> >> > (catalina-exec-5:null)
>> >> >> > ===END=== 172.30.40.135 -- GET command=prepareHostForMain
>> >> >> >
>> >> >> >
>> >>
>> tenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
>> >> >> > 2013-10-31 15:19:59,207 DEBUG [cloud.cluster.ClusterManagerImpl]
>> >> >> > (Job-Executor-1:job-167) Propagating agent change request
>> event:AdminA
>> >> >> > skMaintenace to agent:1
>> >> >> > 2013-10-31 15:19:59,208 DEBUG [cloud.cluster.ClusterManagerImpl]
>> >> >> > (Job-Executor-1:job-167) 233845174730253 -> 233845174730255.1
>> [{"Propa
>> >> >> >
>> >> >> >
>> >>
>> gateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
>> >> >> > 2013-10-31 15:19:59,213 INFO
>> >> [cloud.cluster.ClusterServiceServletImpl]
>> >> >> > (Cluster-Worker-1:null) Setup cluster service servlet. service
>> >> >> > url: http://172.30.45.2:9090/clusterservice, request timeout: 300
>> >> >> > seconds
>> >> >> > 2013-10-31 15:19:59,214 DEBUG [cloud.cluster.ClusterManagerImpl]
>> >> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 ->
>> 233845174730255
>> >> >> > . agent: 1, pdu seq: 1, pdu ack seq: 0, json:
>> >> >> >
>> >> >> >
>> >>
>> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{
>> >> >> > },"wait":0}}]
>> >> >> > 2013-10-31 15:19:59,343 DEBUG [cloud.cluster.ClusterManagerImpl]
>> >> >> > (Cluster-Worker-7:null) Dispatch ->1, json:
>> [{"PropagateResourceEventC
>> >> >> >
>> >> >> >
>> >>
>> ommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
>> >> >> > 2013-10-31 15:19:59,347 DEBUG [cloud.cluster.ClusterManagerImpl]
>> >> >> > (Cluster-Worker-7:null) Intercepting command to propagate event
>> AdminA
>> >> >> > skMaintenace for host 1
>> >> >> > 2013-10-31 15:19:59,351 DEBUG
>> >> [cloud.cluster.ClusterServiceServletImpl]
>> >> >> > (Cluster-Worker-1:null) POST http://172.30.45.2:9090/clusterser
>> >> >> > vice response :true, responding time: 80 ms
>> >> >> > 2013-10-31 15:19:59,351 DEBUG [cloud.cluster.ClusterManagerImpl]
>> >> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 ->
>> 233845174730255
>> >> >> > completed. time: 137ms. agent: 1, pdu seq: 1, pdu ack seq: 0,
>> json:
>> >> >> > [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMai
>> >> >> > ntenace","contextMap":{},"wait":0}}]
>> >> >> > 2013-10-31 15:19:59,356 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> > (Cluster-Worker-7:null) Seq 1-102760483: Forwarding Seq 1-102760483
>> >> >> > : { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> >> > 2013-10-31 15:19:59,360 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> > (AgentManager-Handler-1:null) Seq 1-102760483: Forwarding Seq 1-102
>> >> >> > 760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> >> > 100111,
>> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> >> > 2013-10-31 15:19:59,365 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> > (AgentManager-Handler-13:null) Seq 1-102760483: Forwarding Seq 1-10
>> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> >> > 100111,
>> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> >> > 2013-10-31 15:19:59,369 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> > (AgentManager-Handler-15:null) Seq 1-102760483: Forwarding Seq 1-10
>> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> >> > 100111,
>> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> >> > 2013-10-31 15:19:59,372 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> > (AgentManager-Handler-10:null) Seq 1-102760483: Forwarding Seq 1-10
>> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> >> > 100111,
>> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> >> > 2013-10-31 15:19:59,376 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> > (AgentManager-Handler-11:null) Seq 1-102760483: Forwarding Seq 1-10
>> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> >> > 100111,
>> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> >> >
>> >> >> > These MaintainCommand lines are repeated endlessly and very fast
>> (1GB
>> >> of
>> >> >> > logs in about 20 mins) so I currently have the MS stopped.
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Thu, Oct 31, 2013 at 3:43 PM, Carlos Reategui <
>> carlos@reategui.com
>> >> >
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Hi Dan,
>> >> >> >> Network is definitely not a problem. iptables is disabled on MS
>> and
>> >> >> >> both
>> >> >> >> hosts. I am able to ssh from MS to both hosts. I am also able to
>> >> >> >> connect
>> >> >> >> to port 80 from the MS.
>> >> >> >>
>> >> >> >> I need to head out now and will try what you suggested in a couple
>> >> >> >> hours.
>> >> >> >>
>> >> >> >> thanks,
>> >> >> >> Carlos
>> >> >> >>
>> >> >> >>
>> >> >> >> On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland
>> >> >> >> <da...@gmail.com>
>> >> >> >> wrote:
>> >> >> >>>
>> >> >> >>> Carlos,
>> >> >> >>>
>> >> >> >>> can the ms reach the xs hosts?
>> >> >> >>> if so try this, Stop the ms, set the state for the vm_instances
>> for
>> >> >> >>> cpvm and vr from expunging to stopped. and restart.
>> >> >> >>> if not you need to troubleshoot your network connectivity. It
>> seems
>> >> >> >>> this would not be the issue as you only stopped and started it.
>> Did
>> >> >> >>> you
>> >> >> >>>
>> >> >> >>> Daan
>> >> >> >>>
>> >> >> >>> On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui
>> >> >> >>> <ca...@reategui.com>
>> >> >> >>> wrote:
>> >> >> >>> > Hi Dan,
>> >> >> >>> >
>> >> >> >>> > Unfortunately not. Still trying to solve this. See my last
>> email
>> >> >> >>> > to
>> >> >> >>> > Ahmad
>> >> >> >>> > in the recover cs thread about next steps I am trying to figure
>> >> out.
>> >> >> >>> >
>> >> >> >>> > The CPVM and VR are not running. They are currently in
>> expunging
>> >> >> >>> > state.
>> >> >> >>> >
>> >> >> >>> > If I try to start an instance the MS is unable to communicate
>> the
>> >> >> >>> > start
>> >> >> >>> > vm
>> >> >> >>> > commands to my XS hosts and then starts writing to the logs at
>> an
>> >> >> >>> > amazingly
>> >> >> >>> > high rate. You can see one of my logs here:
>> >> >> >>> > http://reategui.com/cloudstack/management-server.log.new
>> >> >> >>> >
>> >> >> >>> > thank you
>> >> >> >>> > Carlos
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>> > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland
>> >> >> >>> > <da...@gmail.com>
>> >> >> >>> > wrote:
>> >> >> >>> >>
>> >> >> >>> >> H Carlos,
>> >> >> >>> >>
>> >> >> >>> >> Did you solve this problem yet?
>> >> >> >>> >>
>> >> >> >>> >> You can try removing the cpvm and vr at the hypervisor and
>> change
>> >> >> >>> >> their state to 'Stopped' in the db.
>> >> >> >>> >>
>> >> >> >>> >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui
>> >> >> >>> >> <ca...@reategui.com>
>> >> >> >>> >> wrote:
>> >> >> >>> >> > In the last 15 minutes management log file it grew 500MB..
>> >> >> >>> >> >
>> >> >> >>> >> > Any ideas what I should look at to figure out what is
>> >> happening?
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui
>> >> >> >>> >> > <cr...@gmail.com>wrote:
>> >> >> >>> >> >
>> >> >> >>> >> >> I am trying to recover my CloudStack installation (CS
>> 4.1.1 on
>> >> >> >>> >> >> ubuntu +
>> >> >> >>> >> >> XS
>> >> >> >>> >> >> 6.0.2) that I shutdown and cannot bring back to life.
>> >> >> >>> >> >>
>> >> >> >>> >> >> I tried to destroy the CPVM and the VR and all I am seeing
>> in
>> >> >> >>> >> >> the
>> >> >> >>> >> >> logs
>> >> >> >>> >> >> is
>> >> >> >>> >> >> the following and they are filling up fast (management log
>> is
>> >> >> >>> >> >> almost
>> >> >> >>> >> >> 1GB).
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >> 2013-10-29 11:51:08,291 DEBUG
>> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >>> >> >> (AgentManager-Handler-14:null) Seq 1-201919186: Forwarding
>> Seq
>> >> >> >>> >> >> 1-201919186:
>> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> >> >>> >> >> 100111,
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >>> >> >> } to 233845174730255
>> >> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
>> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >>> >> >> (AgentManager-Handler-4:null) Seq 2-1168706613: Forwarding
>> Seq
>> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
>> Ver:
>> >> v1,
>> >> >> >>> >> >> Flags:
>> >> >> >>> >> >> 100111,
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >>> >> >> } to 233845174730255
>> >> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
>> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >>> >> >> (AgentManager-Handler-2:null) Seq 1-201919186: Forwarding
>> Seq
>> >> >> >>> >> >> 1-201919186:
>> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> >> >>> >> >> 100111,
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >>> >> >> } to 233845174730255
>> >> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
>> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >>> >> >> (AgentManager-Handler-8:null) Seq 2-1168706613: Forwarding
>> Seq
>> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
>> Ver:
>> >> v1,
>> >> >> >>> >> >> Flags:
>> >> >> >>> >> >> 100111,
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >>> >> >> } to 233845174730255
>> >> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
>> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >>> >> >> (AgentManager-Handler-9:null) Seq 1-201919186: Forwarding
>> Seq
>> >> >> >>> >> >> 1-201919186:
>> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> >> >>> >> >> 100111,
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >>> >> >> } to 233845174730255
>> >> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
>> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >>> >> >> (AgentManager-Handler-6:null) Seq 2-1168706613: Forwarding
>> Seq
>> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
>> Ver:
>> >> v1,
>> >> >> >>> >> >> Flags:
>> >> >> >>> >> >> 100111,
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >>> >> >> } to 233845174730255
>> >> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
>> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >>> >> >> (AgentManager-Handler-15:null) Seq 1-201919186: Forwarding
>> Seq
>> >> >> >>> >> >> 1-201919186:
>> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> >> >>> >> >> 100111,
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >>> >> >> } to 233845174730255
>> >> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
>> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >>> >> >> (AgentManager-Handler-13:null) Seq 2-1168706613: Forwarding
>> >> Seq
>> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
>> Ver:
>> >> v1,
>> >> >> >>> >> >> Flags:
>> >> >> >>> >> >> 100111,
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >>> >> >> } to 233845174730255
>> >> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
>> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >>> >> >> (AgentManager-Handler-10:null) Seq 1-201919186: Forwarding
>> Seq
>> >> >> >>> >> >> 1-201919186:
>> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> >> >>> >> >> 100111,
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >>> >> >> } to 233845174730255
>> >> >> >>> >> >> 2013-10-29 11:51:08,299 DEBUG
>> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >> >>> >> >> (AgentManager-Handler-3:null) Seq 2-1168706613: Forwarding
>> Seq
>> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
>> Ver:
>> >> v1,
>> >> >> >>> >> >> Flags:
>> >> >> >>> >> >> 100111,
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> >>> >> >> } to 233845174730255
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>
>> >> >> >>
>> >> >> >
>> >> >
>> >> >
>> >>
>>
Re: Can't destroy system VMs
Posted by Carlos Reategui <ca...@reategui.com>.
Hi Daan,
There is no trace of CS communicating with either of my XS hosts in the
SMLog file or the xensource.log files when I try to start one of my stopped
instances.
thanks,
Carlos
On Sat, Nov 2, 2013 at 4:50 AM, Daan Hoogland <da...@gmail.com>wrote:
> H Carlos,
>
> I wouldn't know about the mshost and mshost_peer but they seem
> alright. The expunging ssvm are a real problem. I think Ahmed
> mentioned that as well. You are having problems getting your ssvm in
> the air. All other things are consequential. This is the first thing
> to look at. Check the logs and the SMLog on the hypervisor to see what
> happens when cs tries to start a ssvm. By your db you can see that it
> tried to start it several times. Those are the expunging vm_instance
> entries.
>
> regards,
> Daan
>
> On Fri, Nov 1, 2013 at 9:10 PM, Carlos Reategui <ca...@reategui.com>
> wrote:
> > Hi Daan :)
> >
> > Primary and Secondary storage are on the MS. Primary is on a different
> > subnet and it is currently mounted (and visible as an NFS SR from
> > XenCenter) and working fine from the xen machines. I have also been able
> > to mount Secondary without a problem.
> >
> > Here are my current system VMs:
> > mysql> select
> > id,name,instance_name,state,host_id,type,vm_type,desired_state from
> > vm_instance where type!='User';
> >
> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
> > | id | name | instance_name | state | host_id | type
> |
> > vm_type | desired_state |
> >
> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
> > | 2 | v-2-VM | v-2-VM | Stopped | 1 | ConsoleProxy
> |
> > ConsoleProxy | NULL |
> > | 5 | r-5-VM | r-5-VM | Stopped | 2 | DomainRouter
> |
> > DomainRouter | NULL |
> > | 1 | s-1-VM | s-1-VM | Expunging | NULL |
> SecondaryStorageVm |
> > SecondaryStorageVm | NULL |
> > | 3 | s-3-VM | s-3-VM | Expunging | NULL |
> SecondaryStorageVm |
> > SecondaryStorageVm | NULL |
> > | 52 | s-52-VM | s-52-VM | Expunging | NULL |
> SecondaryStorageVm |
> > SecondaryStorageVm | NULL |
> > | 53 | s-53-VM | s-53-VM | Expunging | NULL |
> SecondaryStorageVm |
> > SecondaryStorageVm | NULL |
> > | 54 | s-54-VM | s-54-VM | Expunging | NULL |
> SecondaryStorageVm |
> > SecondaryStorageVm | NULL |
> > | 55 | s-55-VM | s-55-VM | Expunging | NULL |
> SecondaryStorageVm |
> > SecondaryStorageVm | NULL |
> > | 56 | s-56-VM | s-56-VM | Expunging | NULL |
> SecondaryStorageVm |
> > SecondaryStorageVm | NULL |
> >
> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
> >
> >
> > I have been looking through the DB to see if anything sticks out. I
> notice
> > that the mshost and mshost_peer table have 2 entries but they are the
> same
> > server. Is that ok? Looks like one is from the install of 4.1.0 an the
> > other from after I upgraded to 4.1.1:
> >
> > mysql> select * from mshost;
> >
> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
> > | id | msid | runid | name | state | version |
> > service_ip | service_port | last_update | removed | alert_count
> |
> >
> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
> > | 1 | 233845174730255 | 1375383759777 | isbustor01 | Up | 4.1.0 |
> > 172.30.45.2 | 9090 | 2013-10-24 21:13:10 | NULL | 0
> |
> > | 2 | 233845174730253 | 1383331508626 | isbustor01 | Up | 4.1.1 |
> > 172.30.45.2 | 9090 | 2013-11-01 20:00:04 | NULL | 0
> |
> >
> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
> > 2 rows in set (0.00 sec)
> >
> > mysql> select * from mshost_peer;
> >
> +----+--------------+-------------+---------------+------------+---------------------+
> > | id | owner_mshost | peer_mshost | peer_runid | peer_state |
> > last_update |
> >
> +----+--------------+-------------+---------------+------------+---------------------+
> > | 17 | 1 | 1 | 1375383759777 | Up |
> 2013-08-01
> > 19:02:49 |
> > | 34 | 2 | 2 | 1383331508626 | Up |
> 2013-11-01
> > 18:45:19 |
> >
> +----+--------------+-------------+---------------+------------+---------------------+
> >
> >
> > thanks,
> > Carlos
> >
> >
> >
> > On Fri, Nov 1, 2013 at 12:17 PM, Daan Hoogland <daan.hoogland@gmail.com
> >wrote:
> >
> >> h Carlos,
> >>
> >> the states of virtual machines, especially the entries for the ssvm.
> >> if it is not Stopped. set it to stopped and check if your storage,
> >> both primary and secondary, are reachable botjh from the ms and the
> >> hosts.
> >>
> >> Daan (double a!)
> >>
> >> On Fri, Nov 1, 2013 at 7:52 PM, Carlos Reategui <cr...@gmail.com>
> >> wrote:
> >> > Hi Dan,
> >> > The host value is correct and I am able to connect on ports 22,80 and
> >> 8080
> >> > from the Xen hosts to that IP (same subnet). Port 8787 is not
> listening
> >> on
> >> > my MS.
> >> >
> >> > What can I look for in the DB?
> >> >
> >> > thanks
> >> > Carlos
> >> >
> >> >
> >> >
> >> >
> >> > On Fri, Nov 1, 2013 at 2:47 AM, Daan Hoogland <
> daan.hoogland@gmail.com>
> >> > wrote:
> >> >>
> >> >> H Carlox,
> >> >>
> >> >> It still seems to me you have a communication problem albeit not at
> >> >> the network level. ( I agree this doesn't seem likely) But even so
> the
> >> >> ms can't reach the xen hosts. The next thing to look at would be
> >> >> configuration. I presume you did check the ip addresses for the
> hosts.
> >> >> check the value 'host' in the global vars and try to telnet it on
> port
> >> >> 22 80 8080 8787 or what else from the xen hosts . I am sure the
> >> >> problem is hidden somewhere in the mysql database.
> >> >>
> >> >> regards,
> >> >> Daan
> >> >>
> >> >> On Fri, Nov 1, 2013 at 6:23 AM, Carlos Reategui <carlos@reategui.com
> >
> >> >> wrote:
> >> >> > Hi Dan,
> >> >> > Finally getting around to trying this. I am not able to start any
> vm
> >> >> > (mine
> >> >> > or system).
> >> >> >
> >> >> > I am in the process of trying Ahmad's suggestion of removing one of
> >> the
> >> >> > hosts, reinstalling it and then adding it back in to see if that
> >> helps.
> >> >> > However I am not able to put it in maintenance mode. I am getting
> the
> >> >> > following when I try:
> >> >> >
> >> >> >
> >> >> > 2013-10-31 15:19:59,120 DEBUG [cloud.api.ApiServlet]
> >> >> > (catalina-exec-5:null)
> >> >> > ===START=== 172.30.40.135 -- GET command=prepareHostForMa
> >> >> >
> >> >> >
> >>
> intenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
> >> >> > 2013-10-31 15:19:59,181 DEBUG [cloud.async.AsyncJobManagerImpl]
> >> >> > (catalina-exec-5:null) submit async job-167, details: AsyncJobVO
> >> {id:16
> >> >> > 7, userId: 2, accountId: 2, sessionKey: null, instanceType: Host,
> >> >> > instanceId: 1, cmd:
> org.apache.cloudstack.api.command.admin.host.Prep
> >> >> > areForMaintenanceCmd, cmdOriginator: null, cmdInfo:
> >> >> >
> >> >> >
> >>
> {"id":"c2f3c416-f4ba-4f73-b414-6fd6efc734e3","response":"json","sessionkey":"Yjye5q
> >> >> >
> >> >> >
> >>
> mcLtUBW0/P4XCAcx2ZTAI\u003d","ctxUserId":"2","_":"1383258001952","ctxAccountId":"2","ctxStartEventId":"699"},
> >> >> > cmdVersion: 0, callbackTy
> >> >> > pe: 0, callbackAddress: null, status: 0, processStatus: 0,
> resultCode:
> >> >> > 0,
> >> >> > result: null, initMsid: 233845174730253, completeMsid: null,
> >> >> > lastUpdated: null, lastPolled: null, created: null}
> >> >> > 2013-10-31 15:19:59,185 DEBUG [cloud.async.AsyncJobManagerImpl]
> >> >> > (Job-Executor-1:job-167) Executing
> >> org.apache.cloudstack.api.command.ad
> >> >> > min.host.PrepareForMaintenanceCmd for job-167
> >> >> > 2013-10-31 15:19:59,188 DEBUG [cloud.api.ApiServlet]
> >> >> > (catalina-exec-5:null)
> >> >> > ===END=== 172.30.40.135 -- GET command=prepareHostForMain
> >> >> >
> >> >> >
> >>
> tenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
> >> >> > 2013-10-31 15:19:59,207 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> >> > (Job-Executor-1:job-167) Propagating agent change request
> event:AdminA
> >> >> > skMaintenace to agent:1
> >> >> > 2013-10-31 15:19:59,208 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> >> > (Job-Executor-1:job-167) 233845174730253 -> 233845174730255.1
> [{"Propa
> >> >> >
> >> >> >
> >>
> gateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
> >> >> > 2013-10-31 15:19:59,213 INFO
> >> [cloud.cluster.ClusterServiceServletImpl]
> >> >> > (Cluster-Worker-1:null) Setup cluster service servlet. service
> >> >> > url: http://172.30.45.2:9090/clusterservice, request timeout: 300
> >> >> > seconds
> >> >> > 2013-10-31 15:19:59,214 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 ->
> 233845174730255
> >> >> > . agent: 1, pdu seq: 1, pdu ack seq: 0, json:
> >> >> >
> >> >> >
> >>
> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{
> >> >> > },"wait":0}}]
> >> >> > 2013-10-31 15:19:59,343 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> >> > (Cluster-Worker-7:null) Dispatch ->1, json:
> [{"PropagateResourceEventC
> >> >> >
> >> >> >
> >>
> ommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
> >> >> > 2013-10-31 15:19:59,347 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> >> > (Cluster-Worker-7:null) Intercepting command to propagate event
> AdminA
> >> >> > skMaintenace for host 1
> >> >> > 2013-10-31 15:19:59,351 DEBUG
> >> [cloud.cluster.ClusterServiceServletImpl]
> >> >> > (Cluster-Worker-1:null) POST http://172.30.45.2:9090/clusterser
> >> >> > vice response :true, responding time: 80 ms
> >> >> > 2013-10-31 15:19:59,351 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 ->
> 233845174730255
> >> >> > completed. time: 137ms. agent: 1, pdu seq: 1, pdu ack seq: 0,
> json:
> >> >> > [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMai
> >> >> > ntenace","contextMap":{},"wait":0}}]
> >> >> > 2013-10-31 15:19:59,356 DEBUG [agent.manager.ClusteredAgentAttache]
> >> >> > (Cluster-Worker-7:null) Seq 1-102760483: Forwarding Seq 1-102760483
> >> >> > : { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> >> > 2013-10-31 15:19:59,360 DEBUG [agent.manager.ClusteredAgentAttache]
> >> >> > (AgentManager-Handler-1:null) Seq 1-102760483: Forwarding Seq 1-102
> >> >> > 760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >> > 100111,
> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> >> > 2013-10-31 15:19:59,365 DEBUG [agent.manager.ClusteredAgentAttache]
> >> >> > (AgentManager-Handler-13:null) Seq 1-102760483: Forwarding Seq 1-10
> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >> > 100111,
> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> >> > 2013-10-31 15:19:59,369 DEBUG [agent.manager.ClusteredAgentAttache]
> >> >> > (AgentManager-Handler-15:null) Seq 1-102760483: Forwarding Seq 1-10
> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >> > 100111,
> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> >> > 2013-10-31 15:19:59,372 DEBUG [agent.manager.ClusteredAgentAttache]
> >> >> > (AgentManager-Handler-10:null) Seq 1-102760483: Forwarding Seq 1-10
> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >> > 100111,
> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> >> > 2013-10-31 15:19:59,376 DEBUG [agent.manager.ClusteredAgentAttache]
> >> >> > (AgentManager-Handler-11:null) Seq 1-102760483: Forwarding Seq 1-10
> >> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >> > 100111,
> >> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> >> >
> >> >> > These MaintainCommand lines are repeated endlessly and very fast
> (1GB
> >> of
> >> >> > logs in about 20 mins) so I currently have the MS stopped.
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Thu, Oct 31, 2013 at 3:43 PM, Carlos Reategui <
> carlos@reategui.com
> >> >
> >> >> > wrote:
> >> >> >>
> >> >> >> Hi Dan,
> >> >> >> Network is definitely not a problem. iptables is disabled on MS
> and
> >> >> >> both
> >> >> >> hosts. I am able to ssh from MS to both hosts. I am also able to
> >> >> >> connect
> >> >> >> to port 80 from the MS.
> >> >> >>
> >> >> >> I need to head out now and will try what you suggested in a couple
> >> >> >> hours.
> >> >> >>
> >> >> >> thanks,
> >> >> >> Carlos
> >> >> >>
> >> >> >>
> >> >> >> On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland
> >> >> >> <da...@gmail.com>
> >> >> >> wrote:
> >> >> >>>
> >> >> >>> Carlos,
> >> >> >>>
> >> >> >>> can the ms reach the xs hosts?
> >> >> >>> if so try this, Stop the ms, set the state for the vm_instances
> for
> >> >> >>> cpvm and vr from expunging to stopped. and restart.
> >> >> >>> if not you need to troubleshoot your network connectivity. It
> seems
> >> >> >>> this would not be the issue as you only stopped and started it.
> Did
> >> >> >>> you
> >> >> >>>
> >> >> >>> Daan
> >> >> >>>
> >> >> >>> On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui
> >> >> >>> <ca...@reategui.com>
> >> >> >>> wrote:
> >> >> >>> > Hi Dan,
> >> >> >>> >
> >> >> >>> > Unfortunately not. Still trying to solve this. See my last
> email
> >> >> >>> > to
> >> >> >>> > Ahmad
> >> >> >>> > in the recover cs thread about next steps I am trying to figure
> >> out.
> >> >> >>> >
> >> >> >>> > The CPVM and VR are not running. They are currently in
> expunging
> >> >> >>> > state.
> >> >> >>> >
> >> >> >>> > If I try to start an instance the MS is unable to communicate
> the
> >> >> >>> > start
> >> >> >>> > vm
> >> >> >>> > commands to my XS hosts and then starts writing to the logs at
> an
> >> >> >>> > amazingly
> >> >> >>> > high rate. You can see one of my logs here:
> >> >> >>> > http://reategui.com/cloudstack/management-server.log.new
> >> >> >>> >
> >> >> >>> > thank you
> >> >> >>> > Carlos
> >> >> >>> >
> >> >> >>> >
> >> >> >>> >
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland
> >> >> >>> > <da...@gmail.com>
> >> >> >>> > wrote:
> >> >> >>> >>
> >> >> >>> >> H Carlos,
> >> >> >>> >>
> >> >> >>> >> Did you solve this problem yet?
> >> >> >>> >>
> >> >> >>> >> You can try removing the cpvm and vr at the hypervisor and
> change
> >> >> >>> >> their state to 'Stopped' in the db.
> >> >> >>> >>
> >> >> >>> >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui
> >> >> >>> >> <ca...@reategui.com>
> >> >> >>> >> wrote:
> >> >> >>> >> > In the last 15 minutes management log file it grew 500MB..
> >> >> >>> >> >
> >> >> >>> >> > Any ideas what I should look at to figure out what is
> >> happening?
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui
> >> >> >>> >> > <cr...@gmail.com>wrote:
> >> >> >>> >> >
> >> >> >>> >> >> I am trying to recover my CloudStack installation (CS
> 4.1.1 on
> >> >> >>> >> >> ubuntu +
> >> >> >>> >> >> XS
> >> >> >>> >> >> 6.0.2) that I shutdown and cannot bring back to life.
> >> >> >>> >> >>
> >> >> >>> >> >> I tried to destroy the CPVM and the VR and all I am seeing
> in
> >> >> >>> >> >> the
> >> >> >>> >> >> logs
> >> >> >>> >> >> is
> >> >> >>> >> >> the following and they are filling up fast (management log
> is
> >> >> >>> >> >> almost
> >> >> >>> >> >> 1GB).
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >> 2013-10-29 11:51:08,291 DEBUG
> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >>> >> >> (AgentManager-Handler-14:null) Seq 1-201919186: Forwarding
> Seq
> >> >> >>> >> >> 1-201919186:
> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >> >>> >> >> 100111,
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >>> >> >> } to 233845174730255
> >> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >>> >> >> (AgentManager-Handler-4:null) Seq 2-1168706613: Forwarding
> Seq
> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
> Ver:
> >> v1,
> >> >> >>> >> >> Flags:
> >> >> >>> >> >> 100111,
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >>> >> >> } to 233845174730255
> >> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >>> >> >> (AgentManager-Handler-2:null) Seq 1-201919186: Forwarding
> Seq
> >> >> >>> >> >> 1-201919186:
> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >> >>> >> >> 100111,
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >>> >> >> } to 233845174730255
> >> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >>> >> >> (AgentManager-Handler-8:null) Seq 2-1168706613: Forwarding
> Seq
> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
> Ver:
> >> v1,
> >> >> >>> >> >> Flags:
> >> >> >>> >> >> 100111,
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >>> >> >> } to 233845174730255
> >> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >>> >> >> (AgentManager-Handler-9:null) Seq 1-201919186: Forwarding
> Seq
> >> >> >>> >> >> 1-201919186:
> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >> >>> >> >> 100111,
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >>> >> >> } to 233845174730255
> >> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >>> >> >> (AgentManager-Handler-6:null) Seq 2-1168706613: Forwarding
> Seq
> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
> Ver:
> >> v1,
> >> >> >>> >> >> Flags:
> >> >> >>> >> >> 100111,
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >>> >> >> } to 233845174730255
> >> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >>> >> >> (AgentManager-Handler-15:null) Seq 1-201919186: Forwarding
> Seq
> >> >> >>> >> >> 1-201919186:
> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >> >>> >> >> 100111,
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >>> >> >> } to 233845174730255
> >> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >>> >> >> (AgentManager-Handler-13:null) Seq 2-1168706613: Forwarding
> >> Seq
> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
> Ver:
> >> v1,
> >> >> >>> >> >> Flags:
> >> >> >>> >> >> 100111,
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >>> >> >> } to 233845174730255
> >> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >>> >> >> (AgentManager-Handler-10:null) Seq 1-201919186: Forwarding
> Seq
> >> >> >>> >> >> 1-201919186:
> >> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >> >>> >> >> 100111,
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >>> >> >> } to 233845174730255
> >> >> >>> >> >> 2013-10-29 11:51:08,299 DEBUG
> >> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >> >>> >> >> (AgentManager-Handler-3:null) Seq 2-1168706613: Forwarding
> Seq
> >> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2,
> Ver:
> >> v1,
> >> >> >>> >> >> Flags:
> >> >> >>> >> >> 100111,
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> >>> >> >> } to 233845174730255
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >
> >> >> >>> >
> >> >> >>
> >> >> >>
> >> >> >
> >> >
> >> >
> >>
>
Re: Can't destroy system VMs
Posted by Daan Hoogland <da...@gmail.com>.
H Carlos,
I wouldn't know about the mshost and mshost_peer but they seem
alright. The expunging ssvm are a real problem. I think Ahmed
mentioned that as well. You are having problems getting your ssvm in
the air. All other things are consequential. This is the first thing
to look at. Check the logs and the SMLog on the hypervisor to see what
happens when cs tries to start a ssvm. By your db you can see that it
tried to start it several times. Those are the expunging vm_instance
entries.
regards,
Daan
On Fri, Nov 1, 2013 at 9:10 PM, Carlos Reategui <ca...@reategui.com> wrote:
> Hi Daan :)
>
> Primary and Secondary storage are on the MS. Primary is on a different
> subnet and it is currently mounted (and visible as an NFS SR from
> XenCenter) and working fine from the xen machines. I have also been able
> to mount Secondary without a problem.
>
> Here are my current system VMs:
> mysql> select
> id,name,instance_name,state,host_id,type,vm_type,desired_state from
> vm_instance where type!='User';
> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
> | id | name | instance_name | state | host_id | type |
> vm_type | desired_state |
> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
> | 2 | v-2-VM | v-2-VM | Stopped | 1 | ConsoleProxy |
> ConsoleProxy | NULL |
> | 5 | r-5-VM | r-5-VM | Stopped | 2 | DomainRouter |
> DomainRouter | NULL |
> | 1 | s-1-VM | s-1-VM | Expunging | NULL | SecondaryStorageVm |
> SecondaryStorageVm | NULL |
> | 3 | s-3-VM | s-3-VM | Expunging | NULL | SecondaryStorageVm |
> SecondaryStorageVm | NULL |
> | 52 | s-52-VM | s-52-VM | Expunging | NULL | SecondaryStorageVm |
> SecondaryStorageVm | NULL |
> | 53 | s-53-VM | s-53-VM | Expunging | NULL | SecondaryStorageVm |
> SecondaryStorageVm | NULL |
> | 54 | s-54-VM | s-54-VM | Expunging | NULL | SecondaryStorageVm |
> SecondaryStorageVm | NULL |
> | 55 | s-55-VM | s-55-VM | Expunging | NULL | SecondaryStorageVm |
> SecondaryStorageVm | NULL |
> | 56 | s-56-VM | s-56-VM | Expunging | NULL | SecondaryStorageVm |
> SecondaryStorageVm | NULL |
> +----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
>
>
> I have been looking through the DB to see if anything sticks out. I notice
> that the mshost and mshost_peer table have 2 entries but they are the same
> server. Is that ok? Looks like one is from the install of 4.1.0 an the
> other from after I upgraded to 4.1.1:
>
> mysql> select * from mshost;
> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
> | id | msid | runid | name | state | version |
> service_ip | service_port | last_update | removed | alert_count |
> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
> | 1 | 233845174730255 | 1375383759777 | isbustor01 | Up | 4.1.0 |
> 172.30.45.2 | 9090 | 2013-10-24 21:13:10 | NULL | 0 |
> | 2 | 233845174730253 | 1383331508626 | isbustor01 | Up | 4.1.1 |
> 172.30.45.2 | 9090 | 2013-11-01 20:00:04 | NULL | 0 |
> +----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
> 2 rows in set (0.00 sec)
>
> mysql> select * from mshost_peer;
> +----+--------------+-------------+---------------+------------+---------------------+
> | id | owner_mshost | peer_mshost | peer_runid | peer_state |
> last_update |
> +----+--------------+-------------+---------------+------------+---------------------+
> | 17 | 1 | 1 | 1375383759777 | Up | 2013-08-01
> 19:02:49 |
> | 34 | 2 | 2 | 1383331508626 | Up | 2013-11-01
> 18:45:19 |
> +----+--------------+-------------+---------------+------------+---------------------+
>
>
> thanks,
> Carlos
>
>
>
> On Fri, Nov 1, 2013 at 12:17 PM, Daan Hoogland <da...@gmail.com>wrote:
>
>> h Carlos,
>>
>> the states of virtual machines, especially the entries for the ssvm.
>> if it is not Stopped. set it to stopped and check if your storage,
>> both primary and secondary, are reachable botjh from the ms and the
>> hosts.
>>
>> Daan (double a!)
>>
>> On Fri, Nov 1, 2013 at 7:52 PM, Carlos Reategui <cr...@gmail.com>
>> wrote:
>> > Hi Dan,
>> > The host value is correct and I am able to connect on ports 22,80 and
>> 8080
>> > from the Xen hosts to that IP (same subnet). Port 8787 is not listening
>> on
>> > my MS.
>> >
>> > What can I look for in the DB?
>> >
>> > thanks
>> > Carlos
>> >
>> >
>> >
>> >
>> > On Fri, Nov 1, 2013 at 2:47 AM, Daan Hoogland <da...@gmail.com>
>> > wrote:
>> >>
>> >> H Carlox,
>> >>
>> >> It still seems to me you have a communication problem albeit not at
>> >> the network level. ( I agree this doesn't seem likely) But even so the
>> >> ms can't reach the xen hosts. The next thing to look at would be
>> >> configuration. I presume you did check the ip addresses for the hosts.
>> >> check the value 'host' in the global vars and try to telnet it on port
>> >> 22 80 8080 8787 or what else from the xen hosts . I am sure the
>> >> problem is hidden somewhere in the mysql database.
>> >>
>> >> regards,
>> >> Daan
>> >>
>> >> On Fri, Nov 1, 2013 at 6:23 AM, Carlos Reategui <ca...@reategui.com>
>> >> wrote:
>> >> > Hi Dan,
>> >> > Finally getting around to trying this. I am not able to start any vm
>> >> > (mine
>> >> > or system).
>> >> >
>> >> > I am in the process of trying Ahmad's suggestion of removing one of
>> the
>> >> > hosts, reinstalling it and then adding it back in to see if that
>> helps.
>> >> > However I am not able to put it in maintenance mode. I am getting the
>> >> > following when I try:
>> >> >
>> >> >
>> >> > 2013-10-31 15:19:59,120 DEBUG [cloud.api.ApiServlet]
>> >> > (catalina-exec-5:null)
>> >> > ===START=== 172.30.40.135 -- GET command=prepareHostForMa
>> >> >
>> >> >
>> intenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
>> >> > 2013-10-31 15:19:59,181 DEBUG [cloud.async.AsyncJobManagerImpl]
>> >> > (catalina-exec-5:null) submit async job-167, details: AsyncJobVO
>> {id:16
>> >> > 7, userId: 2, accountId: 2, sessionKey: null, instanceType: Host,
>> >> > instanceId: 1, cmd: org.apache.cloudstack.api.command.admin.host.Prep
>> >> > areForMaintenanceCmd, cmdOriginator: null, cmdInfo:
>> >> >
>> >> >
>> {"id":"c2f3c416-f4ba-4f73-b414-6fd6efc734e3","response":"json","sessionkey":"Yjye5q
>> >> >
>> >> >
>> mcLtUBW0/P4XCAcx2ZTAI\u003d","ctxUserId":"2","_":"1383258001952","ctxAccountId":"2","ctxStartEventId":"699"},
>> >> > cmdVersion: 0, callbackTy
>> >> > pe: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode:
>> >> > 0,
>> >> > result: null, initMsid: 233845174730253, completeMsid: null,
>> >> > lastUpdated: null, lastPolled: null, created: null}
>> >> > 2013-10-31 15:19:59,185 DEBUG [cloud.async.AsyncJobManagerImpl]
>> >> > (Job-Executor-1:job-167) Executing
>> org.apache.cloudstack.api.command.ad
>> >> > min.host.PrepareForMaintenanceCmd for job-167
>> >> > 2013-10-31 15:19:59,188 DEBUG [cloud.api.ApiServlet]
>> >> > (catalina-exec-5:null)
>> >> > ===END=== 172.30.40.135 -- GET command=prepareHostForMain
>> >> >
>> >> >
>> tenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
>> >> > 2013-10-31 15:19:59,207 DEBUG [cloud.cluster.ClusterManagerImpl]
>> >> > (Job-Executor-1:job-167) Propagating agent change request event:AdminA
>> >> > skMaintenace to agent:1
>> >> > 2013-10-31 15:19:59,208 DEBUG [cloud.cluster.ClusterManagerImpl]
>> >> > (Job-Executor-1:job-167) 233845174730253 -> 233845174730255.1 [{"Propa
>> >> >
>> >> >
>> gateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
>> >> > 2013-10-31 15:19:59,213 INFO
>> [cloud.cluster.ClusterServiceServletImpl]
>> >> > (Cluster-Worker-1:null) Setup cluster service servlet. service
>> >> > url: http://172.30.45.2:9090/clusterservice, request timeout: 300
>> >> > seconds
>> >> > 2013-10-31 15:19:59,214 DEBUG [cloud.cluster.ClusterManagerImpl]
>> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 -> 233845174730255
>> >> > . agent: 1, pdu seq: 1, pdu ack seq: 0, json:
>> >> >
>> >> >
>> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{
>> >> > },"wait":0}}]
>> >> > 2013-10-31 15:19:59,343 DEBUG [cloud.cluster.ClusterManagerImpl]
>> >> > (Cluster-Worker-7:null) Dispatch ->1, json: [{"PropagateResourceEventC
>> >> >
>> >> >
>> ommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
>> >> > 2013-10-31 15:19:59,347 DEBUG [cloud.cluster.ClusterManagerImpl]
>> >> > (Cluster-Worker-7:null) Intercepting command to propagate event AdminA
>> >> > skMaintenace for host 1
>> >> > 2013-10-31 15:19:59,351 DEBUG
>> [cloud.cluster.ClusterServiceServletImpl]
>> >> > (Cluster-Worker-1:null) POST http://172.30.45.2:9090/clusterser
>> >> > vice response :true, responding time: 80 ms
>> >> > 2013-10-31 15:19:59,351 DEBUG [cloud.cluster.ClusterManagerImpl]
>> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 -> 233845174730255
>> >> > completed. time: 137ms. agent: 1, pdu seq: 1, pdu ack seq: 0, json:
>> >> > [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMai
>> >> > ntenace","contextMap":{},"wait":0}}]
>> >> > 2013-10-31 15:19:59,356 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> > (Cluster-Worker-7:null) Seq 1-102760483: Forwarding Seq 1-102760483
>> >> > : { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> > 2013-10-31 15:19:59,360 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> > (AgentManager-Handler-1:null) Seq 1-102760483: Forwarding Seq 1-102
>> >> > 760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> > 100111,
>> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> > 2013-10-31 15:19:59,365 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> > (AgentManager-Handler-13:null) Seq 1-102760483: Forwarding Seq 1-10
>> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> > 100111,
>> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> > 2013-10-31 15:19:59,369 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> > (AgentManager-Handler-15:null) Seq 1-102760483: Forwarding Seq 1-10
>> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> > 100111,
>> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> > 2013-10-31 15:19:59,372 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> > (AgentManager-Handler-10:null) Seq 1-102760483: Forwarding Seq 1-10
>> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> > 100111,
>> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> > 2013-10-31 15:19:59,376 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> > (AgentManager-Handler-11:null) Seq 1-102760483: Forwarding Seq 1-10
>> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> > 100111,
>> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >> >
>> >> > These MaintainCommand lines are repeated endlessly and very fast (1GB
>> of
>> >> > logs in about 20 mins) so I currently have the MS stopped.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Thu, Oct 31, 2013 at 3:43 PM, Carlos Reategui <carlos@reategui.com
>> >
>> >> > wrote:
>> >> >>
>> >> >> Hi Dan,
>> >> >> Network is definitely not a problem. iptables is disabled on MS and
>> >> >> both
>> >> >> hosts. I am able to ssh from MS to both hosts. I am also able to
>> >> >> connect
>> >> >> to port 80 from the MS.
>> >> >>
>> >> >> I need to head out now and will try what you suggested in a couple
>> >> >> hours.
>> >> >>
>> >> >> thanks,
>> >> >> Carlos
>> >> >>
>> >> >>
>> >> >> On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland
>> >> >> <da...@gmail.com>
>> >> >> wrote:
>> >> >>>
>> >> >>> Carlos,
>> >> >>>
>> >> >>> can the ms reach the xs hosts?
>> >> >>> if so try this, Stop the ms, set the state for the vm_instances for
>> >> >>> cpvm and vr from expunging to stopped. and restart.
>> >> >>> if not you need to troubleshoot your network connectivity. It seems
>> >> >>> this would not be the issue as you only stopped and started it. Did
>> >> >>> you
>> >> >>>
>> >> >>> Daan
>> >> >>>
>> >> >>> On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui
>> >> >>> <ca...@reategui.com>
>> >> >>> wrote:
>> >> >>> > Hi Dan,
>> >> >>> >
>> >> >>> > Unfortunately not. Still trying to solve this. See my last email
>> >> >>> > to
>> >> >>> > Ahmad
>> >> >>> > in the recover cs thread about next steps I am trying to figure
>> out.
>> >> >>> >
>> >> >>> > The CPVM and VR are not running. They are currently in expunging
>> >> >>> > state.
>> >> >>> >
>> >> >>> > If I try to start an instance the MS is unable to communicate the
>> >> >>> > start
>> >> >>> > vm
>> >> >>> > commands to my XS hosts and then starts writing to the logs at an
>> >> >>> > amazingly
>> >> >>> > high rate. You can see one of my logs here:
>> >> >>> > http://reategui.com/cloudstack/management-server.log.new
>> >> >>> >
>> >> >>> > thank you
>> >> >>> > Carlos
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland
>> >> >>> > <da...@gmail.com>
>> >> >>> > wrote:
>> >> >>> >>
>> >> >>> >> H Carlos,
>> >> >>> >>
>> >> >>> >> Did you solve this problem yet?
>> >> >>> >>
>> >> >>> >> You can try removing the cpvm and vr at the hypervisor and change
>> >> >>> >> their state to 'Stopped' in the db.
>> >> >>> >>
>> >> >>> >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui
>> >> >>> >> <ca...@reategui.com>
>> >> >>> >> wrote:
>> >> >>> >> > In the last 15 minutes management log file it grew 500MB..
>> >> >>> >> >
>> >> >>> >> > Any ideas what I should look at to figure out what is
>> happening?
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui
>> >> >>> >> > <cr...@gmail.com>wrote:
>> >> >>> >> >
>> >> >>> >> >> I am trying to recover my CloudStack installation (CS 4.1.1 on
>> >> >>> >> >> ubuntu +
>> >> >>> >> >> XS
>> >> >>> >> >> 6.0.2) that I shutdown and cannot bring back to life.
>> >> >>> >> >>
>> >> >>> >> >> I tried to destroy the CPVM and the VR and all I am seeing in
>> >> >>> >> >> the
>> >> >>> >> >> logs
>> >> >>> >> >> is
>> >> >>> >> >> the following and they are filling up fast (management log is
>> >> >>> >> >> almost
>> >> >>> >> >> 1GB).
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >> 2013-10-29 11:51:08,291 DEBUG
>> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >>> >> >> (AgentManager-Handler-14:null) Seq 1-201919186: Forwarding Seq
>> >> >>> >> >> 1-201919186:
>> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> >>> >> >> 100111,
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >>> >> >> } to 233845174730255
>> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
>> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >>> >> >> (AgentManager-Handler-4:null) Seq 2-1168706613: Forwarding Seq
>> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver:
>> v1,
>> >> >>> >> >> Flags:
>> >> >>> >> >> 100111,
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >>> >> >> } to 233845174730255
>> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
>> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >>> >> >> (AgentManager-Handler-2:null) Seq 1-201919186: Forwarding Seq
>> >> >>> >> >> 1-201919186:
>> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> >>> >> >> 100111,
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >>> >> >> } to 233845174730255
>> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
>> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >>> >> >> (AgentManager-Handler-8:null) Seq 2-1168706613: Forwarding Seq
>> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver:
>> v1,
>> >> >>> >> >> Flags:
>> >> >>> >> >> 100111,
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >>> >> >> } to 233845174730255
>> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
>> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >>> >> >> (AgentManager-Handler-9:null) Seq 1-201919186: Forwarding Seq
>> >> >>> >> >> 1-201919186:
>> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> >>> >> >> 100111,
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >>> >> >> } to 233845174730255
>> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
>> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >>> >> >> (AgentManager-Handler-6:null) Seq 2-1168706613: Forwarding Seq
>> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver:
>> v1,
>> >> >>> >> >> Flags:
>> >> >>> >> >> 100111,
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >>> >> >> } to 233845174730255
>> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
>> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >>> >> >> (AgentManager-Handler-15:null) Seq 1-201919186: Forwarding Seq
>> >> >>> >> >> 1-201919186:
>> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> >>> >> >> 100111,
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >>> >> >> } to 233845174730255
>> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
>> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >>> >> >> (AgentManager-Handler-13:null) Seq 2-1168706613: Forwarding
>> Seq
>> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver:
>> v1,
>> >> >>> >> >> Flags:
>> >> >>> >> >> 100111,
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >>> >> >> } to 233845174730255
>> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
>> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >>> >> >> (AgentManager-Handler-10:null) Seq 1-201919186: Forwarding Seq
>> >> >>> >> >> 1-201919186:
>> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >> >>> >> >> 100111,
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >>> >> >> } to 233845174730255
>> >> >>> >> >> 2013-10-29 11:51:08,299 DEBUG
>> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >> >>> >> >> (AgentManager-Handler-3:null) Seq 2-1168706613: Forwarding Seq
>> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver:
>> v1,
>> >> >>> >> >> Flags:
>> >> >>> >> >> 100111,
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >>> >> >> } to 233845174730255
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >
>> >> >>> >
>> >> >>
>> >> >>
>> >> >
>> >
>> >
>>
Re: Can't destroy system VMs
Posted by Carlos Reategui <ca...@reategui.com>.
Hi Daan :)
Primary and Secondary storage are on the MS. Primary is on a different
subnet and it is currently mounted (and visible as an NFS SR from
XenCenter) and working fine from the xen machines. I have also been able
to mount Secondary without a problem.
Here are my current system VMs:
mysql> select
id,name,instance_name,state,host_id,type,vm_type,desired_state from
vm_instance where type!='User';
+----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
| id | name | instance_name | state | host_id | type |
vm_type | desired_state |
+----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
| 2 | v-2-VM | v-2-VM | Stopped | 1 | ConsoleProxy |
ConsoleProxy | NULL |
| 5 | r-5-VM | r-5-VM | Stopped | 2 | DomainRouter |
DomainRouter | NULL |
| 1 | s-1-VM | s-1-VM | Expunging | NULL | SecondaryStorageVm |
SecondaryStorageVm | NULL |
| 3 | s-3-VM | s-3-VM | Expunging | NULL | SecondaryStorageVm |
SecondaryStorageVm | NULL |
| 52 | s-52-VM | s-52-VM | Expunging | NULL | SecondaryStorageVm |
SecondaryStorageVm | NULL |
| 53 | s-53-VM | s-53-VM | Expunging | NULL | SecondaryStorageVm |
SecondaryStorageVm | NULL |
| 54 | s-54-VM | s-54-VM | Expunging | NULL | SecondaryStorageVm |
SecondaryStorageVm | NULL |
| 55 | s-55-VM | s-55-VM | Expunging | NULL | SecondaryStorageVm |
SecondaryStorageVm | NULL |
| 56 | s-56-VM | s-56-VM | Expunging | NULL | SecondaryStorageVm |
SecondaryStorageVm | NULL |
+----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
I have been looking through the DB to see if anything sticks out. I notice
that the mshost and mshost_peer table have 2 entries but they are the same
server. Is that ok? Looks like one is from the install of 4.1.0 an the
other from after I upgraded to 4.1.1:
mysql> select * from mshost;
+----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
| id | msid | runid | name | state | version |
service_ip | service_port | last_update | removed | alert_count |
+----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
| 1 | 233845174730255 | 1375383759777 | isbustor01 | Up | 4.1.0 |
172.30.45.2 | 9090 | 2013-10-24 21:13:10 | NULL | 0 |
| 2 | 233845174730253 | 1383331508626 | isbustor01 | Up | 4.1.1 |
172.30.45.2 | 9090 | 2013-11-01 20:00:04 | NULL | 0 |
+----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
2 rows in set (0.00 sec)
mysql> select * from mshost_peer;
+----+--------------+-------------+---------------+------------+---------------------+
| id | owner_mshost | peer_mshost | peer_runid | peer_state |
last_update |
+----+--------------+-------------+---------------+------------+---------------------+
| 17 | 1 | 1 | 1375383759777 | Up | 2013-08-01
19:02:49 |
| 34 | 2 | 2 | 1383331508626 | Up | 2013-11-01
18:45:19 |
+----+--------------+-------------+---------------+------------+---------------------+
thanks,
Carlos
On Fri, Nov 1, 2013 at 12:17 PM, Daan Hoogland <da...@gmail.com>wrote:
> h Carlos,
>
> the states of virtual machines, especially the entries for the ssvm.
> if it is not Stopped. set it to stopped and check if your storage,
> both primary and secondary, are reachable botjh from the ms and the
> hosts.
>
> Daan (double a!)
>
> On Fri, Nov 1, 2013 at 7:52 PM, Carlos Reategui <cr...@gmail.com>
> wrote:
> > Hi Dan,
> > The host value is correct and I am able to connect on ports 22,80 and
> 8080
> > from the Xen hosts to that IP (same subnet). Port 8787 is not listening
> on
> > my MS.
> >
> > What can I look for in the DB?
> >
> > thanks
> > Carlos
> >
> >
> >
> >
> > On Fri, Nov 1, 2013 at 2:47 AM, Daan Hoogland <da...@gmail.com>
> > wrote:
> >>
> >> H Carlox,
> >>
> >> It still seems to me you have a communication problem albeit not at
> >> the network level. ( I agree this doesn't seem likely) But even so the
> >> ms can't reach the xen hosts. The next thing to look at would be
> >> configuration. I presume you did check the ip addresses for the hosts.
> >> check the value 'host' in the global vars and try to telnet it on port
> >> 22 80 8080 8787 or what else from the xen hosts . I am sure the
> >> problem is hidden somewhere in the mysql database.
> >>
> >> regards,
> >> Daan
> >>
> >> On Fri, Nov 1, 2013 at 6:23 AM, Carlos Reategui <ca...@reategui.com>
> >> wrote:
> >> > Hi Dan,
> >> > Finally getting around to trying this. I am not able to start any vm
> >> > (mine
> >> > or system).
> >> >
> >> > I am in the process of trying Ahmad's suggestion of removing one of
> the
> >> > hosts, reinstalling it and then adding it back in to see if that
> helps.
> >> > However I am not able to put it in maintenance mode. I am getting the
> >> > following when I try:
> >> >
> >> >
> >> > 2013-10-31 15:19:59,120 DEBUG [cloud.api.ApiServlet]
> >> > (catalina-exec-5:null)
> >> > ===START=== 172.30.40.135 -- GET command=prepareHostForMa
> >> >
> >> >
> intenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
> >> > 2013-10-31 15:19:59,181 DEBUG [cloud.async.AsyncJobManagerImpl]
> >> > (catalina-exec-5:null) submit async job-167, details: AsyncJobVO
> {id:16
> >> > 7, userId: 2, accountId: 2, sessionKey: null, instanceType: Host,
> >> > instanceId: 1, cmd: org.apache.cloudstack.api.command.admin.host.Prep
> >> > areForMaintenanceCmd, cmdOriginator: null, cmdInfo:
> >> >
> >> >
> {"id":"c2f3c416-f4ba-4f73-b414-6fd6efc734e3","response":"json","sessionkey":"Yjye5q
> >> >
> >> >
> mcLtUBW0/P4XCAcx2ZTAI\u003d","ctxUserId":"2","_":"1383258001952","ctxAccountId":"2","ctxStartEventId":"699"},
> >> > cmdVersion: 0, callbackTy
> >> > pe: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode:
> >> > 0,
> >> > result: null, initMsid: 233845174730253, completeMsid: null,
> >> > lastUpdated: null, lastPolled: null, created: null}
> >> > 2013-10-31 15:19:59,185 DEBUG [cloud.async.AsyncJobManagerImpl]
> >> > (Job-Executor-1:job-167) Executing
> org.apache.cloudstack.api.command.ad
> >> > min.host.PrepareForMaintenanceCmd for job-167
> >> > 2013-10-31 15:19:59,188 DEBUG [cloud.api.ApiServlet]
> >> > (catalina-exec-5:null)
> >> > ===END=== 172.30.40.135 -- GET command=prepareHostForMain
> >> >
> >> >
> tenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
> >> > 2013-10-31 15:19:59,207 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> > (Job-Executor-1:job-167) Propagating agent change request event:AdminA
> >> > skMaintenace to agent:1
> >> > 2013-10-31 15:19:59,208 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> > (Job-Executor-1:job-167) 233845174730253 -> 233845174730255.1 [{"Propa
> >> >
> >> >
> gateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
> >> > 2013-10-31 15:19:59,213 INFO
> [cloud.cluster.ClusterServiceServletImpl]
> >> > (Cluster-Worker-1:null) Setup cluster service servlet. service
> >> > url: http://172.30.45.2:9090/clusterservice, request timeout: 300
> >> > seconds
> >> > 2013-10-31 15:19:59,214 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 -> 233845174730255
> >> > . agent: 1, pdu seq: 1, pdu ack seq: 0, json:
> >> >
> >> >
> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{
> >> > },"wait":0}}]
> >> > 2013-10-31 15:19:59,343 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> > (Cluster-Worker-7:null) Dispatch ->1, json: [{"PropagateResourceEventC
> >> >
> >> >
> ommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
> >> > 2013-10-31 15:19:59,347 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> > (Cluster-Worker-7:null) Intercepting command to propagate event AdminA
> >> > skMaintenace for host 1
> >> > 2013-10-31 15:19:59,351 DEBUG
> [cloud.cluster.ClusterServiceServletImpl]
> >> > (Cluster-Worker-1:null) POST http://172.30.45.2:9090/clusterser
> >> > vice response :true, responding time: 80 ms
> >> > 2013-10-31 15:19:59,351 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 -> 233845174730255
> >> > completed. time: 137ms. agent: 1, pdu seq: 1, pdu ack seq: 0, json:
> >> > [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMai
> >> > ntenace","contextMap":{},"wait":0}}]
> >> > 2013-10-31 15:19:59,356 DEBUG [agent.manager.ClusteredAgentAttache]
> >> > (Cluster-Worker-7:null) Seq 1-102760483: Forwarding Seq 1-102760483
> >> > : { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> > 2013-10-31 15:19:59,360 DEBUG [agent.manager.ClusteredAgentAttache]
> >> > (AgentManager-Handler-1:null) Seq 1-102760483: Forwarding Seq 1-102
> >> > 760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> > 100111,
> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> > 2013-10-31 15:19:59,365 DEBUG [agent.manager.ClusteredAgentAttache]
> >> > (AgentManager-Handler-13:null) Seq 1-102760483: Forwarding Seq 1-10
> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> > 100111,
> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> > 2013-10-31 15:19:59,369 DEBUG [agent.manager.ClusteredAgentAttache]
> >> > (AgentManager-Handler-15:null) Seq 1-102760483: Forwarding Seq 1-10
> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> > 100111,
> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> > 2013-10-31 15:19:59,372 DEBUG [agent.manager.ClusteredAgentAttache]
> >> > (AgentManager-Handler-10:null) Seq 1-102760483: Forwarding Seq 1-10
> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> > 100111,
> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> > 2013-10-31 15:19:59,376 DEBUG [agent.manager.ClusteredAgentAttache]
> >> > (AgentManager-Handler-11:null) Seq 1-102760483: Forwarding Seq 1-10
> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> > 100111,
> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> >
> >> > These MaintainCommand lines are repeated endlessly and very fast (1GB
> of
> >> > logs in about 20 mins) so I currently have the MS stopped.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Oct 31, 2013 at 3:43 PM, Carlos Reategui <carlos@reategui.com
> >
> >> > wrote:
> >> >>
> >> >> Hi Dan,
> >> >> Network is definitely not a problem. iptables is disabled on MS and
> >> >> both
> >> >> hosts. I am able to ssh from MS to both hosts. I am also able to
> >> >> connect
> >> >> to port 80 from the MS.
> >> >>
> >> >> I need to head out now and will try what you suggested in a couple
> >> >> hours.
> >> >>
> >> >> thanks,
> >> >> Carlos
> >> >>
> >> >>
> >> >> On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland
> >> >> <da...@gmail.com>
> >> >> wrote:
> >> >>>
> >> >>> Carlos,
> >> >>>
> >> >>> can the ms reach the xs hosts?
> >> >>> if so try this, Stop the ms, set the state for the vm_instances for
> >> >>> cpvm and vr from expunging to stopped. and restart.
> >> >>> if not you need to troubleshoot your network connectivity. It seems
> >> >>> this would not be the issue as you only stopped and started it. Did
> >> >>> you
> >> >>>
> >> >>> Daan
> >> >>>
> >> >>> On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui
> >> >>> <ca...@reategui.com>
> >> >>> wrote:
> >> >>> > Hi Dan,
> >> >>> >
> >> >>> > Unfortunately not. Still trying to solve this. See my last email
> >> >>> > to
> >> >>> > Ahmad
> >> >>> > in the recover cs thread about next steps I am trying to figure
> out.
> >> >>> >
> >> >>> > The CPVM and VR are not running. They are currently in expunging
> >> >>> > state.
> >> >>> >
> >> >>> > If I try to start an instance the MS is unable to communicate the
> >> >>> > start
> >> >>> > vm
> >> >>> > commands to my XS hosts and then starts writing to the logs at an
> >> >>> > amazingly
> >> >>> > high rate. You can see one of my logs here:
> >> >>> > http://reategui.com/cloudstack/management-server.log.new
> >> >>> >
> >> >>> > thank you
> >> >>> > Carlos
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland
> >> >>> > <da...@gmail.com>
> >> >>> > wrote:
> >> >>> >>
> >> >>> >> H Carlos,
> >> >>> >>
> >> >>> >> Did you solve this problem yet?
> >> >>> >>
> >> >>> >> You can try removing the cpvm and vr at the hypervisor and change
> >> >>> >> their state to 'Stopped' in the db.
> >> >>> >>
> >> >>> >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui
> >> >>> >> <ca...@reategui.com>
> >> >>> >> wrote:
> >> >>> >> > In the last 15 minutes management log file it grew 500MB..
> >> >>> >> >
> >> >>> >> > Any ideas what I should look at to figure out what is
> happening?
> >> >>> >> >
> >> >>> >> >
> >> >>> >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui
> >> >>> >> > <cr...@gmail.com>wrote:
> >> >>> >> >
> >> >>> >> >> I am trying to recover my CloudStack installation (CS 4.1.1 on
> >> >>> >> >> ubuntu +
> >> >>> >> >> XS
> >> >>> >> >> 6.0.2) that I shutdown and cannot bring back to life.
> >> >>> >> >>
> >> >>> >> >> I tried to destroy the CPVM and the VR and all I am seeing in
> >> >>> >> >> the
> >> >>> >> >> logs
> >> >>> >> >> is
> >> >>> >> >> the following and they are filling up fast (management log is
> >> >>> >> >> almost
> >> >>> >> >> 1GB).
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >> 2013-10-29 11:51:08,291 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-14:null) Seq 1-201919186: Forwarding Seq
> >> >>> >> >> 1-201919186:
> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-4:null) Seq 2-1168706613: Forwarding Seq
> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver:
> v1,
> >> >>> >> >> Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-2:null) Seq 1-201919186: Forwarding Seq
> >> >>> >> >> 1-201919186:
> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-8:null) Seq 2-1168706613: Forwarding Seq
> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver:
> v1,
> >> >>> >> >> Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-9:null) Seq 1-201919186: Forwarding Seq
> >> >>> >> >> 1-201919186:
> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-6:null) Seq 2-1168706613: Forwarding Seq
> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver:
> v1,
> >> >>> >> >> Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-15:null) Seq 1-201919186: Forwarding Seq
> >> >>> >> >> 1-201919186:
> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-13:null) Seq 2-1168706613: Forwarding
> Seq
> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver:
> v1,
> >> >>> >> >> Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-10:null) Seq 1-201919186: Forwarding Seq
> >> >>> >> >> 1-201919186:
> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,299 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-3:null) Seq 2-1168706613: Forwarding Seq
> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver:
> v1,
> >> >>> >> >> Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >
> >> >>> >
> >> >>
> >> >>
> >> >
> >
> >
>
Re: Can't destroy system VMs
Posted by Daan Hoogland <da...@gmail.com>.
h Carlos,
the states of virtual machines, especially the entries for the ssvm.
if it is not Stopped. set it to stopped and check if your storage,
both primary and secondary, are reachable botjh from the ms and the
hosts.
Daan (double a!)
On Fri, Nov 1, 2013 at 7:52 PM, Carlos Reategui <cr...@gmail.com> wrote:
> Hi Dan,
> The host value is correct and I am able to connect on ports 22,80 and 8080
> from the Xen hosts to that IP (same subnet). Port 8787 is not listening on
> my MS.
>
> What can I look for in the DB?
>
> thanks
> Carlos
>
>
>
>
> On Fri, Nov 1, 2013 at 2:47 AM, Daan Hoogland <da...@gmail.com>
> wrote:
>>
>> H Carlox,
>>
>> It still seems to me you have a communication problem albeit not at
>> the network level. ( I agree this doesn't seem likely) But even so the
>> ms can't reach the xen hosts. The next thing to look at would be
>> configuration. I presume you did check the ip addresses for the hosts.
>> check the value 'host' in the global vars and try to telnet it on port
>> 22 80 8080 8787 or what else from the xen hosts . I am sure the
>> problem is hidden somewhere in the mysql database.
>>
>> regards,
>> Daan
>>
>> On Fri, Nov 1, 2013 at 6:23 AM, Carlos Reategui <ca...@reategui.com>
>> wrote:
>> > Hi Dan,
>> > Finally getting around to trying this. I am not able to start any vm
>> > (mine
>> > or system).
>> >
>> > I am in the process of trying Ahmad's suggestion of removing one of the
>> > hosts, reinstalling it and then adding it back in to see if that helps.
>> > However I am not able to put it in maintenance mode. I am getting the
>> > following when I try:
>> >
>> >
>> > 2013-10-31 15:19:59,120 DEBUG [cloud.api.ApiServlet]
>> > (catalina-exec-5:null)
>> > ===START=== 172.30.40.135 -- GET command=prepareHostForMa
>> >
>> > intenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
>> > 2013-10-31 15:19:59,181 DEBUG [cloud.async.AsyncJobManagerImpl]
>> > (catalina-exec-5:null) submit async job-167, details: AsyncJobVO {id:16
>> > 7, userId: 2, accountId: 2, sessionKey: null, instanceType: Host,
>> > instanceId: 1, cmd: org.apache.cloudstack.api.command.admin.host.Prep
>> > areForMaintenanceCmd, cmdOriginator: null, cmdInfo:
>> >
>> > {"id":"c2f3c416-f4ba-4f73-b414-6fd6efc734e3","response":"json","sessionkey":"Yjye5q
>> >
>> > mcLtUBW0/P4XCAcx2ZTAI\u003d","ctxUserId":"2","_":"1383258001952","ctxAccountId":"2","ctxStartEventId":"699"},
>> > cmdVersion: 0, callbackTy
>> > pe: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode:
>> > 0,
>> > result: null, initMsid: 233845174730253, completeMsid: null,
>> > lastUpdated: null, lastPolled: null, created: null}
>> > 2013-10-31 15:19:59,185 DEBUG [cloud.async.AsyncJobManagerImpl]
>> > (Job-Executor-1:job-167) Executing org.apache.cloudstack.api.command.ad
>> > min.host.PrepareForMaintenanceCmd for job-167
>> > 2013-10-31 15:19:59,188 DEBUG [cloud.api.ApiServlet]
>> > (catalina-exec-5:null)
>> > ===END=== 172.30.40.135 -- GET command=prepareHostForMain
>> >
>> > tenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
>> > 2013-10-31 15:19:59,207 DEBUG [cloud.cluster.ClusterManagerImpl]
>> > (Job-Executor-1:job-167) Propagating agent change request event:AdminA
>> > skMaintenace to agent:1
>> > 2013-10-31 15:19:59,208 DEBUG [cloud.cluster.ClusterManagerImpl]
>> > (Job-Executor-1:job-167) 233845174730253 -> 233845174730255.1 [{"Propa
>> >
>> > gateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
>> > 2013-10-31 15:19:59,213 INFO [cloud.cluster.ClusterServiceServletImpl]
>> > (Cluster-Worker-1:null) Setup cluster service servlet. service
>> > url: http://172.30.45.2:9090/clusterservice, request timeout: 300
>> > seconds
>> > 2013-10-31 15:19:59,214 DEBUG [cloud.cluster.ClusterManagerImpl]
>> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 -> 233845174730255
>> > . agent: 1, pdu seq: 1, pdu ack seq: 0, json:
>> >
>> > [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{
>> > },"wait":0}}]
>> > 2013-10-31 15:19:59,343 DEBUG [cloud.cluster.ClusterManagerImpl]
>> > (Cluster-Worker-7:null) Dispatch ->1, json: [{"PropagateResourceEventC
>> >
>> > ommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
>> > 2013-10-31 15:19:59,347 DEBUG [cloud.cluster.ClusterManagerImpl]
>> > (Cluster-Worker-7:null) Intercepting command to propagate event AdminA
>> > skMaintenace for host 1
>> > 2013-10-31 15:19:59,351 DEBUG [cloud.cluster.ClusterServiceServletImpl]
>> > (Cluster-Worker-1:null) POST http://172.30.45.2:9090/clusterser
>> > vice response :true, responding time: 80 ms
>> > 2013-10-31 15:19:59,351 DEBUG [cloud.cluster.ClusterManagerImpl]
>> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 -> 233845174730255
>> > completed. time: 137ms. agent: 1, pdu seq: 1, pdu ack seq: 0, json:
>> > [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMai
>> > ntenace","contextMap":{},"wait":0}}]
>> > 2013-10-31 15:19:59,356 DEBUG [agent.manager.ClusteredAgentAttache]
>> > (Cluster-Worker-7:null) Seq 1-102760483: Forwarding Seq 1-102760483
>> > : { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> > 2013-10-31 15:19:59,360 DEBUG [agent.manager.ClusteredAgentAttache]
>> > (AgentManager-Handler-1:null) Seq 1-102760483: Forwarding Seq 1-102
>> > 760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> > 100111,
>> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> > 2013-10-31 15:19:59,365 DEBUG [agent.manager.ClusteredAgentAttache]
>> > (AgentManager-Handler-13:null) Seq 1-102760483: Forwarding Seq 1-10
>> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> > 100111,
>> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> > 2013-10-31 15:19:59,369 DEBUG [agent.manager.ClusteredAgentAttache]
>> > (AgentManager-Handler-15:null) Seq 1-102760483: Forwarding Seq 1-10
>> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> > 100111,
>> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> > 2013-10-31 15:19:59,372 DEBUG [agent.manager.ClusteredAgentAttache]
>> > (AgentManager-Handler-10:null) Seq 1-102760483: Forwarding Seq 1-10
>> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> > 100111,
>> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> > 2013-10-31 15:19:59,376 DEBUG [agent.manager.ClusteredAgentAttache]
>> > (AgentManager-Handler-11:null) Seq 1-102760483: Forwarding Seq 1-10
>> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> > 100111,
>> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
>> >
>> > These MaintainCommand lines are repeated endlessly and very fast (1GB of
>> > logs in about 20 mins) so I currently have the MS stopped.
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Oct 31, 2013 at 3:43 PM, Carlos Reategui <ca...@reategui.com>
>> > wrote:
>> >>
>> >> Hi Dan,
>> >> Network is definitely not a problem. iptables is disabled on MS and
>> >> both
>> >> hosts. I am able to ssh from MS to both hosts. I am also able to
>> >> connect
>> >> to port 80 from the MS.
>> >>
>> >> I need to head out now and will try what you suggested in a couple
>> >> hours.
>> >>
>> >> thanks,
>> >> Carlos
>> >>
>> >>
>> >> On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland
>> >> <da...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Carlos,
>> >>>
>> >>> can the ms reach the xs hosts?
>> >>> if so try this, Stop the ms, set the state for the vm_instances for
>> >>> cpvm and vr from expunging to stopped. and restart.
>> >>> if not you need to troubleshoot your network connectivity. It seems
>> >>> this would not be the issue as you only stopped and started it. Did
>> >>> you
>> >>>
>> >>> Daan
>> >>>
>> >>> On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui
>> >>> <ca...@reategui.com>
>> >>> wrote:
>> >>> > Hi Dan,
>> >>> >
>> >>> > Unfortunately not. Still trying to solve this. See my last email
>> >>> > to
>> >>> > Ahmad
>> >>> > in the recover cs thread about next steps I am trying to figure out.
>> >>> >
>> >>> > The CPVM and VR are not running. They are currently in expunging
>> >>> > state.
>> >>> >
>> >>> > If I try to start an instance the MS is unable to communicate the
>> >>> > start
>> >>> > vm
>> >>> > commands to my XS hosts and then starts writing to the logs at an
>> >>> > amazingly
>> >>> > high rate. You can see one of my logs here:
>> >>> > http://reategui.com/cloudstack/management-server.log.new
>> >>> >
>> >>> > thank you
>> >>> > Carlos
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland
>> >>> > <da...@gmail.com>
>> >>> > wrote:
>> >>> >>
>> >>> >> H Carlos,
>> >>> >>
>> >>> >> Did you solve this problem yet?
>> >>> >>
>> >>> >> You can try removing the cpvm and vr at the hypervisor and change
>> >>> >> their state to 'Stopped' in the db.
>> >>> >>
>> >>> >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui
>> >>> >> <ca...@reategui.com>
>> >>> >> wrote:
>> >>> >> > In the last 15 minutes management log file it grew 500MB..
>> >>> >> >
>> >>> >> > Any ideas what I should look at to figure out what is happening?
>> >>> >> >
>> >>> >> >
>> >>> >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui
>> >>> >> > <cr...@gmail.com>wrote:
>> >>> >> >
>> >>> >> >> I am trying to recover my CloudStack installation (CS 4.1.1 on
>> >>> >> >> ubuntu +
>> >>> >> >> XS
>> >>> >> >> 6.0.2) that I shutdown and cannot bring back to life.
>> >>> >> >>
>> >>> >> >> I tried to destroy the CPVM and the VR and all I am seeing in
>> >>> >> >> the
>> >>> >> >> logs
>> >>> >> >> is
>> >>> >> >> the following and they are filling up fast (management log is
>> >>> >> >> almost
>> >>> >> >> 1GB).
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> 2013-10-29 11:51:08,291 DEBUG
>> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >>> >> >> (AgentManager-Handler-14:null) Seq 1-201919186: Forwarding Seq
>> >>> >> >> 1-201919186:
>> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >>> >> >> 100111,
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >>> >> >> } to 233845174730255
>> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
>> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >>> >> >> (AgentManager-Handler-4:null) Seq 2-1168706613: Forwarding Seq
>> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
>> >>> >> >> Flags:
>> >>> >> >> 100111,
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >>> >> >> } to 233845174730255
>> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
>> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >>> >> >> (AgentManager-Handler-2:null) Seq 1-201919186: Forwarding Seq
>> >>> >> >> 1-201919186:
>> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >>> >> >> 100111,
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >>> >> >> } to 233845174730255
>> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
>> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >>> >> >> (AgentManager-Handler-8:null) Seq 2-1168706613: Forwarding Seq
>> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
>> >>> >> >> Flags:
>> >>> >> >> 100111,
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >>> >> >> } to 233845174730255
>> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
>> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >>> >> >> (AgentManager-Handler-9:null) Seq 1-201919186: Forwarding Seq
>> >>> >> >> 1-201919186:
>> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >>> >> >> 100111,
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >>> >> >> } to 233845174730255
>> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
>> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >>> >> >> (AgentManager-Handler-6:null) Seq 2-1168706613: Forwarding Seq
>> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
>> >>> >> >> Flags:
>> >>> >> >> 100111,
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >>> >> >> } to 233845174730255
>> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
>> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >>> >> >> (AgentManager-Handler-15:null) Seq 1-201919186: Forwarding Seq
>> >>> >> >> 1-201919186:
>> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >>> >> >> 100111,
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >>> >> >> } to 233845174730255
>> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
>> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >>> >> >> (AgentManager-Handler-13:null) Seq 2-1168706613: Forwarding Seq
>> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
>> >>> >> >> Flags:
>> >>> >> >> 100111,
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >>> >> >> } to 233845174730255
>> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
>> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >>> >> >> (AgentManager-Handler-10:null) Seq 1-201919186: Forwarding Seq
>> >>> >> >> 1-201919186:
>> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
>> >>> >> >> 100111,
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >>> >> >> } to 233845174730255
>> >>> >> >> 2013-10-29 11:51:08,299 DEBUG
>> >>> >> >> [agent.manager.ClusteredAgentAttache]
>> >>> >> >> (AgentManager-Handler-3:null) Seq 2-1168706613: Forwarding Seq
>> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
>> >>> >> >> Flags:
>> >>> >> >> 100111,
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >>> >> >> } to 233845174730255
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >
>> >>> >
>> >>
>> >>
>> >
>
>
Re: Can't destroy system VMs
Posted by Carlos Reategui <cr...@gmail.com>.
Hi Dan,
The host value is correct and I am able to connect on ports 22,80 and 8080
from the Xen hosts to that IP (same subnet). Port 8787 is not listening on
my MS.
What can I look for in the DB?
thanks
Carlos
On Fri, Nov 1, 2013 at 2:47 AM, Daan Hoogland <da...@gmail.com>wrote:
> H Carlox,
>
> It still seems to me you have a communication problem albeit not at
> the network level. ( I agree this doesn't seem likely) But even so the
> ms can't reach the xen hosts. The next thing to look at would be
> configuration. I presume you did check the ip addresses for the hosts.
> check the value 'host' in the global vars and try to telnet it on port
> 22 80 8080 8787 or what else from the xen hosts . I am sure the
> problem is hidden somewhere in the mysql database.
>
> regards,
> Daan
>
> On Fri, Nov 1, 2013 at 6:23 AM, Carlos Reategui <ca...@reategui.com>
> wrote:
> > Hi Dan,
> > Finally getting around to trying this. I am not able to start any vm
> (mine
> > or system).
> >
> > I am in the process of trying Ahmad's suggestion of removing one of the
> > hosts, reinstalling it and then adding it back in to see if that helps.
> > However I am not able to put it in maintenance mode. I am getting the
> > following when I try:
> >
> >
> > 2013-10-31 15:19:59,120 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-5:null)
> > ===START=== 172.30.40.135 -- GET command=prepareHostForMa
> >
> intenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
> > 2013-10-31 15:19:59,181 DEBUG [cloud.async.AsyncJobManagerImpl]
> > (catalina-exec-5:null) submit async job-167, details: AsyncJobVO {id:16
> > 7, userId: 2, accountId: 2, sessionKey: null, instanceType: Host,
> > instanceId: 1, cmd: org.apache.cloudstack.api.command.admin.host.Prep
> > areForMaintenanceCmd, cmdOriginator: null, cmdInfo:
> >
> {"id":"c2f3c416-f4ba-4f73-b414-6fd6efc734e3","response":"json","sessionkey":"Yjye5q
> >
> mcLtUBW0/P4XCAcx2ZTAI\u003d","ctxUserId":"2","_":"1383258001952","ctxAccountId":"2","ctxStartEventId":"699"},
> > cmdVersion: 0, callbackTy
> > pe: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 233845174730253, completeMsid: null,
> > lastUpdated: null, lastPolled: null, created: null}
> > 2013-10-31 15:19:59,185 DEBUG [cloud.async.AsyncJobManagerImpl]
> > (Job-Executor-1:job-167) Executing org.apache.cloudstack.api.command.ad
> > min.host.PrepareForMaintenanceCmd for job-167
> > 2013-10-31 15:19:59,188 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-5:null)
> > ===END=== 172.30.40.135 -- GET command=prepareHostForMain
> >
> tenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
> > 2013-10-31 15:19:59,207 DEBUG [cloud.cluster.ClusterManagerImpl]
> > (Job-Executor-1:job-167) Propagating agent change request event:AdminA
> > skMaintenace to agent:1
> > 2013-10-31 15:19:59,208 DEBUG [cloud.cluster.ClusterManagerImpl]
> > (Job-Executor-1:job-167) 233845174730253 -> 233845174730255.1 [{"Propa
> >
> gateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
> > 2013-10-31 15:19:59,213 INFO [cloud.cluster.ClusterServiceServletImpl]
> > (Cluster-Worker-1:null) Setup cluster service servlet. service
> > url: http://172.30.45.2:9090/clusterservice, request timeout: 300
> seconds
> > 2013-10-31 15:19:59,214 DEBUG [cloud.cluster.ClusterManagerImpl]
> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 -> 233845174730255
> > . agent: 1, pdu seq: 1, pdu ack seq: 0, json:
> >
> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{
> > },"wait":0}}]
> > 2013-10-31 15:19:59,343 DEBUG [cloud.cluster.ClusterManagerImpl]
> > (Cluster-Worker-7:null) Dispatch ->1, json: [{"PropagateResourceEventC
> >
> ommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
> > 2013-10-31 15:19:59,347 DEBUG [cloud.cluster.ClusterManagerImpl]
> > (Cluster-Worker-7:null) Intercepting command to propagate event AdminA
> > skMaintenace for host 1
> > 2013-10-31 15:19:59,351 DEBUG [cloud.cluster.ClusterServiceServletImpl]
> > (Cluster-Worker-1:null) POST http://172.30.45.2:9090/clusterser
> > vice response :true, responding time: 80 ms
> > 2013-10-31 15:19:59,351 DEBUG [cloud.cluster.ClusterManagerImpl]
> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 -> 233845174730255
> > completed. time: 137ms. agent: 1, pdu seq: 1, pdu ack seq: 0, json:
> > [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMai
> > ntenace","contextMap":{},"wait":0}}]
> > 2013-10-31 15:19:59,356 DEBUG [agent.manager.ClusteredAgentAttache]
> > (Cluster-Worker-7:null) Seq 1-102760483: Forwarding Seq 1-102760483
> > : { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> > 2013-10-31 15:19:59,360 DEBUG [agent.manager.ClusteredAgentAttache]
> > (AgentManager-Handler-1:null) Seq 1-102760483: Forwarding Seq 1-102
> > 760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> > 2013-10-31 15:19:59,365 DEBUG [agent.manager.ClusteredAgentAttache]
> > (AgentManager-Handler-13:null) Seq 1-102760483: Forwarding Seq 1-10
> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> 100111,
> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> > 2013-10-31 15:19:59,369 DEBUG [agent.manager.ClusteredAgentAttache]
> > (AgentManager-Handler-15:null) Seq 1-102760483: Forwarding Seq 1-10
> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> 100111,
> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> > 2013-10-31 15:19:59,372 DEBUG [agent.manager.ClusteredAgentAttache]
> > (AgentManager-Handler-10:null) Seq 1-102760483: Forwarding Seq 1-10
> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> 100111,
> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> > 2013-10-31 15:19:59,376 DEBUG [agent.manager.ClusteredAgentAttache]
> > (AgentManager-Handler-11:null) Seq 1-102760483: Forwarding Seq 1-10
> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> 100111,
> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >
> > These MaintainCommand lines are repeated endlessly and very fast (1GB of
> > logs in about 20 mins) so I currently have the MS stopped.
> >
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Oct 31, 2013 at 3:43 PM, Carlos Reategui <ca...@reategui.com>
> > wrote:
> >>
> >> Hi Dan,
> >> Network is definitely not a problem. iptables is disabled on MS and
> both
> >> hosts. I am able to ssh from MS to both hosts. I am also able to
> connect
> >> to port 80 from the MS.
> >>
> >> I need to head out now and will try what you suggested in a couple
> hours.
> >>
> >> thanks,
> >> Carlos
> >>
> >>
> >> On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland <daan.hoogland@gmail.com
> >
> >> wrote:
> >>>
> >>> Carlos,
> >>>
> >>> can the ms reach the xs hosts?
> >>> if so try this, Stop the ms, set the state for the vm_instances for
> >>> cpvm and vr from expunging to stopped. and restart.
> >>> if not you need to troubleshoot your network connectivity. It seems
> >>> this would not be the issue as you only stopped and started it. Did
> >>> you
> >>>
> >>> Daan
> >>>
> >>> On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui <carlos@reategui.com
> >
> >>> wrote:
> >>> > Hi Dan,
> >>> >
> >>> > Unfortunately not. Still trying to solve this. See my last email to
> >>> > Ahmad
> >>> > in the recover cs thread about next steps I am trying to figure out.
> >>> >
> >>> > The CPVM and VR are not running. They are currently in expunging
> >>> > state.
> >>> >
> >>> > If I try to start an instance the MS is unable to communicate the
> start
> >>> > vm
> >>> > commands to my XS hosts and then starts writing to the logs at an
> >>> > amazingly
> >>> > high rate. You can see one of my logs here:
> >>> > http://reategui.com/cloudstack/management-server.log.new
> >>> >
> >>> > thank you
> >>> > Carlos
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland
> >>> > <da...@gmail.com>
> >>> > wrote:
> >>> >>
> >>> >> H Carlos,
> >>> >>
> >>> >> Did you solve this problem yet?
> >>> >>
> >>> >> You can try removing the cpvm and vr at the hypervisor and change
> >>> >> their state to 'Stopped' in the db.
> >>> >>
> >>> >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui <
> carlos@reategui.com>
> >>> >> wrote:
> >>> >> > In the last 15 minutes management log file it grew 500MB..
> >>> >> >
> >>> >> > Any ideas what I should look at to figure out what is happening?
> >>> >> >
> >>> >> >
> >>> >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui
> >>> >> > <cr...@gmail.com>wrote:
> >>> >> >
> >>> >> >> I am trying to recover my CloudStack installation (CS 4.1.1 on
> >>> >> >> ubuntu +
> >>> >> >> XS
> >>> >> >> 6.0.2) that I shutdown and cannot bring back to life.
> >>> >> >>
> >>> >> >> I tried to destroy the CPVM and the VR and all I am seeing in the
> >>> >> >> logs
> >>> >> >> is
> >>> >> >> the following and they are filling up fast (management log is
> >>> >> >> almost
> >>> >> >> 1GB).
> >>> >> >>
> >>> >> >>
> >>> >> >> 2013-10-29 11:51:08,291 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-14:null) Seq 1-201919186: Forwarding Seq
> >>> >> >> 1-201919186:
> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-4:null) Seq 2-1168706613: Forwarding Seq
> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
> >>> >> >> Flags:
> >>> >> >> 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-2:null) Seq 1-201919186: Forwarding Seq
> >>> >> >> 1-201919186:
> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-8:null) Seq 2-1168706613: Forwarding Seq
> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
> >>> >> >> Flags:
> >>> >> >> 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-9:null) Seq 1-201919186: Forwarding Seq
> >>> >> >> 1-201919186:
> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-6:null) Seq 2-1168706613: Forwarding Seq
> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
> >>> >> >> Flags:
> >>> >> >> 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-15:null) Seq 1-201919186: Forwarding Seq
> >>> >> >> 1-201919186:
> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-13:null) Seq 2-1168706613: Forwarding Seq
> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
> >>> >> >> Flags:
> >>> >> >> 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-10:null) Seq 1-201919186: Forwarding Seq
> >>> >> >> 1-201919186:
> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,299 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-3:null) Seq 2-1168706613: Forwarding Seq
> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
> >>> >> >> Flags:
> >>> >> >> 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >
> >>> >
> >>
> >>
> >
>
Re: Can't destroy system VMs
Posted by Carlos Reategui <ca...@reategui.com>.
Hi Dan,
Finally getting around to trying this. I am not able to start any vm (mine
or system).
I am in the process of trying Ahmad's suggestion of removing one of the
hosts, reinstalling it and then adding it back in to see if that helps.
However I am not able to put it in maintenance mode. I am getting the
following when I try:
2013-10-31 15:19:59,120 DEBUG [cloud.api.ApiServlet] (catalina-exec-5:null)
===START=== 172.30.40.135 -- GET command=prepareHostForMa
intenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
2013-10-31 15:19:59,181 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-5:null) submit async job-167, details: AsyncJobVO {id:16
7, userId: 2, accountId: 2, sessionKey: null, instanceType: Host,
instanceId: 1, cmd: org.apache.cloudstack.api.command.admin.host.Prep
areForMaintenanceCmd, cmdOriginator: null, cmdInfo:
{"id":"c2f3c416-f4ba-4f73-b414-6fd6efc734e3","response":"json","sessionkey":"Yjye5q
mcLtUBW0/P4XCAcx2ZTAI\u003d","ctxUserId":"2","_":"1383258001952","ctxAccountId":"2","ctxStartEventId":"699"},
cmdVersion: 0, callbackTy
pe: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0,
result: null, initMsid: 233845174730253, completeMsid: null,
lastUpdated: null, lastPolled: null, created: null}
2013-10-31 15:19:59,185 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-1:job-167) Executing org.apache.cloudstack.api.command.ad
min.host.PrepareForMaintenanceCmd for job-167
2013-10-31 15:19:59,188 DEBUG [cloud.api.ApiServlet] (catalina-exec-5:null)
===END=== 172.30.40.135 -- GET command=prepareHostForMain
tenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
2013-10-31 15:19:59,207 DEBUG [cloud.cluster.ClusterManagerImpl]
(Job-Executor-1:job-167) Propagating agent change request event:AdminA
skMaintenace to agent:1
2013-10-31 15:19:59,208 DEBUG [cloud.cluster.ClusterManagerImpl]
(Job-Executor-1:job-167) 233845174730253 -> 233845174730255.1 [{"Propa
gateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
2013-10-31 15:19:59,213 INFO [cloud.cluster.ClusterServiceServletImpl]
(Cluster-Worker-1:null) Setup cluster service servlet. service
url: http://172.30.45.2:9090/clusterservice, request timeout: 300 seconds
2013-10-31 15:19:59,214 DEBUG [cloud.cluster.ClusterManagerImpl]
(Cluster-Worker-1:null) Cluster PDU 233845174730253 -> 233845174730255
. agent: 1, pdu seq: 1, pdu ack seq: 0, json:
[{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{
},"wait":0}}]
2013-10-31 15:19:59,343 DEBUG [cloud.cluster.ClusterManagerImpl]
(Cluster-Worker-7:null) Dispatch ->1, json: [{"PropagateResourceEventC
ommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
2013-10-31 15:19:59,347 DEBUG [cloud.cluster.ClusterManagerImpl]
(Cluster-Worker-7:null) Intercepting command to propagate event AdminA
skMaintenace for host 1
2013-10-31 15:19:59,351 DEBUG [cloud.cluster.ClusterServiceServletImpl]
(Cluster-Worker-1:null) POST http://172.30.45.2:9090/clusterser
vice response :true, responding time: 80 ms
2013-10-31 15:19:59,351 DEBUG [cloud.cluster.ClusterManagerImpl]
(Cluster-Worker-1:null) Cluster PDU 233845174730253 -> 233845174730255
completed. time: 137ms. agent: 1, pdu seq: 1, pdu ack seq: 0, json:
[{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMai
ntenace","contextMap":{},"wait":0}}]
2013-10-31 15:19:59,356 DEBUG [agent.manager.ClusteredAgentAttache]
(Cluster-Worker-7:null) Seq 1-102760483: Forwarding Seq 1-102760483
: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
[{"MaintainCommand":{"wait":0}}] } to 233845174730255
2013-10-31 15:19:59,360 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-1:null) Seq 1-102760483: Forwarding Seq 1-102
760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
[{"MaintainCommand":{"wait":0}}] } to 233845174730255
2013-10-31 15:19:59,365 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-13:null) Seq 1-102760483: Forwarding Seq 1-10
2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
[{"MaintainCommand":{"wait":0}}] } to 233845174730255
2013-10-31 15:19:59,369 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-15:null) Seq 1-102760483: Forwarding Seq 1-10
2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
[{"MaintainCommand":{"wait":0}}] } to 233845174730255
2013-10-31 15:19:59,372 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-10:null) Seq 1-102760483: Forwarding Seq 1-10
2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
[{"MaintainCommand":{"wait":0}}] } to 233845174730255
2013-10-31 15:19:59,376 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-11:null) Seq 1-102760483: Forwarding Seq 1-10
2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
[{"MaintainCommand":{"wait":0}}] } to 233845174730255
These MaintainCommand lines are repeated endlessly and very fast (1GB of
logs in about 20 mins) so I currently have the MS stopped.
On Thu, Oct 31, 2013 at 3:43 PM, Carlos Reategui <ca...@reategui.com>wrote:
> Hi Dan,
> Network is definitely not a problem. iptables is disabled on MS and both
> hosts. I am able to ssh from MS to both hosts. I am also able to connect
> to port 80 from the MS.
>
> I need to head out now and will try what you suggested in a couple hours.
>
> thanks,
> Carlos
>
>
> On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland <da...@gmail.com>wrote:
>
>> Carlos,
>>
>> can the ms reach the xs hosts?
>> if so try this, Stop the ms, set the state for the vm_instances for
>> cpvm and vr from expunging to stopped. and restart.
>> if not you need to troubleshoot your network connectivity. It seems
>> this would not be the issue as you only stopped and started it. Did
>> you
>>
>> Daan
>>
>> On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui <ca...@reategui.com>
>> wrote:
>> > Hi Dan,
>> >
>> > Unfortunately not. Still trying to solve this. See my last email to
>> Ahmad
>> > in the recover cs thread about next steps I am trying to figure out.
>> >
>> > The CPVM and VR are not running. They are currently in expunging state.
>> >
>> > If I try to start an instance the MS is unable to communicate the start
>> vm
>> > commands to my XS hosts and then starts writing to the logs at an
>> amazingly
>> > high rate. You can see one of my logs here:
>> > http://reategui.com/cloudstack/management-server.log.new
>> >
>> > thank you
>> > Carlos
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland <daan.hoogland@gmail.com
>> >
>> > wrote:
>> >>
>> >> H Carlos,
>> >>
>> >> Did you solve this problem yet?
>> >>
>> >> You can try removing the cpvm and vr at the hypervisor and change
>> >> their state to 'Stopped' in the db.
>> >>
>> >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui <ca...@reategui.com>
>> >> wrote:
>> >> > In the last 15 minutes management log file it grew 500MB..
>> >> >
>> >> > Any ideas what I should look at to figure out what is happening?
>> >> >
>> >> >
>> >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui
>> >> > <cr...@gmail.com>wrote:
>> >> >
>> >> >> I am trying to recover my CloudStack installation (CS 4.1.1 on
>> ubuntu +
>> >> >> XS
>> >> >> 6.0.2) that I shutdown and cannot bring back to life.
>> >> >>
>> >> >> I tried to destroy the CPVM and the VR and all I am seeing in the
>> logs
>> >> >> is
>> >> >> the following and they are filling up fast (management log is almost
>> >> >> 1GB).
>> >> >>
>> >> >>
>> >> >> 2013-10-29 11:51:08,291 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-14:null) Seq 1-201919186: Forwarding Seq
>> >> >> 1-201919186:
>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,293 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-4:null) Seq 2-1168706613: Forwarding Seq
>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
>> Flags:
>> >> >> 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,293 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-2:null) Seq 1-201919186: Forwarding Seq
>> >> >> 1-201919186:
>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,294 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-8:null) Seq 2-1168706613: Forwarding Seq
>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
>> Flags:
>> >> >> 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,294 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-9:null) Seq 1-201919186: Forwarding Seq
>> >> >> 1-201919186:
>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,296 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-6:null) Seq 2-1168706613: Forwarding Seq
>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
>> Flags:
>> >> >> 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,296 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-15:null) Seq 1-201919186: Forwarding Seq
>> >> >> 1-201919186:
>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,297 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-13:null) Seq 2-1168706613: Forwarding Seq
>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
>> Flags:
>> >> >> 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,297 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-10:null) Seq 1-201919186: Forwarding Seq
>> >> >> 1-201919186:
>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >> 2013-10-29 11:51:08,299 DEBUG [agent.manager.ClusteredAgentAttache]
>> >> >> (AgentManager-Handler-3:null) Seq 2-1168706613: Forwarding Seq
>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
>> Flags:
>> >> >> 100111,
>> >> >>
>> >> >>
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> >> >> } to 233845174730255
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >
>> >
>>
>
>
Re: Can't destroy system VMs
Posted by Carlos Reategui <ca...@reategui.com>.
Hi Dan,
Network is definitely not a problem. iptables is disabled on MS and both
hosts. I am able to ssh from MS to both hosts. I am also able to connect
to port 80 from the MS.
I need to head out now and will try what you suggested in a couple hours.
thanks,
Carlos
On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland <da...@gmail.com>wrote:
> Carlos,
>
> can the ms reach the xs hosts?
> if so try this, Stop the ms, set the state for the vm_instances for
> cpvm and vr from expunging to stopped. and restart.
> if not you need to troubleshoot your network connectivity. It seems
> this would not be the issue as you only stopped and started it. Did
> you
>
> Daan
>
> On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui <ca...@reategui.com>
> wrote:
> > Hi Dan,
> >
> > Unfortunately not. Still trying to solve this. See my last email to
> Ahmad
> > in the recover cs thread about next steps I am trying to figure out.
> >
> > The CPVM and VR are not running. They are currently in expunging state.
> >
> > If I try to start an instance the MS is unable to communicate the start
> vm
> > commands to my XS hosts and then starts writing to the logs at an
> amazingly
> > high rate. You can see one of my logs here:
> > http://reategui.com/cloudstack/management-server.log.new
> >
> > thank you
> > Carlos
> >
> >
> >
> >
> >
> > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland <da...@gmail.com>
> > wrote:
> >>
> >> H Carlos,
> >>
> >> Did you solve this problem yet?
> >>
> >> You can try removing the cpvm and vr at the hypervisor and change
> >> their state to 'Stopped' in the db.
> >>
> >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui <ca...@reategui.com>
> >> wrote:
> >> > In the last 15 minutes management log file it grew 500MB..
> >> >
> >> > Any ideas what I should look at to figure out what is happening?
> >> >
> >> >
> >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui
> >> > <cr...@gmail.com>wrote:
> >> >
> >> >> I am trying to recover my CloudStack installation (CS 4.1.1 on
> ubuntu +
> >> >> XS
> >> >> 6.0.2) that I shutdown and cannot bring back to life.
> >> >>
> >> >> I tried to destroy the CPVM and the VR and all I am seeing in the
> logs
> >> >> is
> >> >> the following and they are filling up fast (management log is almost
> >> >> 1GB).
> >> >>
> >> >>
> >> >> 2013-10-29 11:51:08,291 DEBUG [agent.manager.ClusteredAgentAttache]
> >> >> (AgentManager-Handler-14:null) Seq 1-201919186: Forwarding Seq
> >> >> 1-201919186:
> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >> >>
> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> } to 233845174730255
> >> >> 2013-10-29 11:51:08,293 DEBUG [agent.manager.ClusteredAgentAttache]
> >> >> (AgentManager-Handler-4:null) Seq 2-1168706613: Forwarding Seq
> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
> Flags:
> >> >> 100111,
> >> >>
> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> } to 233845174730255
> >> >> 2013-10-29 11:51:08,293 DEBUG [agent.manager.ClusteredAgentAttache]
> >> >> (AgentManager-Handler-2:null) Seq 1-201919186: Forwarding Seq
> >> >> 1-201919186:
> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >> >>
> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> } to 233845174730255
> >> >> 2013-10-29 11:51:08,294 DEBUG [agent.manager.ClusteredAgentAttache]
> >> >> (AgentManager-Handler-8:null) Seq 2-1168706613: Forwarding Seq
> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
> Flags:
> >> >> 100111,
> >> >>
> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> } to 233845174730255
> >> >> 2013-10-29 11:51:08,294 DEBUG [agent.manager.ClusteredAgentAttache]
> >> >> (AgentManager-Handler-9:null) Seq 1-201919186: Forwarding Seq
> >> >> 1-201919186:
> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >> >>
> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> } to 233845174730255
> >> >> 2013-10-29 11:51:08,296 DEBUG [agent.manager.ClusteredAgentAttache]
> >> >> (AgentManager-Handler-6:null) Seq 2-1168706613: Forwarding Seq
> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
> Flags:
> >> >> 100111,
> >> >>
> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> } to 233845174730255
> >> >> 2013-10-29 11:51:08,296 DEBUG [agent.manager.ClusteredAgentAttache]
> >> >> (AgentManager-Handler-15:null) Seq 1-201919186: Forwarding Seq
> >> >> 1-201919186:
> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >> >>
> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> } to 233845174730255
> >> >> 2013-10-29 11:51:08,297 DEBUG [agent.manager.ClusteredAgentAttache]
> >> >> (AgentManager-Handler-13:null) Seq 2-1168706613: Forwarding Seq
> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
> Flags:
> >> >> 100111,
> >> >>
> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> } to 233845174730255
> >> >> 2013-10-29 11:51:08,297 DEBUG [agent.manager.ClusteredAgentAttache]
> >> >> (AgentManager-Handler-10:null) Seq 1-201919186: Forwarding Seq
> >> >> 1-201919186:
> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >> >>
> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> } to 233845174730255
> >> >> 2013-10-29 11:51:08,299 DEBUG [agent.manager.ClusteredAgentAttache]
> >> >> (AgentManager-Handler-3:null) Seq 2-1168706613: Forwarding Seq
> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
> Flags:
> >> >> 100111,
> >> >>
> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >> } to 233845174730255
> >> >>
> >> >>
> >> >>
> >> >>
> >
> >
>
Re: Can't destroy system VMs
Posted by Carlos Reategui <ca...@reategui.com>.
Hi Dan,
Unfortunately not. Still trying to solve this. See my last email to Ahmad
in the recover cs thread about next steps I am trying to figure out.
The CPVM and VR are not running. They are currently in expunging state.
If I try to start an instance the MS is unable to communicate the start vm
commands to my XS hosts and then starts writing to the logs at an amazingly
high rate. You can see one of my logs here:
http://reategui.com/cloudstack/management-server.log.new
thank you
Carlos
On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland <da...@gmail.com>wrote:
> H Carlos,
>
> Did you solve this problem yet?
>
> You can try removing the cpvm and vr at the hypervisor and change
> their state to 'Stopped' in the db.
>
> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui <ca...@reategui.com>
> wrote:
> > In the last 15 minutes management log file it grew 500MB..
> >
> > Any ideas what I should look at to figure out what is happening?
> >
> >
> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui <creategui@gmail.com
> >wrote:
> >
> >> I am trying to recover my CloudStack installation (CS 4.1.1 on ubuntu +
> XS
> >> 6.0.2) that I shutdown and cannot bring back to life.
> >>
> >> I tried to destroy the CPVM and the VR and all I am seeing in the logs
> is
> >> the following and they are filling up fast (management log is almost
> 1GB).
> >>
> >>
> >> 2013-10-29 11:51:08,291 DEBUG [agent.manager.ClusteredAgentAttache]
> >> (AgentManager-Handler-14:null) Seq 1-201919186: Forwarding Seq
> 1-201919186:
> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> } to 233845174730255
> >> 2013-10-29 11:51:08,293 DEBUG [agent.manager.ClusteredAgentAttache]
> >> (AgentManager-Handler-4:null) Seq 2-1168706613: Forwarding Seq
> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
> >> 100111,
> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> } to 233845174730255
> >> 2013-10-29 11:51:08,293 DEBUG [agent.manager.ClusteredAgentAttache]
> >> (AgentManager-Handler-2:null) Seq 1-201919186: Forwarding Seq
> 1-201919186:
> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> } to 233845174730255
> >> 2013-10-29 11:51:08,294 DEBUG [agent.manager.ClusteredAgentAttache]
> >> (AgentManager-Handler-8:null) Seq 2-1168706613: Forwarding Seq
> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
> >> 100111,
> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> } to 233845174730255
> >> 2013-10-29 11:51:08,294 DEBUG [agent.manager.ClusteredAgentAttache]
> >> (AgentManager-Handler-9:null) Seq 1-201919186: Forwarding Seq
> 1-201919186:
> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> } to 233845174730255
> >> 2013-10-29 11:51:08,296 DEBUG [agent.manager.ClusteredAgentAttache]
> >> (AgentManager-Handler-6:null) Seq 2-1168706613: Forwarding Seq
> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
> >> 100111,
> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> } to 233845174730255
> >> 2013-10-29 11:51:08,296 DEBUG [agent.manager.ClusteredAgentAttache]
> >> (AgentManager-Handler-15:null) Seq 1-201919186: Forwarding Seq
> 1-201919186:
> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> } to 233845174730255
> >> 2013-10-29 11:51:08,297 DEBUG [agent.manager.ClusteredAgentAttache]
> >> (AgentManager-Handler-13:null) Seq 2-1168706613: Forwarding Seq
> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
> >> 100111,
> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> } to 233845174730255
> >> 2013-10-29 11:51:08,297 DEBUG [agent.manager.ClusteredAgentAttache]
> >> (AgentManager-Handler-10:null) Seq 1-201919186: Forwarding Seq
> 1-201919186:
> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> } to 233845174730255
> >> 2013-10-29 11:51:08,299 DEBUG [agent.manager.ClusteredAgentAttache]
> >> (AgentManager-Handler-3:null) Seq 2-1168706613: Forwarding Seq
> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
> >> 100111,
> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> } to 233845174730255
> >>
> >>
> >>
> >>
>
Re: Can't destroy system VMs
Posted by Daan Hoogland <da...@gmail.com>.
H Carlos,
Did you solve this problem yet?
You can try removing the cpvm and vr at the hypervisor and change
their state to 'Stopped' in the db.
On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui <ca...@reategui.com> wrote:
> In the last 15 minutes management log file it grew 500MB..
>
> Any ideas what I should look at to figure out what is happening?
>
>
> On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui <cr...@gmail.com>wrote:
>
>> I am trying to recover my CloudStack installation (CS 4.1.1 on ubuntu + XS
>> 6.0.2) that I shutdown and cannot bring back to life.
>>
>> I tried to destroy the CPVM and the VR and all I am seeing in the logs is
>> the following and they are filling up fast (management log is almost 1GB).
>>
>>
>> 2013-10-29 11:51:08,291 DEBUG [agent.manager.ClusteredAgentAttache]
>> (AgentManager-Handler-14:null) Seq 1-201919186: Forwarding Seq 1-201919186:
>> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> } to 233845174730255
>> 2013-10-29 11:51:08,293 DEBUG [agent.manager.ClusteredAgentAttache]
>> (AgentManager-Handler-4:null) Seq 2-1168706613: Forwarding Seq
>> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
>> 100111,
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> } to 233845174730255
>> 2013-10-29 11:51:08,293 DEBUG [agent.manager.ClusteredAgentAttache]
>> (AgentManager-Handler-2:null) Seq 1-201919186: Forwarding Seq 1-201919186:
>> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> } to 233845174730255
>> 2013-10-29 11:51:08,294 DEBUG [agent.manager.ClusteredAgentAttache]
>> (AgentManager-Handler-8:null) Seq 2-1168706613: Forwarding Seq
>> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
>> 100111,
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> } to 233845174730255
>> 2013-10-29 11:51:08,294 DEBUG [agent.manager.ClusteredAgentAttache]
>> (AgentManager-Handler-9:null) Seq 1-201919186: Forwarding Seq 1-201919186:
>> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> } to 233845174730255
>> 2013-10-29 11:51:08,296 DEBUG [agent.manager.ClusteredAgentAttache]
>> (AgentManager-Handler-6:null) Seq 2-1168706613: Forwarding Seq
>> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
>> 100111,
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> } to 233845174730255
>> 2013-10-29 11:51:08,296 DEBUG [agent.manager.ClusteredAgentAttache]
>> (AgentManager-Handler-15:null) Seq 1-201919186: Forwarding Seq 1-201919186:
>> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> } to 233845174730255
>> 2013-10-29 11:51:08,297 DEBUG [agent.manager.ClusteredAgentAttache]
>> (AgentManager-Handler-13:null) Seq 2-1168706613: Forwarding Seq
>> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
>> 100111,
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> } to 233845174730255
>> 2013-10-29 11:51:08,297 DEBUG [agent.manager.ClusteredAgentAttache]
>> (AgentManager-Handler-10:null) Seq 1-201919186: Forwarding Seq 1-201919186:
>> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
>> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> } to 233845174730255
>> 2013-10-29 11:51:08,299 DEBUG [agent.manager.ClusteredAgentAttache]
>> (AgentManager-Handler-3:null) Seq 2-1168706613: Forwarding Seq
>> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
>> 100111,
>> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
>> } to 233845174730255
>>
>>
>>
>>
Re: Can't destroy system VMs
Posted by Carlos Reategui <ca...@reategui.com>.
In the last 15 minutes management log file it grew 500MB..
Any ideas what I should look at to figure out what is happening?
On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui <cr...@gmail.com>wrote:
> I am trying to recover my CloudStack installation (CS 4.1.1 on ubuntu + XS
> 6.0.2) that I shutdown and cannot bring back to life.
>
> I tried to destroy the CPVM and the VR and all I am seeing in the logs is
> the following and they are filling up fast (management log is almost 1GB).
>
>
> 2013-10-29 11:51:08,291 DEBUG [agent.manager.ClusteredAgentAttache]
> (AgentManager-Handler-14:null) Seq 1-201919186: Forwarding Seq 1-201919186:
> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> } to 233845174730255
> 2013-10-29 11:51:08,293 DEBUG [agent.manager.ClusteredAgentAttache]
> (AgentManager-Handler-4:null) Seq 2-1168706613: Forwarding Seq
> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
> 100111,
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> } to 233845174730255
> 2013-10-29 11:51:08,293 DEBUG [agent.manager.ClusteredAgentAttache]
> (AgentManager-Handler-2:null) Seq 1-201919186: Forwarding Seq 1-201919186:
> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> } to 233845174730255
> 2013-10-29 11:51:08,294 DEBUG [agent.manager.ClusteredAgentAttache]
> (AgentManager-Handler-8:null) Seq 2-1168706613: Forwarding Seq
> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
> 100111,
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> } to 233845174730255
> 2013-10-29 11:51:08,294 DEBUG [agent.manager.ClusteredAgentAttache]
> (AgentManager-Handler-9:null) Seq 1-201919186: Forwarding Seq 1-201919186:
> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> } to 233845174730255
> 2013-10-29 11:51:08,296 DEBUG [agent.manager.ClusteredAgentAttache]
> (AgentManager-Handler-6:null) Seq 2-1168706613: Forwarding Seq
> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
> 100111,
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> } to 233845174730255
> 2013-10-29 11:51:08,296 DEBUG [agent.manager.ClusteredAgentAttache]
> (AgentManager-Handler-15:null) Seq 1-201919186: Forwarding Seq 1-201919186:
> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> } to 233845174730255
> 2013-10-29 11:51:08,297 DEBUG [agent.manager.ClusteredAgentAttache]
> (AgentManager-Handler-13:null) Seq 2-1168706613: Forwarding Seq
> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
> 100111,
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> } to 233845174730255
> 2013-10-29 11:51:08,297 DEBUG [agent.manager.ClusteredAgentAttache]
> (AgentManager-Handler-10:null) Seq 1-201919186: Forwarding Seq 1-201919186:
> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> } to 233845174730255
> 2013-10-29 11:51:08,299 DEBUG [agent.manager.ClusteredAgentAttache]
> (AgentManager-Handler-3:null) Seq 2-1168706613: Forwarding Seq
> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1, Flags:
> 100111,
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> } to 233845174730255
>
>
>
>