You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by to...@163.com on 2015/07/20 08:26:38 UTC
XenServer is disconnected after CS hosts shutdown
Hello,
I shutdown my cs-manager and xenserver last weekend, And now the ssvm
and cpvm is disconnect, thost two was runing on xenserver. so What
should i do right now ?
Please anybody help me and thanks.
In xenserver I found that the three system VMs are not running.
my xenserver seems can not reconnect to CS-manager. and it seams not
under control of CS.
What is the right steps of shutdown all CS group machines and resume them?
How can i let my xenserver reconnected ?
Thanks,
--
-----------
Cao Tong
Fwd: Re: XenServer is disconnected after CS hosts shutdown
Posted by to...@163.com.
copy mail thread to @dev for seeking more help.
-------- Forwarded Message --------
Subject: Re: XenServer is disconnected after CS hosts shutdown
Date: Wed, 22 Jul 2015 21:03:13 +0800
From: tony_caotong@163.com
Reply-To: users@cloudstack.apache.org
To: users@cloudstack.apache.org, opsrunbook@gmail.com
Hey! help please...
some news.
I think the cause is that the ACS host can't communicate with XenServer
host.
ACS continues outputing logs like this
2015-07-22 20:42:13,555 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-7:null) Seq 5-8174877748607582212: Forwarding Seq
5-8174877748607582212: { Cmd , MgmtId: 279278805451459, via: 5, Ver:
v1, Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
} to 280345368052992
I am not sure that if the ACS status is wrong or some services on
xenserver are not opend.
on xenserver , I found *xenheartbeat.sh is not running.*
*(/bin/bash /opt/cloud/bin/xenheartbeat.sh
00d8e0d0-8561-4b3d-9044-cbc496ff22cc 120 60)*
As some operations about xenserver was pending， xenserver can not be
deleted from web UI.
I got a temporary solution
1. delete jobs from DB cloud.vm_work_job.
2. delete xenserver from DB cloud.host.
3. add xenserver host back from web UI.
then it works.
Does anyone have a idea for this?
Could anyone tell what things does ACS do on xenserver host when adding
a xenserver ?
Thanks,
-----------
Cao Tong
On 07/22/2015 04:26 PM, tony_caotong@163.com wrote:
>
> @prashant, following it the answer of you questions
>
> 1. Yes, primary storage is connected fine for my xenserver.
>
> 2. No, Xenserver's password is not changed.
>
> 3. yes, web UI is fine, and I can login.
>
> 4. before reboot, I unmanaged and disabled resources, and after
> reboot I have enabled all of them.
>
> 5. hosts is states is UP.
>
> 6. No yum update in anywhere.
>
> 7. system VMs status is fine, i think.
>
> -----------
> Cao Tong
>
> On 07/22/2015 04:13 PM, tony_caotong@163.com wrote:
>>
>> Hi,
>>
>> After reinstall, I got the problem again
>>
>> So, I will describe once again.
>>
>> WHAT my environment looks like:
>>
>> I have a ACS server host and a xenserver host, After both reboot, I
>> can not create a VM on xenserver through ACS.
>> A KVM and A NFS are running together in ACS manager host.
>>
>> the status of new VM is always 'staring' on the WEB, but I can create
>> new VM using xencenter.
>>
>> ------------- ERR LOGS ----------
>> 2015-07-22 15:56:56,357 DEBUG [c.c.s.StorageManagerImpl]
>> (StatsCollector-3:ctx-1aa2e8c9) Unable to send storage pool command
>> to Pool[4|NetworkFilesystem] via 4
>> com.cloud.exception.OperationTimedoutException: Commands
>> 2829104990918803478 to Host 4 timed out after 3600
>>
>> 2015-07-22 15:56:56,358 INFO [c.c.s.StatsCollector]
>> (StatsCollector-3:ctx-1aa2e8c9) Unable to reach
>> Pool[4|NetworkFilesystem]
>> com.cloud.exception.StorageUnavailableException: Resource
>> [StoragePool:4] is unreachable: Unable to send command to the pool
>>
>>
>> ------------- and there are lots of DEBUG infos ------- repeat again
>> and again -----------
>>
>> 2015-07-22 15:36:12,887 DEBUG [c.c.a.m.ClusteredAgentAttache]
>> (AgentManager-Handler-14:null) Seq 4-8064821032713715922: Forwarding
>> Seq 4-8064821032713715922: { Cmd , MgmtId: 227448510156211, via: 4,
>> Ver: v1, Flags: 100111,
>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>> 116784073679673
>> 2015-07-22 15:36:12,889 DEBUG [c.c.a.m.ClusteredAgentAttache]
>> (AgentManager-Handler-10:null) Seq 4-8064821032713715883: Forwarding
>> Seq 4-8064821032713715883: { Cmd , MgmtId: 227448510156211, via: 4,
>> Ver: v1, Flags: 100111,
>> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/5/af949612-838f-3a6d-931b-312e612db740.vhd","origUrl":"http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>> 5.6(64-bit) no GUI
>> (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.0.0.100/storage/secondary","_role":"Image"}},"name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>> 5.6(64-bit) no GUI
>> (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2df26406-31bf-3a95-8a61-f5008defd9a0","id":4,"poolType":"NetworkFilesystem","host":"10.0.0.100","path":"/storage/xen/primary","port":2049,"url":"NetworkFilesystem://10.0.0.100/storage/xen/primary/?ROLE=Primary&STOREUUID=2df26406-31bf-3a95-8a61-f5008defd9a0"}},"name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"executeInSequence":true,"options":{},"wait":10800}}]
>> } to 116784073679673
>>
>>
>> -----------------------------------------
>>
>> Anyone have Any ideas? thanks.
>>
>> -----------
>> Cao Tong
>>
>> On 07/21/2015 06:14 PM, tony_caotong@163.com wrote:
>>>
>>> Thanks all,
>>>
>>> I have already reinstall my hosts for preparing a new clear
>>> environment to restart my research.
>>>
>>> -----------
>>> Cao Tong
>>>
>>> On 07/20/2015 09:24 PM, Prashant s wrote:
>>>> some questions :
>>>>
>>>> can you please tell ...
>>>>
>>>> 1. is your NFS storage or your primary Storage Repository in connected
>>>> mode with no red cross mark on them in xencenter.
>>>> 2. did you change any passwords on the xenservers ?
>>>> 3. is the cloudstack web ui up , can you login to the cloudstack
>>>> web page.
>>>> 4. *are the zone , pod, or clusters in unmanaged or disabled state ? *
>>>> *5. is all the hosts in connected state ? *
>>>> *6. did you run yum update on host reboot on the cs manager vm ? *
>>>> *7. system vms are stateless you can kill them and cs will recreate
>>>> a new
>>>> one .. so dont worry :-) *
>>>>
>>>>
>>>> *thanks *
>>>> *prashant *
>>>>
>>>>
>>>>
>>>> On Mon, Jul 20, 2015 at 3:47 AM, <to...@163.com> wrote:
>>>>
>>>>> Hi, I restartd All hosts (one mgr and xenserver) again.
>>>>>
>>>>>
>>>>> Following is the error log.
>>>>>
>>>>>
>>>>> 2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
>>>>> (StatsCollector-3:ctx-692a5392) Could not find exception:
>>>>> com.cloud.exception.OperationTimedoutException in error code list for
>>>>> exceptions
>>>>> 2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed
>>>>> out on null
>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972:
>>>>> Cancelling.
>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
>>>>> (StatsCollector-3:ctx-692a5392) Unable to send storage pool
>>>>> command to
>>>>> Pool[1|NetworkFilesystem] via 1
>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>> 3176445112179752972 to Host 1 timed out after 3600
>>>>> at
>>>>> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
>>>>> at
>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:433)
>>>>>
>>>>> at
>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
>>>>>
>>>>> at
>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1000)
>>>>>
>>>>> at
>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
>>>>>
>>>>> at
>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
>>>>>
>>>>> at
>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>
>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>
>>>>> at
>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>> 2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
>>>>> (StatsCollector-3:ctx-692a5392) Unable to reach
>>>>> Pool[1|NetworkFilesystem]
>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>> [StoragePool:1]
>>>>> is unreachable: Unable to send command to the pool
>>>>> at
>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1010)
>>>>>
>>>>> at
>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
>>>>>
>>>>> at
>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
>>>>>
>>>>> at
>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>
>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>
>>>>> at
>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> -----------
>>>>> Cao Tong
>>>>>
>>>>>
>>>>> On 07/20/2015 02:52 PM, tony_caotong@163.com wrote:
>>>>>
>>>>>> No, no one's IP was changed.
>>>>>>
>>>>>> 1. In xenserver I can not login systemvms using the internal IP like
>>>>>> '169.254.1.112', There shoud be a bridge network for this
>>>>>> right? it is
>>>>>> gone.
>>>>>>
>>>>>> 2. I try to delete xenserver host from CS on web, it also failed
>>>>>> with
>>>>>> lots of logs like following, then memory is full and mangement
>>>>>> down...
>>>>>>
>>>>>> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>> (AgentManager-Handler-15:null) Seq 1-7282039122481381399:
>>>>>> Forwarding Seq
>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>> Ver: v1,
>>>>>> Flags: 100111,
>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>> 192405008094602
>>>>>> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>> (AgentManager-Handler-5:null) Seq 1-7282039122481381399:
>>>>>> Forwarding Seq
>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>> Ver: v1,
>>>>>> Flags: 100111,
>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>> 192405008094602
>>>>>> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>> (AgentManager-Handler-1:null) Seq 1-7282039122481381399:
>>>>>> Forwarding Seq
>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>> Ver: v1,
>>>>>> Flags: 100111,
>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>> 192405008094602
>>>>>> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>> (AgentManager-Handler-14:null) Seq 1-7282039122481381399:
>>>>>> Forwarding Seq
>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>> Ver: v1,
>>>>>> Flags: 100111,
>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>> 192405008094602
>>>>>>
>>>>>>
>>>>>> I guess that, is there some service or daemons working for CS is
>>>>>> not up
>>>>>> on Xenserver ?
>>>>>>
>>>>>>
>>>>>> -----------
>>>>>> Cao Tong
>>>>>> On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
>>>>>>
>>>>>>> Did the management server ip change?
>>>>>>> management server ip in the configuration table is used my
>>>>>>> systemvms.
>>>>>>> select * from configuration where name like 'host';
>>>>>>>
>>>>>>> If it changed, correct the value in db and restart systemvms.
>>>>>>>
>>>>>>>
>>>>>>> ~Rajani
>>>>>>>
>>>>>>> On Mon, Jul 20, 2015 at 11:56 AM,<to...@163.com> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>> I shutdown my cs-manager and xenserver last weekend, And now
>>>>>>>> the ssvm
>>>>>>>> and cpvm is disconnect, thost two was runing on xenserver. so What
>>>>>>>> should i do right now ?
>>>>>>>> Please anybody help me and thanks.
>>>>>>>>
>>>>>>>> In xenserver I found that the three system VMs are not running.
>>>>>>>> my xenserver seems can not reconnect to CS-manager. and it
>>>>>>>> seams not
>>>>>>>> under control of CS.
>>>>>>>>
>>>>>>>>
>>>>>>>> What is the right steps of shutdown all CS group machines and
>>>>>>>> resume
>>>>>>>> them?
>>>>>>>> How can i let my xenserver reconnected ?
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> --
>>>>>>>> -----------
>>>>>>>> Cao Tong
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>
>>>
>>
>>
>
>
RE: XenServer is disconnected after CS hosts shutdown
Posted by Somesh Naidu <So...@citrix.com>.
> starting XS agent manually
Either unmanage/manage cluster or perform a force reconnect to the host.
Regards,
Somesh
-----Original Message-----
From: tony_caotong@163.com [mailto:tony_caotong@163.com]
Sent: Wednesday, August 12, 2015 5:14 AM
To: users@cloudstack.apache.org
Subject: Re: XenServer is disconnected after CS hosts shutdown
After I disabled and enabled XS cluster & primary. it works.
seems XS agent have started by ACS host. so what is the correct
behavior of starting XS agent manually ?
-----------
Cao Tong
On 08/12/2015 04:41 PM, tony_caotong@163.com wrote:
>
> Hello,
>
> Almost one month gone, my problem is still here. I really really need
> someone to help me.
>
> new Settings ACS 4.4.4 XS 6.2
> After reboot, errors here:
>
> 2015-08-12 16:18:54,557 INFO [c.c.a.t.Request]
> (StatsCollector-3:ctx-b287085a) not building log message for '[{}]',
> _cmds.length == 1
> 2015-08-12 16:18:54,557 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Forwarding
> null to 191386435611186
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-12:null) Seq 9-4450963806725603425: Routing from
> 249082151178140
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-12:null) Seq 9-4450963806725603425: Link is closed
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-12:null) Seq 9-4450963806725603425: MgmtId
> 249082151178140: Req: Resource [Host:9] is unreachable: Host 9: Link
> is closed
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-12:null) Seq 9--1: MgmtId 249082151178140: Req:
> Routing to peer
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-15:null) Seq 9--1: MgmtId 249082151178140: Req:
> Cancel request received
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
> (AgentManager-Handler-15:null) Seq 9-4450963806725603425: Cancelling.
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Waiting
> some more time because this is the current command
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Waiting
> some more time because this is the current command
> 2015-08-12 16:18:54,559 INFO [c.c.u.e.CSExceptionErrorCode]
> (StatsCollector-3:ctx-b287085a) Could not find exception:
> com.cloud.exception.OperationTimedoutException in error code list for
> exceptions
> 2015-08-12 16:18:54,559 INFO [c.c.a.t.Request]
> (StatsCollector-3:ctx-b287085a) not building log message for '[{}]',
> _cmds.length == 1
> 2015-08-12 16:18:54,559 WARN [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Timed out
> on null
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Cancelling.
> 2015-08-12 16:18:54,559 DEBUG [c.c.s.StorageManagerImpl]
> (StatsCollector-3:ctx-b287085a) Unable to send storage pool command to
> Pool[3|NetworkFilesystem] via 9
> com.cloud.exception.OperationTimedoutException: Commands
> 4450963806725603425 to Host 9 timed out after 3600
> at
> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:434)
> at
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:418)
> at
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:965)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:390)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:404)
> at
> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
>
>
>
> -----------
> Cao Tong
>
> On 07/30/2015 10:55 AM, tony_caotong@163.com wrote:
>>
>> Hi Yiping, Thanks for your reply first.
>>
>>
>> My NFS server deployed in ACS manager host, another host is a single
>> xenserver.
>> A KVM environment deployed in ACS manager host too.
>> there is three storages named, Kprimary, Xprimary, Secondary.
>>
>> First, I add KVM cluster with zone-wide Kprimary, it works fine even
>> if after reboot.
>> Second, I add Xenserver with Xprimary(cluster-wide), it still works
>> fine.
>> Then, I set zone to disable, shutdown all system VMs, disable
>> those two hosts. finally shutdown xenserver host.
>> When xenserver's shutdown finish, I stop those services by order (
>> cloudstack-agent, cloudstack-management, libvirtd, nfs, rpcbind,
>> mysqld).
>> At last shutdown ACS host.
>>
>> The start process is totally reverse of this order.
>>
>> when done of starting, ACS says unable to send cmd to pool via host
>> xenserver.
>>
>> I am sure nothing have change in my environment during reboot time.
>> When I fix this problem, the only change is change Kprimary from
>> zone-wide to cluster-wide.
>>
>> I guess that after reboot some status have been initial from
>> beginning, ACS found that Xenserver host have two primary can be used.
>> so it use the high priority one， and it is Kprimary.
>>
>> Whatever, Maybe it could help you peoples to get ACS better.
>>
>> BTW, Error logs attached some days before is already point out this
>> is a storage problem
>> like:
>>
>> Unable to send storage pool command
>> to Pool[4|NetworkFilesystem] via 4
>>
>>
>>
>> -----------
>> Cao Tong
>>
>> On 07/30/2015 12:44 AM, Yiping Zhang wrote:
>>> Well, sometimes people can’t answer a question because of lack of
>>> relevant information, or simply because no one has encountered a
>>> similar
>>> situation before.
>>>
>>> Looking at your past messages on this thread, there were no mentions
>>> about
>>> primary storage. Obviously, your primary storage configuration had
>>> changed
>>> between the time you shut down CS manager and xenservers and the
>>> time you
>>> restarted them. That is the vital info the list didn’t know.
>>>
>>> To best of my knowledge, zone wide primary storage has never been
>>> supported for Xen hypervisors.
>>>
>>> I do have to say that quite often CloudStack error messages are very
>>> cryptic, do not provide enough *useful* information to help users
>>> identify
>>> and trouble shoot actual problems. Those stack trace output might be a
>>> gold mine to developers, but they are utterly useless for end users.
>>>
>>> Just my $0.02
>>>
>>> Yiping
>>>
>>> On 7/28/15, 11:19 PM, "tony_caotong@163.com" <to...@163.com>
>>> wrote:
>>>
>>>> Hi, Finally I resolved this problem by my self.
>>>>
>>>> * Primary Storage: A storage resource typically provided to a single
>>>> cluster for the actual running of instance disk images. (Zone-wide
>>>> primary storage is an option, though not typically used.)
>>>>
>>>> This line above is from
>>>> http://docs.cloudstack.apache.org/en/master/concepts.html
>>>>
>>>> Because I have a Zone-wide primary storage, ACS can not find the
>>>> correct
>>>> primary which belong to XenServer cluster after reboot.
>>>>
>>>> Then I change the Zone-wide primary to cluster-wide, it resolved.
>>>>
>>>> Right now, I have two primary storage, one is kvm cluster-wide,
>>>> another
>>>> is xenserver cluster-wide.
>>>>
>>>> Above is for people who have the same problem oneday.
>>>>
>>>> by the way, I am very curious why I never receive replys from this
>>>> a big
>>>> community ?? of course except the very beginning.
>>>>
>>>> Is my English skill really really poor, result in no body can
>>>> understood
>>>> what language I am speaking ?
>>>>
>>>> -----------
>>>> Cao Tong
>>>>
>>>> On 07/22/2015 09:03 PM, tony_caotong@163.com wrote:
>>>>> Hey! help please...
>>>>>
>>>>> some news.
>>>>> I think the cause is that the ACS host can't communicate with
>>>>> XenServer host.
>>>>> ACS continues outputing logs like this
>>>>>
>>>>> 2015-07-22 20:42:13,555 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>> (AgentManager-Handler-7:null) Seq 5-8174877748607582212: Forwarding
>>>>> Seq 5-8174877748607582212: { Cmd , MgmtId: 279278805451459, via: 5,
>>>>> Ver: v1, Flags: 100111,
>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>> 280345368052992
>>>>>
>>>>> I am not sure that if the ACS status is wrong or some services on
>>>>> xenserver are not opend.
>>>>>
>>>>> on xenserver , I found *xenheartbeat.sh is not running.*
>>>>> *(/bin/bash /opt/cloud/bin/xenheartbeat.sh
>>>>> 00d8e0d0-8561-4b3d-9044-cbc496ff22cc 120 60)*
>>>>>
>>>>> As some operations about xenserver was pending， xenserver can not be
>>>>> deleted from web UI.
>>>>>
>>>>> I got a temporary solution
>>>>>
>>>>> 1. delete jobs from DB cloud.vm_work_job.
>>>>> 2. delete xenserver from DB cloud.host.
>>>>> 3. add xenserver host back from web UI.
>>>>>
>>>>> then it works.
>>>>>
>>>>> Does anyone have a idea for this?
>>>>>
>>>>> Could anyone tell what things does ACS do on xenserver host when
>>>>> adding a xenserver ?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -----------
>>>>> Cao Tong
>>>>>
>>>>> On 07/22/2015 04:26 PM, tony_caotong@163.com wrote:
>>>>>> @prashant, following it the answer of you questions
>>>>>>
>>>>>> 1. Yes, primary storage is connected fine for my xenserver.
>>>>>>
>>>>>> 2. No, Xenserver's password is not changed.
>>>>>>
>>>>>> 3. yes, web UI is fine, and I can login.
>>>>>>
>>>>>> 4. before reboot, I unmanaged and disabled resources, and after
>>>>>> reboot I have enabled all of them.
>>>>>>
>>>>>> 5. hosts is states is UP.
>>>>>>
>>>>>> 6. No yum update in anywhere.
>>>>>>
>>>>>> 7. system VMs status is fine, i think.
>>>>>>
>>>>>> -----------
>>>>>> Cao Tong
>>>>>>
>>>>>> On 07/22/2015 04:13 PM, tony_caotong@163.com wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> After reinstall, I got the problem again
>>>>>>>
>>>>>>> So, I will describe once again.
>>>>>>>
>>>>>>> WHAT my environment looks like:
>>>>>>>
>>>>>>> I have a ACS server host and a xenserver host, After both reboot, I
>>>>>>> can not create a VM on xenserver through ACS.
>>>>>>> A KVM and A NFS are running together in ACS manager host.
>>>>>>>
>>>>>>> the status of new VM is always 'staring' on the WEB, but I can
>>>>>>> create new VM using xencenter.
>>>>>>>
>>>>>>> ------------- ERR LOGS ----------
>>>>>>> 2015-07-22 15:56:56,357 DEBUG [c.c.s.StorageManagerImpl]
>>>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to send storage pool command
>>>>>>> to Pool[4|NetworkFilesystem] via 4
>>>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>>>> 2829104990918803478 to Host 4 timed out after 3600
>>>>>>>
>>>>>>> 2015-07-22 15:56:56,358 INFO [c.c.s.StatsCollector]
>>>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to reach
>>>>>>> Pool[4|NetworkFilesystem]
>>>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>>>> [StoragePool:4] is unreachable: Unable to send command to the pool
>>>>>>>
>>>>>>>
>>>>>>> ------------- and there are lots of DEBUG infos ------- repeat
>>>>>>> again and again -----------
>>>>>>>
>>>>>>> 2015-07-22 15:36:12,887 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>> (AgentManager-Handler-14:null) Seq 4-8064821032713715922:
>>>>>>> Forwarding
>>>>>>> Seq 4-8064821032713715922: { Cmd , MgmtId: 227448510156211,
>>>>>>> via: 4,
>>>>>>> Ver: v1, Flags: 100111,
>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>> 116784073679673
>>>>>>> 2015-07-22 15:36:12,889 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>> (AgentManager-Handler-10:null) Seq 4-8064821032713715883:
>>>>>>> Forwarding
>>>>>>> Seq 4-8064821032713715883: { Cmd , MgmtId: 227448510156211,
>>>>>>> via: 4,
>>>>>>> Ver: v1, Flags: 100111,
>>>>>>>
>>>>>>> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap
>>>>>>>
>>>>>>> ache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/5
>>>>>>>
>>>>>>> /af949612-838f-3a6d-931b-312e612db740.vhd","origUrl":"http://download.c
>>>>>>>
>>>>>>> loud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-30
>>>>>>>
>>>>>>> 17-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksu
>>>>>>>
>>>>>>> m":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>>>>>>
>>>>>>>
>>>>>>> 5.6(64-bit) no GUI
>>>>>>>
>>>>>>> (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"
>>>>>>>
>>>>>>> nfs://10.0.0.100/storage/secondary","_role":"Image"}},"name":"centos56-
>>>>>>>
>>>>>>> x86_64-xen","hypervisorType":"XenServer"}},"destTO":{"org.apache.clouds
>>>>>>>
>>>>>>> tack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com
>>>>>>>
>>>>>>> /templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-
>>>>>>>
>>>>>>> 8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905c
>>>>>>>
>>>>>>> ec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>>>>>> 5.6(64-bit) no GUI
>>>>>>>
>>>>>>> (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.Primar
>>>>>>>
>>>>>>> yDataStoreTO":{"uuid":"2df26406-31bf-3a95-8a61-f5008defd9a0","id":4,"po
>>>>>>>
>>>>>>> olType":"NetworkFilesystem","host":"10.0.0.100","path":"/storage/xen/pr
>>>>>>>
>>>>>>> imary","port":2049,"url":"NetworkFilesystem://10.0.0.100/storage/xen/pr
>>>>>>>
>>>>>>> imary/?ROLE=Primary&STOREUUID=2df26406-31bf-3a95-8a61-f5008defd9a0"}},"
>>>>>>>
>>>>>>> name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"executeInSe
>>>>>>>
>>>>>>> quence":true,"options":{},"wait":10800}}]
>>>>>>> } to 116784073679673
>>>>>>>
>>>>>>>
>>>>>>> -----------------------------------------
>>>>>>>
>>>>>>> Anyone have Any ideas? thanks.
>>>>>>>
>>>>>>> -----------
>>>>>>> Cao Tong
>>>>>>>
>>>>>>> On 07/21/2015 06:14 PM, tony_caotong@163.com wrote:
>>>>>>>> Thanks all,
>>>>>>>>
>>>>>>>> I have already reinstall my hosts for preparing a new clear
>>>>>>>> environment to restart my research.
>>>>>>>>
>>>>>>>> -----------
>>>>>>>> Cao Tong
>>>>>>>>
>>>>>>>> On 07/20/2015 09:24 PM, Prashant s wrote:
>>>>>>>>> some questions :
>>>>>>>>>
>>>>>>>>> can you please tell ...
>>>>>>>>>
>>>>>>>>> 1. is your NFS storage or your primary Storage Repository in
>>>>>>>>> connected
>>>>>>>>> mode with no red cross mark on them in xencenter.
>>>>>>>>> 2. did you change any passwords on the xenservers ?
>>>>>>>>> 3. is the cloudstack web ui up , can you login to the cloudstack
>>>>>>>>> web page.
>>>>>>>>> 4. *are the zone , pod, or clusters in unmanaged or disabled
>>>>>>>>> state
>>>>>>>>> ? *
>>>>>>>>> *5. is all the hosts in connected state ? *
>>>>>>>>> *6. did you run yum update on host reboot on the cs manager
>>>>>>>>> vm ? *
>>>>>>>>> *7. system vms are stateless you can kill them and cs will
>>>>>>>>> recreate a new
>>>>>>>>> one .. so dont worry :-) *
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *thanks *
>>>>>>>>> *prashant *
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jul 20, 2015 at 3:47 AM, <to...@163.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi, I restartd All hosts (one mgr and xenserver) again.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Following is the error log.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Could not find exception:
>>>>>>>>>> com.cloud.exception.OperationTimedoutException in error code
>>>>>>>>>> list
>>>>>>>>>> for
>>>>>>>>>> exceptions
>>>>>>>>>> 2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed
>>>>>>>>>> out on null
>>>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972:
>>>>>>>>>> Cancelling.
>>>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to send storage pool
>>>>>>>>>> command to
>>>>>>>>>> Pool[1|NetworkFilesystem] via 1
>>>>>>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>>>>>>> 3176445112179752972 to Host 1 timed out after 3600
>>>>>>>>>> at
>>>>>>>>>> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:
>>>>>>>>>>
>>>>>>>>>> 433)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:
>>>>>>>>>>
>>>>>>>>>> 362)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:1000)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:392)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:406)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC
>>>>>>>>>>
>>>>>>>>>> ollector.java:642)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
>>>>>>>>>>
>>>>>>>>>> anagedContextRunnable.java:49)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
>>>>>>>>>>
>>>>>>>>>> all(DefaultManagedContext.java:56)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
>>>>>>>>>>
>>>>>>>>>> lWithContext(DefaultManagedContext.java:103)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
>>>>>>>>>>
>>>>>>>>>> WithContext(DefaultManagedContext.java:53)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
>>>>>>>>>>
>>>>>>>>>> agedContextRunnable.java:46)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>>>>>>
>>>>>>>>>> 71)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>>
>>>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>>
>>>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>>>>>>>>>
>>>>>>>>>> .java:1145)
>>>>>>>>>>
>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>> 2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to reach
>>>>>>>>>> Pool[1|NetworkFilesystem]
>>>>>>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>>>>>>> [StoragePool:1]
>>>>>>>>>> is unreachable: Unable to send command to the pool
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:1010)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:392)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:406)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC
>>>>>>>>>>
>>>>>>>>>> ollector.java:642)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
>>>>>>>>>>
>>>>>>>>>> anagedContextRunnable.java:49)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
>>>>>>>>>>
>>>>>>>>>> all(DefaultManagedContext.java:56)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
>>>>>>>>>>
>>>>>>>>>> lWithContext(DefaultManagedContext.java:103)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
>>>>>>>>>>
>>>>>>>>>> WithContext(DefaultManagedContext.java:53)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
>>>>>>>>>>
>>>>>>>>>> agedContextRunnable.java:46)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>>>>>>
>>>>>>>>>> 71)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>>
>>>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>>
>>>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>>>>>>>>>
>>>>>>>>>> .java:1145)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto
>>>>>>>>>>
>>>>>>>>>> r.java:615)
>>>>>>>>>>
>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>>
>>>>>>>>>> -----------
>>>>>>>>>> Cao Tong
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 07/20/2015 02:52 PM, tony_caotong@163.com wrote:
>>>>>>>>>>
>>>>>>>>>>> No, no one's IP was changed.
>>>>>>>>>>>
>>>>>>>>>>> 1. In xenserver I can not login systemvms using the internal IP
>>>>>>>>>>> like
>>>>>>>>>>> '169.254.1.112', There shoud be a bridge network for this
>>>>>>>>>>> right? it is
>>>>>>>>>>> gone.
>>>>>>>>>>>
>>>>>>>>>>> 2. I try to delete xenserver host from CS on web, it also
>>>>>>>>>>> failed
>>>>>>>>>>> with
>>>>>>>>>>> lots of logs like following, then memory is full and mangement
>>>>>>>>>>> down...
>>>>>>>>>>>
>>>>>>>>>>> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-15:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411,
>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-5:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411,
>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-1:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411,
>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-14:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411,
>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I guess that, is there some service or daemons working for CS
>>>>>>>>>>> is not up
>>>>>>>>>>> on Xenserver ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -----------
>>>>>>>>>>> Cao Tong
>>>>>>>>>>> On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Did the management server ip change?
>>>>>>>>>>>> management server ip in the configuration table is used my
>>>>>>>>>>>> systemvms.
>>>>>>>>>>>> select * from configuration where name like 'host';
>>>>>>>>>>>>
>>>>>>>>>>>> If it changed, correct the value in db and restart systemvms.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ~Rajani
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jul 20, 2015 at 11:56 AM,<to...@163.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>> I shutdown my cs-manager and xenserver last weekend, And now
>>>>>>>>>>>>> the ssvm
>>>>>>>>>>>>> and cpvm is disconnect, thost two was runing on xenserver. so
>>>>>>>>>>>>> What
>>>>>>>>>>>>> should i do right now ?
>>>>>>>>>>>>> Please anybody help me and thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In xenserver I found that the three system VMs are not
>>>>>>>>>>>>> running.
>>>>>>>>>>>>> my xenserver seems can not reconnect to CS-manager. and it
>>>>>>>>>>>>> seams not
>>>>>>>>>>>>> under control of CS.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> What is the right steps of shutdown all CS group machines and
>>>>>>>>>>>>> resume
>>>>>>>>>>>>> them?
>>>>>>>>>>>>> How can i let my xenserver reconnected ?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> -----------
>>>>>>>>>>>>> Cao Tong
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>
>>
>
>
Re: XenServer is disconnected after CS hosts shutdown
Posted by to...@163.com.
After I disabled and enabled XS cluster & primary. it works.
seems XS agent have started by ACS host. so what is the correct
behavior of starting XS agent manually ?
-----------
Cao Tong
On 08/12/2015 04:41 PM, tony_caotong@163.com wrote:
>
> Hello,
>
> Almost one month gone, my problem is still here. I really really need
> someone to help me.
>
> new Settings ACS 4.4.4 XS 6.2
> After reboot, errors here:
>
> 2015-08-12 16:18:54,557 INFO [c.c.a.t.Request]
> (StatsCollector-3:ctx-b287085a) not building log message for '[{}]',
> _cmds.length == 1
> 2015-08-12 16:18:54,557 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Forwarding
> null to 191386435611186
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-12:null) Seq 9-4450963806725603425: Routing from
> 249082151178140
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-12:null) Seq 9-4450963806725603425: Link is closed
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-12:null) Seq 9-4450963806725603425: MgmtId
> 249082151178140: Req: Resource [Host:9] is unreachable: Host 9: Link
> is closed
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-12:null) Seq 9--1: MgmtId 249082151178140: Req:
> Routing to peer
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-15:null) Seq 9--1: MgmtId 249082151178140: Req:
> Cancel request received
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
> (AgentManager-Handler-15:null) Seq 9-4450963806725603425: Cancelling.
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Waiting
> some more time because this is the current command
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Waiting
> some more time because this is the current command
> 2015-08-12 16:18:54,559 INFO [c.c.u.e.CSExceptionErrorCode]
> (StatsCollector-3:ctx-b287085a) Could not find exception:
> com.cloud.exception.OperationTimedoutException in error code list for
> exceptions
> 2015-08-12 16:18:54,559 INFO [c.c.a.t.Request]
> (StatsCollector-3:ctx-b287085a) not building log message for '[{}]',
> _cmds.length == 1
> 2015-08-12 16:18:54,559 WARN [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Timed out
> on null
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Cancelling.
> 2015-08-12 16:18:54,559 DEBUG [c.c.s.StorageManagerImpl]
> (StatsCollector-3:ctx-b287085a) Unable to send storage pool command to
> Pool[3|NetworkFilesystem] via 9
> com.cloud.exception.OperationTimedoutException: Commands
> 4450963806725603425 to Host 9 timed out after 3600
> at
> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:434)
> at
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:418)
> at
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:965)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:390)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:404)
> at
> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
>
>
>
> -----------
> Cao Tong
>
> On 07/30/2015 10:55 AM, tony_caotong@163.com wrote:
>>
>> Hi Yiping, Thanks for your reply first.
>>
>>
>> My NFS server deployed in ACS manager host, another host is a single
>> xenserver.
>> A KVM environment deployed in ACS manager host too.
>> there is three storages named, Kprimary, Xprimary, Secondary.
>>
>> First, I add KVM cluster with zone-wide Kprimary, it works fine even
>> if after reboot.
>> Second, I add Xenserver with Xprimary(cluster-wide), it still works
>> fine.
>> Then, I set zone to disable, shutdown all system VMs, disable
>> those two hosts. finally shutdown xenserver host.
>> When xenserver's shutdown finish, I stop those services by order (
>> cloudstack-agent, cloudstack-management, libvirtd, nfs, rpcbind,
>> mysqld).
>> At last shutdown ACS host.
>>
>> The start process is totally reverse of this order.
>>
>> when done of starting, ACS says unable to send cmd to pool via host
>> xenserver.
>>
>> I am sure nothing have change in my environment during reboot time.
>> When I fix this problem, the only change is change Kprimary from
>> zone-wide to cluster-wide.
>>
>> I guess that after reboot some status have been initial from
>> beginning, ACS found that Xenserver host have two primary can be used.
>> so it use the high priority one， and it is Kprimary.
>>
>> Whatever, Maybe it could help you peoples to get ACS better.
>>
>> BTW, Error logs attached some days before is already point out this
>> is a storage problem
>> like:
>>
>> Unable to send storage pool command
>> to Pool[4|NetworkFilesystem] via 4
>>
>>
>>
>> -----------
>> Cao Tong
>>
>> On 07/30/2015 12:44 AM, Yiping Zhang wrote:
>>> Well, sometimes people can’t answer a question because of lack of
>>> relevant information, or simply because no one has encountered a
>>> similar
>>> situation before.
>>>
>>> Looking at your past messages on this thread, there were no mentions
>>> about
>>> primary storage. Obviously, your primary storage configuration had
>>> changed
>>> between the time you shut down CS manager and xenservers and the
>>> time you
>>> restarted them. That is the vital info the list didn’t know.
>>>
>>> To best of my knowledge, zone wide primary storage has never been
>>> supported for Xen hypervisors.
>>>
>>> I do have to say that quite often CloudStack error messages are very
>>> cryptic, do not provide enough *useful* information to help users
>>> identify
>>> and trouble shoot actual problems. Those stack trace output might be a
>>> gold mine to developers, but they are utterly useless for end users.
>>>
>>> Just my $0.02
>>>
>>> Yiping
>>>
>>> On 7/28/15, 11:19 PM, "tony_caotong@163.com" <to...@163.com>
>>> wrote:
>>>
>>>> Hi, Finally I resolved this problem by my self.
>>>>
>>>> * Primary Storage: A storage resource typically provided to a single
>>>> cluster for the actual running of instance disk images. (Zone-wide
>>>> primary storage is an option, though not typically used.)
>>>>
>>>> This line above is from
>>>> http://docs.cloudstack.apache.org/en/master/concepts.html
>>>>
>>>> Because I have a Zone-wide primary storage, ACS can not find the
>>>> correct
>>>> primary which belong to XenServer cluster after reboot.
>>>>
>>>> Then I change the Zone-wide primary to cluster-wide, it resolved.
>>>>
>>>> Right now, I have two primary storage, one is kvm cluster-wide,
>>>> another
>>>> is xenserver cluster-wide.
>>>>
>>>> Above is for people who have the same problem oneday.
>>>>
>>>> by the way, I am very curious why I never receive replys from this
>>>> a big
>>>> community ?? of course except the very beginning.
>>>>
>>>> Is my English skill really really poor, result in no body can
>>>> understood
>>>> what language I am speaking ?
>>>>
>>>> -----------
>>>> Cao Tong
>>>>
>>>> On 07/22/2015 09:03 PM, tony_caotong@163.com wrote:
>>>>> Hey! help please...
>>>>>
>>>>> some news.
>>>>> I think the cause is that the ACS host can't communicate with
>>>>> XenServer host.
>>>>> ACS continues outputing logs like this
>>>>>
>>>>> 2015-07-22 20:42:13,555 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>> (AgentManager-Handler-7:null) Seq 5-8174877748607582212: Forwarding
>>>>> Seq 5-8174877748607582212: { Cmd , MgmtId: 279278805451459, via: 5,
>>>>> Ver: v1, Flags: 100111,
>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>> 280345368052992
>>>>>
>>>>> I am not sure that if the ACS status is wrong or some services on
>>>>> xenserver are not opend.
>>>>>
>>>>> on xenserver , I found *xenheartbeat.sh is not running.*
>>>>> *(/bin/bash /opt/cloud/bin/xenheartbeat.sh
>>>>> 00d8e0d0-8561-4b3d-9044-cbc496ff22cc 120 60)*
>>>>>
>>>>> As some operations about xenserver was pending， xenserver can not be
>>>>> deleted from web UI.
>>>>>
>>>>> I got a temporary solution
>>>>>
>>>>> 1. delete jobs from DB cloud.vm_work_job.
>>>>> 2. delete xenserver from DB cloud.host.
>>>>> 3. add xenserver host back from web UI.
>>>>>
>>>>> then it works.
>>>>>
>>>>> Does anyone have a idea for this?
>>>>>
>>>>> Could anyone tell what things does ACS do on xenserver host when
>>>>> adding a xenserver ?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -----------
>>>>> Cao Tong
>>>>>
>>>>> On 07/22/2015 04:26 PM, tony_caotong@163.com wrote:
>>>>>> @prashant, following it the answer of you questions
>>>>>>
>>>>>> 1. Yes, primary storage is connected fine for my xenserver.
>>>>>>
>>>>>> 2. No, Xenserver's password is not changed.
>>>>>>
>>>>>> 3. yes, web UI is fine, and I can login.
>>>>>>
>>>>>> 4. before reboot, I unmanaged and disabled resources, and after
>>>>>> reboot I have enabled all of them.
>>>>>>
>>>>>> 5. hosts is states is UP.
>>>>>>
>>>>>> 6. No yum update in anywhere.
>>>>>>
>>>>>> 7. system VMs status is fine, i think.
>>>>>>
>>>>>> -----------
>>>>>> Cao Tong
>>>>>>
>>>>>> On 07/22/2015 04:13 PM, tony_caotong@163.com wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> After reinstall, I got the problem again
>>>>>>>
>>>>>>> So, I will describe once again.
>>>>>>>
>>>>>>> WHAT my environment looks like:
>>>>>>>
>>>>>>> I have a ACS server host and a xenserver host, After both reboot, I
>>>>>>> can not create a VM on xenserver through ACS.
>>>>>>> A KVM and A NFS are running together in ACS manager host.
>>>>>>>
>>>>>>> the status of new VM is always 'staring' on the WEB, but I can
>>>>>>> create new VM using xencenter.
>>>>>>>
>>>>>>> ------------- ERR LOGS ----------
>>>>>>> 2015-07-22 15:56:56,357 DEBUG [c.c.s.StorageManagerImpl]
>>>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to send storage pool command
>>>>>>> to Pool[4|NetworkFilesystem] via 4
>>>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>>>> 2829104990918803478 to Host 4 timed out after 3600
>>>>>>>
>>>>>>> 2015-07-22 15:56:56,358 INFO [c.c.s.StatsCollector]
>>>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to reach
>>>>>>> Pool[4|NetworkFilesystem]
>>>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>>>> [StoragePool:4] is unreachable: Unable to send command to the pool
>>>>>>>
>>>>>>>
>>>>>>> ------------- and there are lots of DEBUG infos ------- repeat
>>>>>>> again and again -----------
>>>>>>>
>>>>>>> 2015-07-22 15:36:12,887 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>> (AgentManager-Handler-14:null) Seq 4-8064821032713715922:
>>>>>>> Forwarding
>>>>>>> Seq 4-8064821032713715922: { Cmd , MgmtId: 227448510156211,
>>>>>>> via: 4,
>>>>>>> Ver: v1, Flags: 100111,
>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>> 116784073679673
>>>>>>> 2015-07-22 15:36:12,889 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>> (AgentManager-Handler-10:null) Seq 4-8064821032713715883:
>>>>>>> Forwarding
>>>>>>> Seq 4-8064821032713715883: { Cmd , MgmtId: 227448510156211,
>>>>>>> via: 4,
>>>>>>> Ver: v1, Flags: 100111,
>>>>>>>
>>>>>>> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap
>>>>>>>
>>>>>>> ache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/5
>>>>>>>
>>>>>>> /af949612-838f-3a6d-931b-312e612db740.vhd","origUrl":"http://download.c
>>>>>>>
>>>>>>> loud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-30
>>>>>>>
>>>>>>> 17-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksu
>>>>>>>
>>>>>>> m":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>>>>>>
>>>>>>>
>>>>>>> 5.6(64-bit) no GUI
>>>>>>>
>>>>>>> (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"
>>>>>>>
>>>>>>> nfs://10.0.0.100/storage/secondary","_role":"Image"}},"name":"centos56-
>>>>>>>
>>>>>>> x86_64-xen","hypervisorType":"XenServer"}},"destTO":{"org.apache.clouds
>>>>>>>
>>>>>>> tack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com
>>>>>>>
>>>>>>> /templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-
>>>>>>>
>>>>>>> 8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905c
>>>>>>>
>>>>>>> ec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>>>>>> 5.6(64-bit) no GUI
>>>>>>>
>>>>>>> (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.Primar
>>>>>>>
>>>>>>> yDataStoreTO":{"uuid":"2df26406-31bf-3a95-8a61-f5008defd9a0","id":4,"po
>>>>>>>
>>>>>>> olType":"NetworkFilesystem","host":"10.0.0.100","path":"/storage/xen/pr
>>>>>>>
>>>>>>> imary","port":2049,"url":"NetworkFilesystem://10.0.0.100/storage/xen/pr
>>>>>>>
>>>>>>> imary/?ROLE=Primary&STOREUUID=2df26406-31bf-3a95-8a61-f5008defd9a0"}},"
>>>>>>>
>>>>>>> name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"executeInSe
>>>>>>>
>>>>>>> quence":true,"options":{},"wait":10800}}]
>>>>>>> } to 116784073679673
>>>>>>>
>>>>>>>
>>>>>>> -----------------------------------------
>>>>>>>
>>>>>>> Anyone have Any ideas? thanks.
>>>>>>>
>>>>>>> -----------
>>>>>>> Cao Tong
>>>>>>>
>>>>>>> On 07/21/2015 06:14 PM, tony_caotong@163.com wrote:
>>>>>>>> Thanks all,
>>>>>>>>
>>>>>>>> I have already reinstall my hosts for preparing a new clear
>>>>>>>> environment to restart my research.
>>>>>>>>
>>>>>>>> -----------
>>>>>>>> Cao Tong
>>>>>>>>
>>>>>>>> On 07/20/2015 09:24 PM, Prashant s wrote:
>>>>>>>>> some questions :
>>>>>>>>>
>>>>>>>>> can you please tell ...
>>>>>>>>>
>>>>>>>>> 1. is your NFS storage or your primary Storage Repository in
>>>>>>>>> connected
>>>>>>>>> mode with no red cross mark on them in xencenter.
>>>>>>>>> 2. did you change any passwords on the xenservers ?
>>>>>>>>> 3. is the cloudstack web ui up , can you login to the cloudstack
>>>>>>>>> web page.
>>>>>>>>> 4. *are the zone , pod, or clusters in unmanaged or disabled
>>>>>>>>> state
>>>>>>>>> ? *
>>>>>>>>> *5. is all the hosts in connected state ? *
>>>>>>>>> *6. did you run yum update on host reboot on the cs manager
>>>>>>>>> vm ? *
>>>>>>>>> *7. system vms are stateless you can kill them and cs will
>>>>>>>>> recreate a new
>>>>>>>>> one .. so dont worry :-) *
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *thanks *
>>>>>>>>> *prashant *
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jul 20, 2015 at 3:47 AM, <to...@163.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi, I restartd All hosts (one mgr and xenserver) again.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Following is the error log.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Could not find exception:
>>>>>>>>>> com.cloud.exception.OperationTimedoutException in error code
>>>>>>>>>> list
>>>>>>>>>> for
>>>>>>>>>> exceptions
>>>>>>>>>> 2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed
>>>>>>>>>> out on null
>>>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972:
>>>>>>>>>> Cancelling.
>>>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to send storage pool
>>>>>>>>>> command to
>>>>>>>>>> Pool[1|NetworkFilesystem] via 1
>>>>>>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>>>>>>> 3176445112179752972 to Host 1 timed out after 3600
>>>>>>>>>> at
>>>>>>>>>> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:
>>>>>>>>>>
>>>>>>>>>> 433)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:
>>>>>>>>>>
>>>>>>>>>> 362)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:1000)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:392)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:406)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC
>>>>>>>>>>
>>>>>>>>>> ollector.java:642)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
>>>>>>>>>>
>>>>>>>>>> anagedContextRunnable.java:49)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
>>>>>>>>>>
>>>>>>>>>> all(DefaultManagedContext.java:56)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
>>>>>>>>>>
>>>>>>>>>> lWithContext(DefaultManagedContext.java:103)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
>>>>>>>>>>
>>>>>>>>>> WithContext(DefaultManagedContext.java:53)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
>>>>>>>>>>
>>>>>>>>>> agedContextRunnable.java:46)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>>>>>>
>>>>>>>>>> 71)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>>
>>>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>>
>>>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>>>>>>>>>
>>>>>>>>>> .java:1145)
>>>>>>>>>>
>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>> 2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to reach
>>>>>>>>>> Pool[1|NetworkFilesystem]
>>>>>>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>>>>>>> [StoragePool:1]
>>>>>>>>>> is unreachable: Unable to send command to the pool
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:1010)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:392)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:406)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC
>>>>>>>>>>
>>>>>>>>>> ollector.java:642)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
>>>>>>>>>>
>>>>>>>>>> anagedContextRunnable.java:49)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
>>>>>>>>>>
>>>>>>>>>> all(DefaultManagedContext.java:56)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
>>>>>>>>>>
>>>>>>>>>> lWithContext(DefaultManagedContext.java:103)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
>>>>>>>>>>
>>>>>>>>>> WithContext(DefaultManagedContext.java:53)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
>>>>>>>>>>
>>>>>>>>>> agedContextRunnable.java:46)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>>>>>>
>>>>>>>>>> 71)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>>
>>>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>>
>>>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>>>>>>>>>
>>>>>>>>>> .java:1145)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto
>>>>>>>>>>
>>>>>>>>>> r.java:615)
>>>>>>>>>>
>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>>
>>>>>>>>>> -----------
>>>>>>>>>> Cao Tong
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 07/20/2015 02:52 PM, tony_caotong@163.com wrote:
>>>>>>>>>>
>>>>>>>>>>> No, no one's IP was changed.
>>>>>>>>>>>
>>>>>>>>>>> 1. In xenserver I can not login systemvms using the internal IP
>>>>>>>>>>> like
>>>>>>>>>>> '169.254.1.112', There shoud be a bridge network for this
>>>>>>>>>>> right? it is
>>>>>>>>>>> gone.
>>>>>>>>>>>
>>>>>>>>>>> 2. I try to delete xenserver host from CS on web, it also
>>>>>>>>>>> failed
>>>>>>>>>>> with
>>>>>>>>>>> lots of logs like following, then memory is full and mangement
>>>>>>>>>>> down...
>>>>>>>>>>>
>>>>>>>>>>> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-15:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411,
>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-5:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411,
>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-1:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411,
>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-14:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411,
>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I guess that, is there some service or daemons working for CS
>>>>>>>>>>> is not up
>>>>>>>>>>> on Xenserver ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -----------
>>>>>>>>>>> Cao Tong
>>>>>>>>>>> On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Did the management server ip change?
>>>>>>>>>>>> management server ip in the configuration table is used my
>>>>>>>>>>>> systemvms.
>>>>>>>>>>>> select * from configuration where name like 'host';
>>>>>>>>>>>>
>>>>>>>>>>>> If it changed, correct the value in db and restart systemvms.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ~Rajani
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jul 20, 2015 at 11:56 AM,<to...@163.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>> I shutdown my cs-manager and xenserver last weekend, And now
>>>>>>>>>>>>> the ssvm
>>>>>>>>>>>>> and cpvm is disconnect, thost two was runing on xenserver. so
>>>>>>>>>>>>> What
>>>>>>>>>>>>> should i do right now ?
>>>>>>>>>>>>> Please anybody help me and thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In xenserver I found that the three system VMs are not
>>>>>>>>>>>>> running.
>>>>>>>>>>>>> my xenserver seems can not reconnect to CS-manager. and it
>>>>>>>>>>>>> seams not
>>>>>>>>>>>>> under control of CS.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> What is the right steps of shutdown all CS group machines and
>>>>>>>>>>>>> resume
>>>>>>>>>>>>> them?
>>>>>>>>>>>>> How can i let my xenserver reconnected ?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> -----------
>>>>>>>>>>>>> Cao Tong
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>
>>
>
>
Re: XenServer is disconnected after CS hosts shutdown
Posted by to...@163.com.
Hello,
Almost one month gone, my problem is still here. I really really need
someone to help me.
new Settings ACS 4.4.4 XS 6.2
After reboot, errors here:
2015-08-12 16:18:54,557 INFO [c.c.a.t.Request]
(StatsCollector-3:ctx-b287085a) not building log message for '[{}]',
_cmds.length == 1
2015-08-12 16:18:54,557 DEBUG [c.c.a.m.ClusteredAgentAttache]
(StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Forwarding
null to 191386435611186
2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-12:null) Seq 9-4450963806725603425: Routing from
249082151178140
2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-12:null) Seq 9-4450963806725603425: Link is closed
2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-12:null) Seq 9-4450963806725603425: MgmtId
249082151178140: Req: Resource [Host:9] is unreachable: Host 9: Link is
closed
2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-12:null) Seq 9--1: MgmtId 249082151178140: Req:
Routing to peer
2015-08-12 16:18:54,559 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-15:null) Seq 9--1: MgmtId 249082151178140: Req:
Cancel request received
2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
(AgentManager-Handler-15:null) Seq 9-4450963806725603425: Cancelling.
2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
(StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Waiting some
more time because this is the current command
2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
(StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Waiting some
more time because this is the current command
2015-08-12 16:18:54,559 INFO [c.c.u.e.CSExceptionErrorCode]
(StatsCollector-3:ctx-b287085a) Could not find exception:
com.cloud.exception.OperationTimedoutException in error code list for
exceptions
2015-08-12 16:18:54,559 INFO [c.c.a.t.Request]
(StatsCollector-3:ctx-b287085a) not building log message for '[{}]',
_cmds.length == 1
2015-08-12 16:18:54,559 WARN [c.c.a.m.AgentAttache]
(StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Timed out on null
2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
(StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Cancelling.
2015-08-12 16:18:54,559 DEBUG [c.c.s.StorageManagerImpl]
(StatsCollector-3:ctx-b287085a) Unable to send storage pool command to
Pool[3|NetworkFilesystem] via 9
com.cloud.exception.OperationTimedoutException: Commands
4450963806725603425 to Host 9 timed out after 3600
at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:434)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:418)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:965)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:390)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:404)
at
com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
-----------
Cao Tong
On 07/30/2015 10:55 AM, tony_caotong@163.com wrote:
>
> Hi Yiping, Thanks for your reply first.
>
>
> My NFS server deployed in ACS manager host, another host is a single
> xenserver.
> A KVM environment deployed in ACS manager host too.
> there is three storages named, Kprimary, Xprimary, Secondary.
>
> First, I add KVM cluster with zone-wide Kprimary, it works fine even
> if after reboot.
> Second, I add Xenserver with Xprimary(cluster-wide), it still works
> fine.
> Then, I set zone to disable, shutdown all system VMs, disable those
> two hosts. finally shutdown xenserver host.
> When xenserver's shutdown finish, I stop those services by order (
> cloudstack-agent, cloudstack-management, libvirtd, nfs, rpcbind,
> mysqld).
> At last shutdown ACS host.
>
> The start process is totally reverse of this order.
>
> when done of starting, ACS says unable to send cmd to pool via host
> xenserver.
>
> I am sure nothing have change in my environment during reboot time.
> When I fix this problem, the only change is change Kprimary from
> zone-wide to cluster-wide.
>
> I guess that after reboot some status have been initial from
> beginning, ACS found that Xenserver host have two primary can be used.
> so it use the high priority one， and it is Kprimary.
>
> Whatever, Maybe it could help you peoples to get ACS better.
>
> BTW, Error logs attached some days before is already point out this is
> a storage problem
> like:
>
> Unable to send storage pool command
> to Pool[4|NetworkFilesystem] via 4
>
>
>
> -----------
> Cao Tong
>
> On 07/30/2015 12:44 AM, Yiping Zhang wrote:
>> Well, sometimes people can’t answer a question because of lack of
>> relevant information, or simply because no one has encountered a similar
>> situation before.
>>
>> Looking at your past messages on this thread, there were no mentions
>> about
>> primary storage. Obviously, your primary storage configuration had
>> changed
>> between the time you shut down CS manager and xenservers and the time
>> you
>> restarted them. That is the vital info the list didn’t know.
>>
>> To best of my knowledge, zone wide primary storage has never been
>> supported for Xen hypervisors.
>>
>> I do have to say that quite often CloudStack error messages are very
>> cryptic, do not provide enough *useful* information to help users
>> identify
>> and trouble shoot actual problems. Those stack trace output might be a
>> gold mine to developers, but they are utterly useless for end users.
>>
>> Just my $0.02
>>
>> Yiping
>>
>> On 7/28/15, 11:19 PM, "tony_caotong@163.com" <to...@163.com>
>> wrote:
>>
>>> Hi, Finally I resolved this problem by my self.
>>>
>>> * Primary Storage: A storage resource typically provided to a single
>>> cluster for the actual running of instance disk images. (Zone-wide
>>> primary storage is an option, though not typically used.)
>>>
>>> This line above is from
>>> http://docs.cloudstack.apache.org/en/master/concepts.html
>>>
>>> Because I have a Zone-wide primary storage, ACS can not find the
>>> correct
>>> primary which belong to XenServer cluster after reboot.
>>>
>>> Then I change the Zone-wide primary to cluster-wide, it resolved.
>>>
>>> Right now, I have two primary storage, one is kvm cluster-wide, another
>>> is xenserver cluster-wide.
>>>
>>> Above is for people who have the same problem oneday.
>>>
>>> by the way, I am very curious why I never receive replys from this a
>>> big
>>> community ?? of course except the very beginning.
>>>
>>> Is my English skill really really poor, result in no body can
>>> understood
>>> what language I am speaking ?
>>>
>>> -----------
>>> Cao Tong
>>>
>>> On 07/22/2015 09:03 PM, tony_caotong@163.com wrote:
>>>> Hey! help please...
>>>>
>>>> some news.
>>>> I think the cause is that the ACS host can't communicate with
>>>> XenServer host.
>>>> ACS continues outputing logs like this
>>>>
>>>> 2015-07-22 20:42:13,555 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>> (AgentManager-Handler-7:null) Seq 5-8174877748607582212: Forwarding
>>>> Seq 5-8174877748607582212: { Cmd , MgmtId: 279278805451459, via: 5,
>>>> Ver: v1, Flags: 100111,
>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>> 280345368052992
>>>>
>>>> I am not sure that if the ACS status is wrong or some services on
>>>> xenserver are not opend.
>>>>
>>>> on xenserver , I found *xenheartbeat.sh is not running.*
>>>> *(/bin/bash /opt/cloud/bin/xenheartbeat.sh
>>>> 00d8e0d0-8561-4b3d-9044-cbc496ff22cc 120 60)*
>>>>
>>>> As some operations about xenserver was pending， xenserver can not be
>>>> deleted from web UI.
>>>>
>>>> I got a temporary solution
>>>>
>>>> 1. delete jobs from DB cloud.vm_work_job.
>>>> 2. delete xenserver from DB cloud.host.
>>>> 3. add xenserver host back from web UI.
>>>>
>>>> then it works.
>>>>
>>>> Does anyone have a idea for this?
>>>>
>>>> Could anyone tell what things does ACS do on xenserver host when
>>>> adding a xenserver ?
>>>>
>>>> Thanks,
>>>>
>>>> -----------
>>>> Cao Tong
>>>>
>>>> On 07/22/2015 04:26 PM, tony_caotong@163.com wrote:
>>>>> @prashant, following it the answer of you questions
>>>>>
>>>>> 1. Yes, primary storage is connected fine for my xenserver.
>>>>>
>>>>> 2. No, Xenserver's password is not changed.
>>>>>
>>>>> 3. yes, web UI is fine, and I can login.
>>>>>
>>>>> 4. before reboot, I unmanaged and disabled resources, and after
>>>>> reboot I have enabled all of them.
>>>>>
>>>>> 5. hosts is states is UP.
>>>>>
>>>>> 6. No yum update in anywhere.
>>>>>
>>>>> 7. system VMs status is fine, i think.
>>>>>
>>>>> -----------
>>>>> Cao Tong
>>>>>
>>>>> On 07/22/2015 04:13 PM, tony_caotong@163.com wrote:
>>>>>> Hi,
>>>>>>
>>>>>> After reinstall, I got the problem again
>>>>>>
>>>>>> So, I will describe once again.
>>>>>>
>>>>>> WHAT my environment looks like:
>>>>>>
>>>>>> I have a ACS server host and a xenserver host, After both reboot, I
>>>>>> can not create a VM on xenserver through ACS.
>>>>>> A KVM and A NFS are running together in ACS manager host.
>>>>>>
>>>>>> the status of new VM is always 'staring' on the WEB, but I can
>>>>>> create new VM using xencenter.
>>>>>>
>>>>>> ------------- ERR LOGS ----------
>>>>>> 2015-07-22 15:56:56,357 DEBUG [c.c.s.StorageManagerImpl]
>>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to send storage pool command
>>>>>> to Pool[4|NetworkFilesystem] via 4
>>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>>> 2829104990918803478 to Host 4 timed out after 3600
>>>>>>
>>>>>> 2015-07-22 15:56:56,358 INFO [c.c.s.StatsCollector]
>>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to reach
>>>>>> Pool[4|NetworkFilesystem]
>>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>>> [StoragePool:4] is unreachable: Unable to send command to the pool
>>>>>>
>>>>>>
>>>>>> ------------- and there are lots of DEBUG infos ------- repeat
>>>>>> again and again -----------
>>>>>>
>>>>>> 2015-07-22 15:36:12,887 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>> (AgentManager-Handler-14:null) Seq 4-8064821032713715922: Forwarding
>>>>>> Seq 4-8064821032713715922: { Cmd , MgmtId: 227448510156211, via: 4,
>>>>>> Ver: v1, Flags: 100111,
>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>> 116784073679673
>>>>>> 2015-07-22 15:36:12,889 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>> (AgentManager-Handler-10:null) Seq 4-8064821032713715883: Forwarding
>>>>>> Seq 4-8064821032713715883: { Cmd , MgmtId: 227448510156211, via: 4,
>>>>>> Ver: v1, Flags: 100111,
>>>>>>
>>>>>> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap
>>>>>>
>>>>>> ache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/5
>>>>>>
>>>>>> /af949612-838f-3a6d-931b-312e612db740.vhd","origUrl":"http://download.c
>>>>>>
>>>>>> loud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-30
>>>>>>
>>>>>> 17-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksu
>>>>>>
>>>>>> m":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>>>>>
>>>>>>
>>>>>> 5.6(64-bit) no GUI
>>>>>>
>>>>>> (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"
>>>>>>
>>>>>> nfs://10.0.0.100/storage/secondary","_role":"Image"}},"name":"centos56-
>>>>>>
>>>>>> x86_64-xen","hypervisorType":"XenServer"}},"destTO":{"org.apache.clouds
>>>>>>
>>>>>> tack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com
>>>>>>
>>>>>> /templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-
>>>>>>
>>>>>> 8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905c
>>>>>>
>>>>>> ec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>>>>> 5.6(64-bit) no GUI
>>>>>>
>>>>>> (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.Primar
>>>>>>
>>>>>> yDataStoreTO":{"uuid":"2df26406-31bf-3a95-8a61-f5008defd9a0","id":4,"po
>>>>>>
>>>>>> olType":"NetworkFilesystem","host":"10.0.0.100","path":"/storage/xen/pr
>>>>>>
>>>>>> imary","port":2049,"url":"NetworkFilesystem://10.0.0.100/storage/xen/pr
>>>>>>
>>>>>> imary/?ROLE=Primary&STOREUUID=2df26406-31bf-3a95-8a61-f5008defd9a0"}},"
>>>>>>
>>>>>> name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"executeInSe
>>>>>>
>>>>>> quence":true,"options":{},"wait":10800}}]
>>>>>> } to 116784073679673
>>>>>>
>>>>>>
>>>>>> -----------------------------------------
>>>>>>
>>>>>> Anyone have Any ideas? thanks.
>>>>>>
>>>>>> -----------
>>>>>> Cao Tong
>>>>>>
>>>>>> On 07/21/2015 06:14 PM, tony_caotong@163.com wrote:
>>>>>>> Thanks all,
>>>>>>>
>>>>>>> I have already reinstall my hosts for preparing a new clear
>>>>>>> environment to restart my research.
>>>>>>>
>>>>>>> -----------
>>>>>>> Cao Tong
>>>>>>>
>>>>>>> On 07/20/2015 09:24 PM, Prashant s wrote:
>>>>>>>> some questions :
>>>>>>>>
>>>>>>>> can you please tell ...
>>>>>>>>
>>>>>>>> 1. is your NFS storage or your primary Storage Repository in
>>>>>>>> connected
>>>>>>>> mode with no red cross mark on them in xencenter.
>>>>>>>> 2. did you change any passwords on the xenservers ?
>>>>>>>> 3. is the cloudstack web ui up , can you login to the cloudstack
>>>>>>>> web page.
>>>>>>>> 4. *are the zone , pod, or clusters in unmanaged or disabled state
>>>>>>>> ? *
>>>>>>>> *5. is all the hosts in connected state ? *
>>>>>>>> *6. did you run yum update on host reboot on the cs manager vm
>>>>>>>> ? *
>>>>>>>> *7. system vms are stateless you can kill them and cs will
>>>>>>>> recreate a new
>>>>>>>> one .. so dont worry :-) *
>>>>>>>>
>>>>>>>>
>>>>>>>> *thanks *
>>>>>>>> *prashant *
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jul 20, 2015 at 3:47 AM, <to...@163.com> wrote:
>>>>>>>>
>>>>>>>>> Hi, I restartd All hosts (one mgr and xenserver) again.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Following is the error log.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
>>>>>>>>> (StatsCollector-3:ctx-692a5392) Could not find exception:
>>>>>>>>> com.cloud.exception.OperationTimedoutException in error code list
>>>>>>>>> for
>>>>>>>>> exceptions
>>>>>>>>> 2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
>>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed
>>>>>>>>> out on null
>>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
>>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972:
>>>>>>>>> Cancelling.
>>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
>>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to send storage pool
>>>>>>>>> command to
>>>>>>>>> Pool[1|NetworkFilesystem] via 1
>>>>>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>>>>>> 3176445112179752972 to Host 1 timed out after 3600
>>>>>>>>> at
>>>>>>>>> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:
>>>>>>>>>
>>>>>>>>> 433)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:
>>>>>>>>>
>>>>>>>>> 362)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>
>>>>>>>>> ava:1000)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>
>>>>>>>>> ava:392)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>
>>>>>>>>> ava:406)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC
>>>>>>>>>
>>>>>>>>> ollector.java:642)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
>>>>>>>>>
>>>>>>>>> anagedContextRunnable.java:49)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
>>>>>>>>>
>>>>>>>>> all(DefaultManagedContext.java:56)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
>>>>>>>>>
>>>>>>>>> lWithContext(DefaultManagedContext.java:103)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
>>>>>>>>>
>>>>>>>>> WithContext(DefaultManagedContext.java:53)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
>>>>>>>>>
>>>>>>>>> agedContextRunnable.java:46)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>>>>>
>>>>>>>>> 71)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>
>>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>
>>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>>>>>>>>
>>>>>>>>> .java:1145)
>>>>>>>>>
>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>> 2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
>>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to reach
>>>>>>>>> Pool[1|NetworkFilesystem]
>>>>>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>>>>>> [StoragePool:1]
>>>>>>>>> is unreachable: Unable to send command to the pool
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>
>>>>>>>>> ava:1010)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>
>>>>>>>>> ava:392)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>
>>>>>>>>> ava:406)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC
>>>>>>>>>
>>>>>>>>> ollector.java:642)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
>>>>>>>>>
>>>>>>>>> anagedContextRunnable.java:49)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
>>>>>>>>>
>>>>>>>>> all(DefaultManagedContext.java:56)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
>>>>>>>>>
>>>>>>>>> lWithContext(DefaultManagedContext.java:103)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
>>>>>>>>>
>>>>>>>>> WithContext(DefaultManagedContext.java:53)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
>>>>>>>>>
>>>>>>>>> agedContextRunnable.java:46)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>>>>>
>>>>>>>>> 71)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>
>>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>
>>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>>>>>>>>
>>>>>>>>> .java:1145)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto
>>>>>>>>>
>>>>>>>>> r.java:615)
>>>>>>>>>
>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>
>>>>>>>>> -----------
>>>>>>>>> Cao Tong
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 07/20/2015 02:52 PM, tony_caotong@163.com wrote:
>>>>>>>>>
>>>>>>>>>> No, no one's IP was changed.
>>>>>>>>>>
>>>>>>>>>> 1. In xenserver I can not login systemvms using the internal IP
>>>>>>>>>> like
>>>>>>>>>> '169.254.1.112', There shoud be a bridge network for this
>>>>>>>>>> right? it is
>>>>>>>>>> gone.
>>>>>>>>>>
>>>>>>>>>> 2. I try to delete xenserver host from CS on web, it also failed
>>>>>>>>>> with
>>>>>>>>>> lots of logs like following, then memory is full and mangement
>>>>>>>>>> down...
>>>>>>>>>>
>>>>>>>>>> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>> (AgentManager-Handler-15:null) Seq 1-7282039122481381399:
>>>>>>>>>> Forwarding Seq
>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>>>>> Ver: v1,
>>>>>>>>>> Flags: 100111,
>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>> 192405008094602
>>>>>>>>>> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>> (AgentManager-Handler-5:null) Seq 1-7282039122481381399:
>>>>>>>>>> Forwarding Seq
>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>>>>> Ver: v1,
>>>>>>>>>> Flags: 100111,
>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>> 192405008094602
>>>>>>>>>> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>> (AgentManager-Handler-1:null) Seq 1-7282039122481381399:
>>>>>>>>>> Forwarding Seq
>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>>>>> Ver: v1,
>>>>>>>>>> Flags: 100111,
>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>> 192405008094602
>>>>>>>>>> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>> (AgentManager-Handler-14:null) Seq 1-7282039122481381399:
>>>>>>>>>> Forwarding Seq
>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>>>>> Ver: v1,
>>>>>>>>>> Flags: 100111,
>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>> 192405008094602
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I guess that, is there some service or daemons working for CS
>>>>>>>>>> is not up
>>>>>>>>>> on Xenserver ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -----------
>>>>>>>>>> Cao Tong
>>>>>>>>>> On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
>>>>>>>>>>
>>>>>>>>>>> Did the management server ip change?
>>>>>>>>>>> management server ip in the configuration table is used my
>>>>>>>>>>> systemvms.
>>>>>>>>>>> select * from configuration where name like 'host';
>>>>>>>>>>>
>>>>>>>>>>> If it changed, correct the value in db and restart systemvms.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ~Rajani
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jul 20, 2015 at 11:56 AM,<to...@163.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>> I shutdown my cs-manager and xenserver last weekend, And now
>>>>>>>>>>>> the ssvm
>>>>>>>>>>>> and cpvm is disconnect, thost two was runing on xenserver. so
>>>>>>>>>>>> What
>>>>>>>>>>>> should i do right now ?
>>>>>>>>>>>> Please anybody help me and thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> In xenserver I found that the three system VMs are not
>>>>>>>>>>>> running.
>>>>>>>>>>>> my xenserver seems can not reconnect to CS-manager. and it
>>>>>>>>>>>> seams not
>>>>>>>>>>>> under control of CS.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> What is the right steps of shutdown all CS group machines and
>>>>>>>>>>>> resume
>>>>>>>>>>>> them?
>>>>>>>>>>>> How can i let my xenserver reconnected ?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> -----------
>>>>>>>>>>>> Cao Tong
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>
>
Re: XenServer is disconnected after CS hosts shutdown
Posted by to...@163.com.
Hi Yiping, Thanks for your reply first.
My NFS server deployed in ACS manager host, another host is a single
xenserver.
A KVM environment deployed in ACS manager host too.
there is three storages named, Kprimary, Xprimary, Secondary.
First, I add KVM cluster with zone-wide Kprimary, it works fine even if
after reboot.
Second, I add Xenserver with Xprimary(cluster-wide), it still works fine.
Then, I set zone to disable, shutdown all system VMs, disable those
two hosts. finally shutdown xenserver host.
When xenserver's shutdown finish, I stop those services by order (
cloudstack-agent, cloudstack-management, libvirtd, nfs, rpcbind, mysqld).
At last shutdown ACS host.
The start process is totally reverse of this order.
when done of starting, ACS says unable to send cmd to pool via host
xenserver.
I am sure nothing have change in my environment during reboot time.
When I fix this problem, the only change is change Kprimary from
zone-wide to cluster-wide.
I guess that after reboot some status have been initial from beginning,
ACS found that Xenserver host have two primary can be used.
so it use the high priority one， and it is Kprimary.
Whatever, Maybe it could help you peoples to get ACS better.
BTW, Error logs attached some days before is already point out this is a
storage problem
like:
Unable to send storage pool command
to Pool[4|NetworkFilesystem] via 4
-----------
Cao Tong
On 07/30/2015 12:44 AM, Yiping Zhang wrote:
> Well, sometimes people can’t answer a question because of lack of
> relevant information, or simply because no one has encountered a similar
> situation before.
>
> Looking at your past messages on this thread, there were no mentions about
> primary storage. Obviously, your primary storage configuration had changed
> between the time you shut down CS manager and xenservers and the time you
> restarted them. That is the vital info the list didn’t know.
>
> To best of my knowledge, zone wide primary storage has never been
> supported for Xen hypervisors.
>
> I do have to say that quite often CloudStack error messages are very
> cryptic, do not provide enough *useful* information to help users identify
> and trouble shoot actual problems. Those stack trace output might be a
> gold mine to developers, but they are utterly useless for end users.
>
> Just my $0.02
>
> Yiping
>
> On 7/28/15, 11:19 PM, "tony_caotong@163.com" <to...@163.com> wrote:
>
>> Hi, Finally I resolved this problem by my self.
>>
>> * Primary Storage: A storage resource typically provided to a single
>> cluster for the actual running of instance disk images. (Zone-wide
>> primary storage is an option, though not typically used.)
>>
>> This line above is from
>> http://docs.cloudstack.apache.org/en/master/concepts.html
>>
>> Because I have a Zone-wide primary storage, ACS can not find the correct
>> primary which belong to XenServer cluster after reboot.
>>
>> Then I change the Zone-wide primary to cluster-wide, it resolved.
>>
>> Right now, I have two primary storage, one is kvm cluster-wide, another
>> is xenserver cluster-wide.
>>
>> Above is for people who have the same problem oneday.
>>
>> by the way, I am very curious why I never receive replys from this a big
>> community ?? of course except the very beginning.
>>
>> Is my English skill really really poor, result in no body can understood
>> what language I am speaking ?
>>
>> -----------
>> Cao Tong
>>
>> On 07/22/2015 09:03 PM, tony_caotong@163.com wrote:
>>> Hey! help please...
>>>
>>> some news.
>>> I think the cause is that the ACS host can't communicate with
>>> XenServer host.
>>> ACS continues outputing logs like this
>>>
>>> 2015-07-22 20:42:13,555 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>> (AgentManager-Handler-7:null) Seq 5-8174877748607582212: Forwarding
>>> Seq 5-8174877748607582212: { Cmd , MgmtId: 279278805451459, via: 5,
>>> Ver: v1, Flags: 100111,
>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>> 280345368052992
>>>
>>> I am not sure that if the ACS status is wrong or some services on
>>> xenserver are not opend.
>>>
>>> on xenserver , I found *xenheartbeat.sh is not running.*
>>> *(/bin/bash /opt/cloud/bin/xenheartbeat.sh
>>> 00d8e0d0-8561-4b3d-9044-cbc496ff22cc 120 60)*
>>>
>>> As some operations about xenserver was pending， xenserver can not be
>>> deleted from web UI.
>>>
>>> I got a temporary solution
>>>
>>> 1. delete jobs from DB cloud.vm_work_job.
>>> 2. delete xenserver from DB cloud.host.
>>> 3. add xenserver host back from web UI.
>>>
>>> then it works.
>>>
>>> Does anyone have a idea for this?
>>>
>>> Could anyone tell what things does ACS do on xenserver host when
>>> adding a xenserver ?
>>>
>>> Thanks,
>>>
>>> -----------
>>> Cao Tong
>>>
>>> On 07/22/2015 04:26 PM, tony_caotong@163.com wrote:
>>>> @prashant, following it the answer of you questions
>>>>
>>>> 1. Yes, primary storage is connected fine for my xenserver.
>>>>
>>>> 2. No, Xenserver's password is not changed.
>>>>
>>>> 3. yes, web UI is fine, and I can login.
>>>>
>>>> 4. before reboot, I unmanaged and disabled resources, and after
>>>> reboot I have enabled all of them.
>>>>
>>>> 5. hosts is states is UP.
>>>>
>>>> 6. No yum update in anywhere.
>>>>
>>>> 7. system VMs status is fine, i think.
>>>>
>>>> -----------
>>>> Cao Tong
>>>>
>>>> On 07/22/2015 04:13 PM, tony_caotong@163.com wrote:
>>>>> Hi,
>>>>>
>>>>> After reinstall, I got the problem again
>>>>>
>>>>> So, I will describe once again.
>>>>>
>>>>> WHAT my environment looks like:
>>>>>
>>>>> I have a ACS server host and a xenserver host, After both reboot, I
>>>>> can not create a VM on xenserver through ACS.
>>>>> A KVM and A NFS are running together in ACS manager host.
>>>>>
>>>>> the status of new VM is always 'staring' on the WEB, but I can
>>>>> create new VM using xencenter.
>>>>>
>>>>> ------------- ERR LOGS ----------
>>>>> 2015-07-22 15:56:56,357 DEBUG [c.c.s.StorageManagerImpl]
>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to send storage pool command
>>>>> to Pool[4|NetworkFilesystem] via 4
>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>> 2829104990918803478 to Host 4 timed out after 3600
>>>>>
>>>>> 2015-07-22 15:56:56,358 INFO [c.c.s.StatsCollector]
>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to reach
>>>>> Pool[4|NetworkFilesystem]
>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>> [StoragePool:4] is unreachable: Unable to send command to the pool
>>>>>
>>>>>
>>>>> ------------- and there are lots of DEBUG infos ------- repeat
>>>>> again and again -----------
>>>>>
>>>>> 2015-07-22 15:36:12,887 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>> (AgentManager-Handler-14:null) Seq 4-8064821032713715922: Forwarding
>>>>> Seq 4-8064821032713715922: { Cmd , MgmtId: 227448510156211, via: 4,
>>>>> Ver: v1, Flags: 100111,
>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>> 116784073679673
>>>>> 2015-07-22 15:36:12,889 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>> (AgentManager-Handler-10:null) Seq 4-8064821032713715883: Forwarding
>>>>> Seq 4-8064821032713715883: { Cmd , MgmtId: 227448510156211, via: 4,
>>>>> Ver: v1, Flags: 100111,
>>>>>
>>>>> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap
>>>>> ache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/5
>>>>> /af949612-838f-3a6d-931b-312e612db740.vhd","origUrl":"http://download.c
>>>>> loud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-30
>>>>> 17-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksu
>>>>> m":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>>>>
>>>>> 5.6(64-bit) no GUI
>>>>>
>>>>> (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"
>>>>> nfs://10.0.0.100/storage/secondary","_role":"Image"}},"name":"centos56-
>>>>> x86_64-xen","hypervisorType":"XenServer"}},"destTO":{"org.apache.clouds
>>>>> tack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com
>>>>> /templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-
>>>>> 8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905c
>>>>> ec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>>>> 5.6(64-bit) no GUI
>>>>>
>>>>> (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.Primar
>>>>> yDataStoreTO":{"uuid":"2df26406-31bf-3a95-8a61-f5008defd9a0","id":4,"po
>>>>> olType":"NetworkFilesystem","host":"10.0.0.100","path":"/storage/xen/pr
>>>>> imary","port":2049,"url":"NetworkFilesystem://10.0.0.100/storage/xen/pr
>>>>> imary/?ROLE=Primary&STOREUUID=2df26406-31bf-3a95-8a61-f5008defd9a0"}},"
>>>>> name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"executeInSe
>>>>> quence":true,"options":{},"wait":10800}}]
>>>>> } to 116784073679673
>>>>>
>>>>>
>>>>> -----------------------------------------
>>>>>
>>>>> Anyone have Any ideas? thanks.
>>>>>
>>>>> -----------
>>>>> Cao Tong
>>>>>
>>>>> On 07/21/2015 06:14 PM, tony_caotong@163.com wrote:
>>>>>> Thanks all,
>>>>>>
>>>>>> I have already reinstall my hosts for preparing a new clear
>>>>>> environment to restart my research.
>>>>>>
>>>>>> -----------
>>>>>> Cao Tong
>>>>>>
>>>>>> On 07/20/2015 09:24 PM, Prashant s wrote:
>>>>>>> some questions :
>>>>>>>
>>>>>>> can you please tell ...
>>>>>>>
>>>>>>> 1. is your NFS storage or your primary Storage Repository in
>>>>>>> connected
>>>>>>> mode with no red cross mark on them in xencenter.
>>>>>>> 2. did you change any passwords on the xenservers ?
>>>>>>> 3. is the cloudstack web ui up , can you login to the cloudstack
>>>>>>> web page.
>>>>>>> 4. *are the zone , pod, or clusters in unmanaged or disabled state
>>>>>>> ? *
>>>>>>> *5. is all the hosts in connected state ? *
>>>>>>> *6. did you run yum update on host reboot on the cs manager vm ? *
>>>>>>> *7. system vms are stateless you can kill them and cs will
>>>>>>> recreate a new
>>>>>>> one .. so dont worry :-) *
>>>>>>>
>>>>>>>
>>>>>>> *thanks *
>>>>>>> *prashant *
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jul 20, 2015 at 3:47 AM, <to...@163.com> wrote:
>>>>>>>
>>>>>>>> Hi, I restartd All hosts (one mgr and xenserver) again.
>>>>>>>>
>>>>>>>>
>>>>>>>> Following is the error log.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
>>>>>>>> (StatsCollector-3:ctx-692a5392) Could not find exception:
>>>>>>>> com.cloud.exception.OperationTimedoutException in error code list
>>>>>>>> for
>>>>>>>> exceptions
>>>>>>>> 2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed
>>>>>>>> out on null
>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972:
>>>>>>>> Cancelling.
>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to send storage pool
>>>>>>>> command to
>>>>>>>> Pool[1|NetworkFilesystem] via 1
>>>>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>>>>> 3176445112179752972 to Host 1 timed out after 3600
>>>>>>>> at
>>>>>>>> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
>>>>>>>> at
>>>>>>>>
>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:
>>>>>>>> 433)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:
>>>>>>>> 362)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>> ava:1000)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>> ava:392)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>> ava:406)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC
>>>>>>>> ollector.java:642)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
>>>>>>>> anagedContextRunnable.java:49)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
>>>>>>>> all(DefaultManagedContext.java:56)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
>>>>>>>> lWithContext(DefaultManagedContext.java:103)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
>>>>>>>> WithContext(DefaultManagedContext.java:53)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
>>>>>>>> agedContextRunnable.java:46)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>>>> 71)
>>>>>>>>
>>>>>>>> at
>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>>> at
>>>>>>>>
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>>>>>>> .java:1145)
>>>>>>>>
>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>> 2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to reach
>>>>>>>> Pool[1|NetworkFilesystem]
>>>>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>>>>> [StoragePool:1]
>>>>>>>> is unreachable: Unable to send command to the pool
>>>>>>>> at
>>>>>>>>
>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>> ava:1010)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>> ava:392)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>> ava:406)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC
>>>>>>>> ollector.java:642)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
>>>>>>>> anagedContextRunnable.java:49)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
>>>>>>>> all(DefaultManagedContext.java:56)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
>>>>>>>> lWithContext(DefaultManagedContext.java:103)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
>>>>>>>> WithContext(DefaultManagedContext.java:53)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
>>>>>>>> agedContextRunnable.java:46)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>>>> 71)
>>>>>>>>
>>>>>>>> at
>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>>> at
>>>>>>>>
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>>>>>>> .java:1145)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto
>>>>>>>> r.java:615)
>>>>>>>>
>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>
>>>>>>>> -----------
>>>>>>>> Cao Tong
>>>>>>>>
>>>>>>>>
>>>>>>>> On 07/20/2015 02:52 PM, tony_caotong@163.com wrote:
>>>>>>>>
>>>>>>>>> No, no one's IP was changed.
>>>>>>>>>
>>>>>>>>> 1. In xenserver I can not login systemvms using the internal IP
>>>>>>>>> like
>>>>>>>>> '169.254.1.112', There shoud be a bridge network for this
>>>>>>>>> right? it is
>>>>>>>>> gone.
>>>>>>>>>
>>>>>>>>> 2. I try to delete xenserver host from CS on web, it also failed
>>>>>>>>> with
>>>>>>>>> lots of logs like following, then memory is full and mangement
>>>>>>>>> down...
>>>>>>>>>
>>>>>>>>> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>> (AgentManager-Handler-15:null) Seq 1-7282039122481381399:
>>>>>>>>> Forwarding Seq
>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>>>> Ver: v1,
>>>>>>>>> Flags: 100111,
>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>> 192405008094602
>>>>>>>>> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>> (AgentManager-Handler-5:null) Seq 1-7282039122481381399:
>>>>>>>>> Forwarding Seq
>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>>>> Ver: v1,
>>>>>>>>> Flags: 100111,
>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>> 192405008094602
>>>>>>>>> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>> (AgentManager-Handler-1:null) Seq 1-7282039122481381399:
>>>>>>>>> Forwarding Seq
>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>>>> Ver: v1,
>>>>>>>>> Flags: 100111,
>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>> 192405008094602
>>>>>>>>> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>> (AgentManager-Handler-14:null) Seq 1-7282039122481381399:
>>>>>>>>> Forwarding Seq
>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>>>> Ver: v1,
>>>>>>>>> Flags: 100111,
>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>> 192405008094602
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I guess that, is there some service or daemons working for CS
>>>>>>>>> is not up
>>>>>>>>> on Xenserver ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----------
>>>>>>>>> Cao Tong
>>>>>>>>> On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
>>>>>>>>>
>>>>>>>>>> Did the management server ip change?
>>>>>>>>>> management server ip in the configuration table is used my
>>>>>>>>>> systemvms.
>>>>>>>>>> select * from configuration where name like 'host';
>>>>>>>>>>
>>>>>>>>>> If it changed, correct the value in db and restart systemvms.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ~Rajani
>>>>>>>>>>
>>>>>>>>>> On Mon, Jul 20, 2015 at 11:56 AM,<to...@163.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>> I shutdown my cs-manager and xenserver last weekend, And now
>>>>>>>>>>> the ssvm
>>>>>>>>>>> and cpvm is disconnect, thost two was runing on xenserver. so
>>>>>>>>>>> What
>>>>>>>>>>> should i do right now ?
>>>>>>>>>>> Please anybody help me and thanks.
>>>>>>>>>>>
>>>>>>>>>>> In xenserver I found that the three system VMs are not running.
>>>>>>>>>>> my xenserver seems can not reconnect to CS-manager. and it
>>>>>>>>>>> seams not
>>>>>>>>>>> under control of CS.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> What is the right steps of shutdown all CS group machines and
>>>>>>>>>>> resume
>>>>>>>>>>> them?
>>>>>>>>>>> How can i let my xenserver reconnected ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> -----------
>>>>>>>>>>> Cao Tong
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
Re: XenServer is disconnected after CS hosts shutdown
Posted by Yiping Zhang <yz...@marketo.com>.
Well, sometimes people can’t answer a question because of lack of
relevant information, or simply because no one has encountered a similar
situation before.
Looking at your past messages on this thread, there were no mentions about
primary storage. Obviously, your primary storage configuration had changed
between the time you shut down CS manager and xenservers and the time you
restarted them. That is the vital info the list didn’t know.
To best of my knowledge, zone wide primary storage has never been
supported for Xen hypervisors.
I do have to say that quite often CloudStack error messages are very
cryptic, do not provide enough *useful* information to help users identify
and trouble shoot actual problems. Those stack trace output might be a
gold mine to developers, but they are utterly useless for end users.
Just my $0.02
Yiping
On 7/28/15, 11:19 PM, "tony_caotong@163.com" <to...@163.com> wrote:
>
>Hi, Finally I resolved this problem by my self.
>
> * Primary Storage: A storage resource typically provided to a single
> cluster for the actual running of instance disk images. (Zone-wide
> primary storage is an option, though not typically used.)
>
>This line above is from
>http://docs.cloudstack.apache.org/en/master/concepts.html
>
>Because I have a Zone-wide primary storage, ACS can not find the correct
>primary which belong to XenServer cluster after reboot.
>
>Then I change the Zone-wide primary to cluster-wide, it resolved.
>
>Right now, I have two primary storage, one is kvm cluster-wide, another
>is xenserver cluster-wide.
>
>Above is for people who have the same problem oneday.
>
>by the way, I am very curious why I never receive replys from this a big
>community ?? of course except the very beginning.
>
>Is my English skill really really poor, result in no body can understood
>what language I am speaking ?
>
>-----------
>Cao Tong
>
>On 07/22/2015 09:03 PM, tony_caotong@163.com wrote:
>>
>> Hey! help please...
>>
>> some news.
>> I think the cause is that the ACS host can't communicate with
>> XenServer host.
>> ACS continues outputing logs like this
>>
>> 2015-07-22 20:42:13,555 DEBUG [c.c.a.m.ClusteredAgentAttache]
>> (AgentManager-Handler-7:null) Seq 5-8174877748607582212: Forwarding
>> Seq 5-8174877748607582212: { Cmd , MgmtId: 279278805451459, via: 5,
>> Ver: v1, Flags: 100111,
>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>280345368052992
>>
>> I am not sure that if the ACS status is wrong or some services on
>> xenserver are not opend.
>>
>> on xenserver , I found *xenheartbeat.sh is not running.*
>> *(/bin/bash /opt/cloud/bin/xenheartbeat.sh
>> 00d8e0d0-8561-4b3d-9044-cbc496ff22cc 120 60)*
>>
>> As some operations about xenserver was pending， xenserver can not be
>> deleted from web UI.
>>
>> I got a temporary solution
>>
>> 1. delete jobs from DB cloud.vm_work_job.
>> 2. delete xenserver from DB cloud.host.
>> 3. add xenserver host back from web UI.
>>
>> then it works.
>>
>> Does anyone have a idea for this?
>>
>> Could anyone tell what things does ACS do on xenserver host when
>> adding a xenserver ?
>>
>> Thanks,
>>
>> -----------
>> Cao Tong
>>
>> On 07/22/2015 04:26 PM, tony_caotong@163.com wrote:
>>>
>>> @prashant, following it the answer of you questions
>>>
>>> 1. Yes, primary storage is connected fine for my xenserver.
>>>
>>> 2. No, Xenserver's password is not changed.
>>>
>>> 3. yes, web UI is fine, and I can login.
>>>
>>> 4. before reboot, I unmanaged and disabled resources, and after
>>> reboot I have enabled all of them.
>>>
>>> 5. hosts is states is UP.
>>>
>>> 6. No yum update in anywhere.
>>>
>>> 7. system VMs status is fine, i think.
>>>
>>> -----------
>>> Cao Tong
>>>
>>> On 07/22/2015 04:13 PM, tony_caotong@163.com wrote:
>>>>
>>>> Hi,
>>>>
>>>> After reinstall, I got the problem again
>>>>
>>>> So, I will describe once again.
>>>>
>>>> WHAT my environment looks like:
>>>>
>>>> I have a ACS server host and a xenserver host, After both reboot, I
>>>> can not create a VM on xenserver through ACS.
>>>> A KVM and A NFS are running together in ACS manager host.
>>>>
>>>> the status of new VM is always 'staring' on the WEB, but I can
>>>> create new VM using xencenter.
>>>>
>>>> ------------- ERR LOGS ----------
>>>> 2015-07-22 15:56:56,357 DEBUG [c.c.s.StorageManagerImpl]
>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to send storage pool command
>>>> to Pool[4|NetworkFilesystem] via 4
>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>> 2829104990918803478 to Host 4 timed out after 3600
>>>>
>>>> 2015-07-22 15:56:56,358 INFO [c.c.s.StatsCollector]
>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to reach
>>>> Pool[4|NetworkFilesystem]
>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>> [StoragePool:4] is unreachable: Unable to send command to the pool
>>>>
>>>>
>>>> ------------- and there are lots of DEBUG infos ------- repeat
>>>> again and again -----------
>>>>
>>>> 2015-07-22 15:36:12,887 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>> (AgentManager-Handler-14:null) Seq 4-8064821032713715922: Forwarding
>>>> Seq 4-8064821032713715922: { Cmd , MgmtId: 227448510156211, via: 4,
>>>> Ver: v1, Flags: 100111,
>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>> 116784073679673
>>>> 2015-07-22 15:36:12,889 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>> (AgentManager-Handler-10:null) Seq 4-8064821032713715883: Forwarding
>>>> Seq 4-8064821032713715883: { Cmd , MgmtId: 227448510156211, via: 4,
>>>> Ver: v1, Flags: 100111,
>>>>
>>>>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap
>>>>ache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/5
>>>>/af949612-838f-3a6d-931b-312e612db740.vhd","origUrl":"http://download.c
>>>>loud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-30
>>>>17-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksu
>>>>m":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>>>
>>>> 5.6(64-bit) no GUI
>>>>
>>>>(XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"
>>>>nfs://10.0.0.100/storage/secondary","_role":"Image"}},"name":"centos56-
>>>>x86_64-xen","hypervisorType":"XenServer"}},"destTO":{"org.apache.clouds
>>>>tack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com
>>>>/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-
>>>>8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905c
>>>>ec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>>> 5.6(64-bit) no GUI
>>>>
>>>>(XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.Primar
>>>>yDataStoreTO":{"uuid":"2df26406-31bf-3a95-8a61-f5008defd9a0","id":4,"po
>>>>olType":"NetworkFilesystem","host":"10.0.0.100","path":"/storage/xen/pr
>>>>imary","port":2049,"url":"NetworkFilesystem://10.0.0.100/storage/xen/pr
>>>>imary/?ROLE=Primary&STOREUUID=2df26406-31bf-3a95-8a61-f5008defd9a0"}},"
>>>>name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"executeInSe
>>>>quence":true,"options":{},"wait":10800}}]
>>>> } to 116784073679673
>>>>
>>>>
>>>> -----------------------------------------
>>>>
>>>> Anyone have Any ideas? thanks.
>>>>
>>>> -----------
>>>> Cao Tong
>>>>
>>>> On 07/21/2015 06:14 PM, tony_caotong@163.com wrote:
>>>>>
>>>>> Thanks all,
>>>>>
>>>>> I have already reinstall my hosts for preparing a new clear
>>>>> environment to restart my research.
>>>>>
>>>>> -----------
>>>>> Cao Tong
>>>>>
>>>>> On 07/20/2015 09:24 PM, Prashant s wrote:
>>>>>> some questions :
>>>>>>
>>>>>> can you please tell ...
>>>>>>
>>>>>> 1. is your NFS storage or your primary Storage Repository in
>>>>>> connected
>>>>>> mode with no red cross mark on them in xencenter.
>>>>>> 2. did you change any passwords on the xenservers ?
>>>>>> 3. is the cloudstack web ui up , can you login to the cloudstack
>>>>>> web page.
>>>>>> 4. *are the zone , pod, or clusters in unmanaged or disabled state
>>>>>> ? *
>>>>>> *5. is all the hosts in connected state ? *
>>>>>> *6. did you run yum update on host reboot on the cs manager vm ? *
>>>>>> *7. system vms are stateless you can kill them and cs will
>>>>>> recreate a new
>>>>>> one .. so dont worry :-) *
>>>>>>
>>>>>>
>>>>>> *thanks *
>>>>>> *prashant *
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Jul 20, 2015 at 3:47 AM, <to...@163.com> wrote:
>>>>>>
>>>>>>> Hi, I restartd All hosts (one mgr and xenserver) again.
>>>>>>>
>>>>>>>
>>>>>>> Following is the error log.
>>>>>>>
>>>>>>>
>>>>>>> 2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
>>>>>>> (StatsCollector-3:ctx-692a5392) Could not find exception:
>>>>>>> com.cloud.exception.OperationTimedoutException in error code list
>>>>>>> for
>>>>>>> exceptions
>>>>>>> 2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed
>>>>>>> out on null
>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972:
>>>>>>> Cancelling.
>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to send storage pool
>>>>>>> command to
>>>>>>> Pool[1|NetworkFilesystem] via 1
>>>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>>>> 3176445112179752972 to Host 1 timed out after 3600
>>>>>>> at
>>>>>>> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:
>>>>>>>433)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:
>>>>>>>362)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>ava:1000)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>ava:392)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>ava:406)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC
>>>>>>>ollector.java:642)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
>>>>>>>anagedContextRunnable.java:49)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
>>>>>>>all(DefaultManagedContext.java:56)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
>>>>>>>lWithContext(DefaultManagedContext.java:103)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
>>>>>>>WithContext(DefaultManagedContext.java:53)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
>>>>>>>agedContextRunnable.java:46)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>>>71)
>>>>>>>
>>>>>>> at
>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>> at
>>>>>>>
>>>>>>>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>>>>>>.java:1145)
>>>>>>>
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to reach
>>>>>>> Pool[1|NetworkFilesystem]
>>>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>>>> [StoragePool:1]
>>>>>>> is unreachable: Unable to send command to the pool
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>ava:1010)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>ava:392)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>ava:406)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC
>>>>>>>ollector.java:642)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
>>>>>>>anagedContextRunnable.java:49)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
>>>>>>>all(DefaultManagedContext.java:56)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
>>>>>>>lWithContext(DefaultManagedContext.java:103)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
>>>>>>>WithContext(DefaultManagedContext.java:53)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
>>>>>>>agedContextRunnable.java:46)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>>>71)
>>>>>>>
>>>>>>> at
>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>> at
>>>>>>>
>>>>>>>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>>>>>>.java:1145)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto
>>>>>>>r.java:615)
>>>>>>>
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> -----------
>>>>>>> Cao Tong
>>>>>>>
>>>>>>>
>>>>>>> On 07/20/2015 02:52 PM, tony_caotong@163.com wrote:
>>>>>>>
>>>>>>>> No, no one's IP was changed.
>>>>>>>>
>>>>>>>> 1. In xenserver I can not login systemvms using the internal IP
>>>>>>>> like
>>>>>>>> '169.254.1.112', There shoud be a bridge network for this
>>>>>>>> right? it is
>>>>>>>> gone.
>>>>>>>>
>>>>>>>> 2. I try to delete xenserver host from CS on web, it also failed
>>>>>>>> with
>>>>>>>> lots of logs like following, then memory is full and mangement
>>>>>>>> down...
>>>>>>>>
>>>>>>>> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>> (AgentManager-Handler-15:null) Seq 1-7282039122481381399:
>>>>>>>> Forwarding Seq
>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>>> Ver: v1,
>>>>>>>> Flags: 100111,
>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>> 192405008094602
>>>>>>>> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>> (AgentManager-Handler-5:null) Seq 1-7282039122481381399:
>>>>>>>> Forwarding Seq
>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>>> Ver: v1,
>>>>>>>> Flags: 100111,
>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>> 192405008094602
>>>>>>>> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>> (AgentManager-Handler-1:null) Seq 1-7282039122481381399:
>>>>>>>> Forwarding Seq
>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>>> Ver: v1,
>>>>>>>> Flags: 100111,
>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>> 192405008094602
>>>>>>>> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>> (AgentManager-Handler-14:null) Seq 1-7282039122481381399:
>>>>>>>> Forwarding Seq
>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>>> Ver: v1,
>>>>>>>> Flags: 100111,
>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>> 192405008094602
>>>>>>>>
>>>>>>>>
>>>>>>>> I guess that, is there some service or daemons working for CS
>>>>>>>> is not up
>>>>>>>> on Xenserver ?
>>>>>>>>
>>>>>>>>
>>>>>>>> -----------
>>>>>>>> Cao Tong
>>>>>>>> On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
>>>>>>>>
>>>>>>>>> Did the management server ip change?
>>>>>>>>> management server ip in the configuration table is used my
>>>>>>>>> systemvms.
>>>>>>>>> select * from configuration where name like 'host';
>>>>>>>>>
>>>>>>>>> If it changed, correct the value in db and restart systemvms.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ~Rajani
>>>>>>>>>
>>>>>>>>> On Mon, Jul 20, 2015 at 11:56 AM,<to...@163.com> wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>> I shutdown my cs-manager and xenserver last weekend, And now
>>>>>>>>>> the ssvm
>>>>>>>>>> and cpvm is disconnect, thost two was runing on xenserver. so
>>>>>>>>>> What
>>>>>>>>>> should i do right now ?
>>>>>>>>>> Please anybody help me and thanks.
>>>>>>>>>>
>>>>>>>>>> In xenserver I found that the three system VMs are not running.
>>>>>>>>>> my xenserver seems can not reconnect to CS-manager. and it
>>>>>>>>>> seams not
>>>>>>>>>> under control of CS.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> What is the right steps of shutdown all CS group machines and
>>>>>>>>>> resume
>>>>>>>>>> them?
>>>>>>>>>> How can i let my xenserver reconnected ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> -----------
>>>>>>>>>> Cao Tong
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
Re: XenServer is disconnected after CS hosts shutdown
Posted by to...@163.com.
Hi, Finally I resolved this problem by my self.
* Primary Storage: A storage resource typically provided to a single
cluster for the actual running of instance disk images. (Zone-wide
primary storage is an option, though not typically used.)
This line above is from
http://docs.cloudstack.apache.org/en/master/concepts.html
Because I have a Zone-wide primary storage, ACS can not find the correct
primary which belong to XenServer cluster after reboot.
Then I change the Zone-wide primary to cluster-wide, it resolved.
Right now, I have two primary storage, one is kvm cluster-wide, another
is xenserver cluster-wide.
Above is for people who have the same problem oneday.
by the way, I am very curious why I never receive replys from this a big
community ?? of course except the very beginning.
Is my English skill really really poor, result in no body can understood
what language I am speaking ?
-----------
Cao Tong
On 07/22/2015 09:03 PM, tony_caotong@163.com wrote:
>
> Hey! help please...
>
> some news.
> I think the cause is that the ACS host can't communicate with
> XenServer host.
> ACS continues outputing logs like this
>
> 2015-07-22 20:42:13,555 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-7:null) Seq 5-8174877748607582212: Forwarding
> Seq 5-8174877748607582212: { Cmd , MgmtId: 279278805451459, via: 5,
> Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to 280345368052992
>
> I am not sure that if the ACS status is wrong or some services on
> xenserver are not opend.
>
> on xenserver , I found *xenheartbeat.sh is not running.*
> *(/bin/bash /opt/cloud/bin/xenheartbeat.sh
> 00d8e0d0-8561-4b3d-9044-cbc496ff22cc 120 60)*
>
> As some operations about xenserver was pending， xenserver can not be
> deleted from web UI.
>
> I got a temporary solution
>
> 1. delete jobs from DB cloud.vm_work_job.
> 2. delete xenserver from DB cloud.host.
> 3. add xenserver host back from web UI.
>
> then it works.
>
> Does anyone have a idea for this?
>
> Could anyone tell what things does ACS do on xenserver host when
> adding a xenserver ?
>
> Thanks,
>
> -----------
> Cao Tong
>
> On 07/22/2015 04:26 PM, tony_caotong@163.com wrote:
>>
>> @prashant, following it the answer of you questions
>>
>> 1. Yes, primary storage is connected fine for my xenserver.
>>
>> 2. No, Xenserver's password is not changed.
>>
>> 3. yes, web UI is fine, and I can login.
>>
>> 4. before reboot, I unmanaged and disabled resources, and after
>> reboot I have enabled all of them.
>>
>> 5. hosts is states is UP.
>>
>> 6. No yum update in anywhere.
>>
>> 7. system VMs status is fine, i think.
>>
>> -----------
>> Cao Tong
>>
>> On 07/22/2015 04:13 PM, tony_caotong@163.com wrote:
>>>
>>> Hi,
>>>
>>> After reinstall, I got the problem again
>>>
>>> So, I will describe once again.
>>>
>>> WHAT my environment looks like:
>>>
>>> I have a ACS server host and a xenserver host, After both reboot, I
>>> can not create a VM on xenserver through ACS.
>>> A KVM and A NFS are running together in ACS manager host.
>>>
>>> the status of new VM is always 'staring' on the WEB, but I can
>>> create new VM using xencenter.
>>>
>>> ------------- ERR LOGS ----------
>>> 2015-07-22 15:56:56,357 DEBUG [c.c.s.StorageManagerImpl]
>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to send storage pool command
>>> to Pool[4|NetworkFilesystem] via 4
>>> com.cloud.exception.OperationTimedoutException: Commands
>>> 2829104990918803478 to Host 4 timed out after 3600
>>>
>>> 2015-07-22 15:56:56,358 INFO [c.c.s.StatsCollector]
>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to reach
>>> Pool[4|NetworkFilesystem]
>>> com.cloud.exception.StorageUnavailableException: Resource
>>> [StoragePool:4] is unreachable: Unable to send command to the pool
>>>
>>>
>>> ------------- and there are lots of DEBUG infos ------- repeat
>>> again and again -----------
>>>
>>> 2015-07-22 15:36:12,887 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>> (AgentManager-Handler-14:null) Seq 4-8064821032713715922: Forwarding
>>> Seq 4-8064821032713715922: { Cmd , MgmtId: 227448510156211, via: 4,
>>> Ver: v1, Flags: 100111,
>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>> 116784073679673
>>> 2015-07-22 15:36:12,889 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>> (AgentManager-Handler-10:null) Seq 4-8064821032713715883: Forwarding
>>> Seq 4-8064821032713715883: { Cmd , MgmtId: 227448510156211, via: 4,
>>> Ver: v1, Flags: 100111,
>>> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/5/af949612-838f-3a6d-931b-312e612db740.vhd","origUrl":"http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>> 5.6(64-bit) no GUI
>>> (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.0.0.100/storage/secondary","_role":"Image"}},"name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>> 5.6(64-bit) no GUI
>>> (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2df26406-31bf-3a95-8a61-f5008defd9a0","id":4,"poolType":"NetworkFilesystem","host":"10.0.0.100","path":"/storage/xen/primary","port":2049,"url":"NetworkFilesystem://10.0.0.100/storage/xen/primary/?ROLE=Primary&STOREUUID=2df26406-31bf-3a95-8a61-f5008defd9a0"}},"name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"executeInSequence":true,"options":{},"wait":10800}}]
>>> } to 116784073679673
>>>
>>>
>>> -----------------------------------------
>>>
>>> Anyone have Any ideas? thanks.
>>>
>>> -----------
>>> Cao Tong
>>>
>>> On 07/21/2015 06:14 PM, tony_caotong@163.com wrote:
>>>>
>>>> Thanks all,
>>>>
>>>> I have already reinstall my hosts for preparing a new clear
>>>> environment to restart my research.
>>>>
>>>> -----------
>>>> Cao Tong
>>>>
>>>> On 07/20/2015 09:24 PM, Prashant s wrote:
>>>>> some questions :
>>>>>
>>>>> can you please tell ...
>>>>>
>>>>> 1. is your NFS storage or your primary Storage Repository in
>>>>> connected
>>>>> mode with no red cross mark on them in xencenter.
>>>>> 2. did you change any passwords on the xenservers ?
>>>>> 3. is the cloudstack web ui up , can you login to the cloudstack
>>>>> web page.
>>>>> 4. *are the zone , pod, or clusters in unmanaged or disabled state
>>>>> ? *
>>>>> *5. is all the hosts in connected state ? *
>>>>> *6. did you run yum update on host reboot on the cs manager vm ? *
>>>>> *7. system vms are stateless you can kill them and cs will
>>>>> recreate a new
>>>>> one .. so dont worry :-) *
>>>>>
>>>>>
>>>>> *thanks *
>>>>> *prashant *
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jul 20, 2015 at 3:47 AM, <to...@163.com> wrote:
>>>>>
>>>>>> Hi, I restartd All hosts (one mgr and xenserver) again.
>>>>>>
>>>>>>
>>>>>> Following is the error log.
>>>>>>
>>>>>>
>>>>>> 2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
>>>>>> (StatsCollector-3:ctx-692a5392) Could not find exception:
>>>>>> com.cloud.exception.OperationTimedoutException in error code list
>>>>>> for
>>>>>> exceptions
>>>>>> 2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed
>>>>>> out on null
>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972:
>>>>>> Cancelling.
>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
>>>>>> (StatsCollector-3:ctx-692a5392) Unable to send storage pool
>>>>>> command to
>>>>>> Pool[1|NetworkFilesystem] via 1
>>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>>> 3176445112179752972 to Host 1 timed out after 3600
>>>>>> at
>>>>>> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
>>>>>> at
>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:433)
>>>>>>
>>>>>> at
>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
>>>>>>
>>>>>> at
>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1000)
>>>>>>
>>>>>> at
>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
>>>>>>
>>>>>> at
>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
>>>>>>
>>>>>> at
>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
>>>>>>
>>>>>> at
>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>
>>>>>> at
>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>
>>>>>> at
>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>
>>>>>> at
>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>
>>>>>> at
>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>
>>>>>> at
>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
>>>>>> (StatsCollector-3:ctx-692a5392) Unable to reach
>>>>>> Pool[1|NetworkFilesystem]
>>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>>> [StoragePool:1]
>>>>>> is unreachable: Unable to send command to the pool
>>>>>> at
>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1010)
>>>>>>
>>>>>> at
>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
>>>>>>
>>>>>> at
>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
>>>>>>
>>>>>> at
>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
>>>>>>
>>>>>> at
>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>
>>>>>> at
>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>
>>>>>> at
>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>
>>>>>> at
>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>
>>>>>> at
>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>
>>>>>> at
>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> -----------
>>>>>> Cao Tong
>>>>>>
>>>>>>
>>>>>> On 07/20/2015 02:52 PM, tony_caotong@163.com wrote:
>>>>>>
>>>>>>> No, no one's IP was changed.
>>>>>>>
>>>>>>> 1. In xenserver I can not login systemvms using the internal IP
>>>>>>> like
>>>>>>> '169.254.1.112', There shoud be a bridge network for this
>>>>>>> right? it is
>>>>>>> gone.
>>>>>>>
>>>>>>> 2. I try to delete xenserver host from CS on web, it also failed
>>>>>>> with
>>>>>>> lots of logs like following, then memory is full and mangement
>>>>>>> down...
>>>>>>>
>>>>>>> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>> (AgentManager-Handler-15:null) Seq 1-7282039122481381399:
>>>>>>> Forwarding Seq
>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>> Ver: v1,
>>>>>>> Flags: 100111,
>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>> 192405008094602
>>>>>>> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>> (AgentManager-Handler-5:null) Seq 1-7282039122481381399:
>>>>>>> Forwarding Seq
>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>> Ver: v1,
>>>>>>> Flags: 100111,
>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>> 192405008094602
>>>>>>> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>> (AgentManager-Handler-1:null) Seq 1-7282039122481381399:
>>>>>>> Forwarding Seq
>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>> Ver: v1,
>>>>>>> Flags: 100111,
>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>> 192405008094602
>>>>>>> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>> (AgentManager-Handler-14:null) Seq 1-7282039122481381399:
>>>>>>> Forwarding Seq
>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>> Ver: v1,
>>>>>>> Flags: 100111,
>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>> 192405008094602
>>>>>>>
>>>>>>>
>>>>>>> I guess that, is there some service or daemons working for CS
>>>>>>> is not up
>>>>>>> on Xenserver ?
>>>>>>>
>>>>>>>
>>>>>>> -----------
>>>>>>> Cao Tong
>>>>>>> On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
>>>>>>>
>>>>>>>> Did the management server ip change?
>>>>>>>> management server ip in the configuration table is used my
>>>>>>>> systemvms.
>>>>>>>> select * from configuration where name like 'host';
>>>>>>>>
>>>>>>>> If it changed, correct the value in db and restart systemvms.
>>>>>>>>
>>>>>>>>
>>>>>>>> ~Rajani
>>>>>>>>
>>>>>>>> On Mon, Jul 20, 2015 at 11:56 AM,<to...@163.com> wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>> I shutdown my cs-manager and xenserver last weekend, And now
>>>>>>>>> the ssvm
>>>>>>>>> and cpvm is disconnect, thost two was runing on xenserver. so
>>>>>>>>> What
>>>>>>>>> should i do right now ?
>>>>>>>>> Please anybody help me and thanks.
>>>>>>>>>
>>>>>>>>> In xenserver I found that the three system VMs are not running.
>>>>>>>>> my xenserver seems can not reconnect to CS-manager. and it
>>>>>>>>> seams not
>>>>>>>>> under control of CS.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> What is the right steps of shutdown all CS group machines and
>>>>>>>>> resume
>>>>>>>>> them?
>>>>>>>>> How can i let my xenserver reconnected ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> -----------
>>>>>>>>> Cao Tong
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
Re: XenServer is disconnected after CS hosts shutdown
Posted by to...@163.com.
Hey! help please...
some news.
I think the cause is that the ACS host can't communicate with XenServer
host.
ACS continues outputing logs like this
2015-07-22 20:42:13,555 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-7:null) Seq 5-8174877748607582212: Forwarding Seq
5-8174877748607582212: { Cmd , MgmtId: 279278805451459, via: 5, Ver:
v1, Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
} to 280345368052992
I am not sure that if the ACS status is wrong or some services on
xenserver are not opend.
on xenserver , I found *xenheartbeat.sh is not running.*
*(/bin/bash /opt/cloud/bin/xenheartbeat.sh
00d8e0d0-8561-4b3d-9044-cbc496ff22cc 120 60)*
As some operations about xenserver was pending， xenserver can not be
deleted from web UI.
I got a temporary solution
1. delete jobs from DB cloud.vm_work_job.
2. delete xenserver from DB cloud.host.
3. add xenserver host back from web UI.
then it works.
Does anyone have a idea for this?
Could anyone tell what things does ACS do on xenserver host when adding
a xenserver ?
Thanks,
-----------
Cao Tong
On 07/22/2015 04:26 PM, tony_caotong@163.com wrote:
>
> @prashant, following it the answer of you questions
>
> 1. Yes, primary storage is connected fine for my xenserver.
>
> 2. No, Xenserver's password is not changed.
>
> 3. yes, web UI is fine, and I can login.
>
> 4. before reboot, I unmanaged and disabled resources, and after
> reboot I have enabled all of them.
>
> 5. hosts is states is UP.
>
> 6. No yum update in anywhere.
>
> 7. system VMs status is fine, i think.
>
> -----------
> Cao Tong
>
> On 07/22/2015 04:13 PM, tony_caotong@163.com wrote:
>>
>> Hi,
>>
>> After reinstall, I got the problem again
>>
>> So, I will describe once again.
>>
>> WHAT my environment looks like:
>>
>> I have a ACS server host and a xenserver host, After both reboot, I
>> can not create a VM on xenserver through ACS.
>> A KVM and A NFS are running together in ACS manager host.
>>
>> the status of new VM is always 'staring' on the WEB, but I can create
>> new VM using xencenter.
>>
>> ------------- ERR LOGS ----------
>> 2015-07-22 15:56:56,357 DEBUG [c.c.s.StorageManagerImpl]
>> (StatsCollector-3:ctx-1aa2e8c9) Unable to send storage pool command
>> to Pool[4|NetworkFilesystem] via 4
>> com.cloud.exception.OperationTimedoutException: Commands
>> 2829104990918803478 to Host 4 timed out after 3600
>>
>> 2015-07-22 15:56:56,358 INFO [c.c.s.StatsCollector]
>> (StatsCollector-3:ctx-1aa2e8c9) Unable to reach
>> Pool[4|NetworkFilesystem]
>> com.cloud.exception.StorageUnavailableException: Resource
>> [StoragePool:4] is unreachable: Unable to send command to the pool
>>
>>
>> ------------- and there are lots of DEBUG infos ------- repeat again
>> and again -----------
>>
>> 2015-07-22 15:36:12,887 DEBUG [c.c.a.m.ClusteredAgentAttache]
>> (AgentManager-Handler-14:null) Seq 4-8064821032713715922: Forwarding
>> Seq 4-8064821032713715922: { Cmd , MgmtId: 227448510156211, via: 4,
>> Ver: v1, Flags: 100111,
>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>> 116784073679673
>> 2015-07-22 15:36:12,889 DEBUG [c.c.a.m.ClusteredAgentAttache]
>> (AgentManager-Handler-10:null) Seq 4-8064821032713715883: Forwarding
>> Seq 4-8064821032713715883: { Cmd , MgmtId: 227448510156211, via: 4,
>> Ver: v1, Flags: 100111,
>> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/5/af949612-838f-3a6d-931b-312e612db740.vhd","origUrl":"http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>> 5.6(64-bit) no GUI
>> (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.0.0.100/storage/secondary","_role":"Image"}},"name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>> 5.6(64-bit) no GUI
>> (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2df26406-31bf-3a95-8a61-f5008defd9a0","id":4,"poolType":"NetworkFilesystem","host":"10.0.0.100","path":"/storage/xen/primary","port":2049,"url":"NetworkFilesystem://10.0.0.100/storage/xen/primary/?ROLE=Primary&STOREUUID=2df26406-31bf-3a95-8a61-f5008defd9a0"}},"name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"executeInSequence":true,"options":{},"wait":10800}}]
>> } to 116784073679673
>>
>>
>> -----------------------------------------
>>
>> Anyone have Any ideas? thanks.
>>
>> -----------
>> Cao Tong
>>
>> On 07/21/2015 06:14 PM, tony_caotong@163.com wrote:
>>>
>>> Thanks all,
>>>
>>> I have already reinstall my hosts for preparing a new clear
>>> environment to restart my research.
>>>
>>> -----------
>>> Cao Tong
>>>
>>> On 07/20/2015 09:24 PM, Prashant s wrote:
>>>> some questions :
>>>>
>>>> can you please tell ...
>>>>
>>>> 1. is your NFS storage or your primary Storage Repository in connected
>>>> mode with no red cross mark on them in xencenter.
>>>> 2. did you change any passwords on the xenservers ?
>>>> 3. is the cloudstack web ui up , can you login to the cloudstack
>>>> web page.
>>>> 4. *are the zone , pod, or clusters in unmanaged or disabled state ? *
>>>> *5. is all the hosts in connected state ? *
>>>> *6. did you run yum update on host reboot on the cs manager vm ? *
>>>> *7. system vms are stateless you can kill them and cs will recreate
>>>> a new
>>>> one .. so dont worry :-) *
>>>>
>>>>
>>>> *thanks *
>>>> *prashant *
>>>>
>>>>
>>>>
>>>> On Mon, Jul 20, 2015 at 3:47 AM, <to...@163.com> wrote:
>>>>
>>>>> Hi, I restartd All hosts (one mgr and xenserver) again.
>>>>>
>>>>>
>>>>> Following is the error log.
>>>>>
>>>>>
>>>>> 2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
>>>>> (StatsCollector-3:ctx-692a5392) Could not find exception:
>>>>> com.cloud.exception.OperationTimedoutException in error code list for
>>>>> exceptions
>>>>> 2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed
>>>>> out on null
>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972:
>>>>> Cancelling.
>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
>>>>> (StatsCollector-3:ctx-692a5392) Unable to send storage pool
>>>>> command to
>>>>> Pool[1|NetworkFilesystem] via 1
>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>> 3176445112179752972 to Host 1 timed out after 3600
>>>>> at
>>>>> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
>>>>> at
>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:433)
>>>>>
>>>>> at
>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
>>>>>
>>>>> at
>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1000)
>>>>>
>>>>> at
>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
>>>>>
>>>>> at
>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
>>>>>
>>>>> at
>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>
>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>
>>>>> at
>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>> 2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
>>>>> (StatsCollector-3:ctx-692a5392) Unable to reach
>>>>> Pool[1|NetworkFilesystem]
>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>> [StoragePool:1]
>>>>> is unreachable: Unable to send command to the pool
>>>>> at
>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1010)
>>>>>
>>>>> at
>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
>>>>>
>>>>> at
>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
>>>>>
>>>>> at
>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>
>>>>> at
>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>
>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>
>>>>> at
>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> -----------
>>>>> Cao Tong
>>>>>
>>>>>
>>>>> On 07/20/2015 02:52 PM, tony_caotong@163.com wrote:
>>>>>
>>>>>> No, no one's IP was changed.
>>>>>>
>>>>>> 1. In xenserver I can not login systemvms using the internal IP like
>>>>>> '169.254.1.112', There shoud be a bridge network for this
>>>>>> right? it is
>>>>>> gone.
>>>>>>
>>>>>> 2. I try to delete xenserver host from CS on web, it also failed
>>>>>> with
>>>>>> lots of logs like following, then memory is full and mangement
>>>>>> down...
>>>>>>
>>>>>> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>> (AgentManager-Handler-15:null) Seq 1-7282039122481381399:
>>>>>> Forwarding Seq
>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>> Ver: v1,
>>>>>> Flags: 100111,
>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>> 192405008094602
>>>>>> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>> (AgentManager-Handler-5:null) Seq 1-7282039122481381399:
>>>>>> Forwarding Seq
>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>> Ver: v1,
>>>>>> Flags: 100111,
>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>> 192405008094602
>>>>>> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>> (AgentManager-Handler-1:null) Seq 1-7282039122481381399:
>>>>>> Forwarding Seq
>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>> Ver: v1,
>>>>>> Flags: 100111,
>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>> 192405008094602
>>>>>> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>> (AgentManager-Handler-14:null) Seq 1-7282039122481381399:
>>>>>> Forwarding Seq
>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>> Ver: v1,
>>>>>> Flags: 100111,
>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>> 192405008094602
>>>>>>
>>>>>>
>>>>>> I guess that, is there some service or daemons working for CS is
>>>>>> not up
>>>>>> on Xenserver ?
>>>>>>
>>>>>>
>>>>>> -----------
>>>>>> Cao Tong
>>>>>> On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
>>>>>>
>>>>>>> Did the management server ip change?
>>>>>>> management server ip in the configuration table is used my
>>>>>>> systemvms.
>>>>>>> select * from configuration where name like 'host';
>>>>>>>
>>>>>>> If it changed, correct the value in db and restart systemvms.
>>>>>>>
>>>>>>>
>>>>>>> ~Rajani
>>>>>>>
>>>>>>> On Mon, Jul 20, 2015 at 11:56 AM,<to...@163.com> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>> I shutdown my cs-manager and xenserver last weekend, And now
>>>>>>>> the ssvm
>>>>>>>> and cpvm is disconnect, thost two was runing on xenserver. so What
>>>>>>>> should i do right now ?
>>>>>>>> Please anybody help me and thanks.
>>>>>>>>
>>>>>>>> In xenserver I found that the three system VMs are not running.
>>>>>>>> my xenserver seems can not reconnect to CS-manager. and it
>>>>>>>> seams not
>>>>>>>> under control of CS.
>>>>>>>>
>>>>>>>>
>>>>>>>> What is the right steps of shutdown all CS group machines and
>>>>>>>> resume
>>>>>>>> them?
>>>>>>>> How can i let my xenserver reconnected ?
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> --
>>>>>>>> -----------
>>>>>>>> Cao Tong
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>
>>>
>>
>>
>
>
Re: XenServer is disconnected after CS hosts shutdown
Posted by to...@163.com.
@prashant, following it the answer of you questions
1. Yes, primary storage is connected fine for my xenserver.
2. No, Xenserver's password is not changed.
3. yes, web UI is fine, and I can login.
4. before reboot, I unmanaged and disabled resources, and after reboot
I have enabled all of them.
5. hosts is states is UP.
6. No yum update in anywhere.
7. system VMs status is fine, i think.
-----------
Cao Tong
On 07/22/2015 04:13 PM, tony_caotong@163.com wrote:
>
> Hi,
>
> After reinstall, I got the problem again
>
> So, I will describe once again.
>
> WHAT my environment looks like:
>
> I have a ACS server host and a xenserver host, After both reboot, I
> can not create a VM on xenserver through ACS.
> A KVM and A NFS are running together in ACS manager host.
>
> the status of new VM is always 'staring' on the WEB, but I can create
> new VM using xencenter.
>
> ------------- ERR LOGS ----------
> 2015-07-22 15:56:56,357 DEBUG [c.c.s.StorageManagerImpl]
> (StatsCollector-3:ctx-1aa2e8c9) Unable to send storage pool command to
> Pool[4|NetworkFilesystem] via 4
> com.cloud.exception.OperationTimedoutException: Commands
> 2829104990918803478 to Host 4 timed out after 3600
>
> 2015-07-22 15:56:56,358 INFO [c.c.s.StatsCollector]
> (StatsCollector-3:ctx-1aa2e8c9) Unable to reach Pool[4|NetworkFilesystem]
> com.cloud.exception.StorageUnavailableException: Resource
> [StoragePool:4] is unreachable: Unable to send command to the pool
>
>
> ------------- and there are lots of DEBUG infos ------- repeat again
> and again -----------
>
> 2015-07-22 15:36:12,887 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-14:null) Seq 4-8064821032713715922: Forwarding
> Seq 4-8064821032713715922: { Cmd , MgmtId: 227448510156211, via: 4,
> Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to 116784073679673
> 2015-07-22 15:36:12,889 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-10:null) Seq 4-8064821032713715883: Forwarding
> Seq 4-8064821032713715883: { Cmd , MgmtId: 227448510156211, via: 4,
> Ver: v1, Flags: 100111,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/5/af949612-838f-3a6d-931b-312e612db740.vhd","origUrl":"http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
> 5.6(64-bit) no GUI
> (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.0.0.100/storage/secondary","_role":"Image"}},"name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
> 5.6(64-bit) no GUI
> (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2df26406-31bf-3a95-8a61-f5008defd9a0","id":4,"poolType":"NetworkFilesystem","host":"10.0.0.100","path":"/storage/xen/primary","port":2049,"url":"NetworkFilesystem://10.0.0.100/storage/xen/primary/?ROLE=Primary&STOREUUID=2df26406-31bf-3a95-8a61-f5008defd9a0"}},"name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"executeInSequence":true,"options":{},"wait":10800}}]
> } to 116784073679673
>
>
> -----------------------------------------
>
> Anyone have Any ideas? thanks.
>
> -----------
> Cao Tong
>
> On 07/21/2015 06:14 PM, tony_caotong@163.com wrote:
>>
>> Thanks all,
>>
>> I have already reinstall my hosts for preparing a new clear
>> environment to restart my research.
>>
>> -----------
>> Cao Tong
>>
>> On 07/20/2015 09:24 PM, Prashant s wrote:
>>> some questions :
>>>
>>> can you please tell ...
>>>
>>> 1. is your NFS storage or your primary Storage Repository in connected
>>> mode with no red cross mark on them in xencenter.
>>> 2. did you change any passwords on the xenservers ?
>>> 3. is the cloudstack web ui up , can you login to the cloudstack web
>>> page.
>>> 4. *are the zone , pod, or clusters in unmanaged or disabled state ? *
>>> *5. is all the hosts in connected state ? *
>>> *6. did you run yum update on host reboot on the cs manager vm ? *
>>> *7. system vms are stateless you can kill them and cs will recreate
>>> a new
>>> one .. so dont worry :-) *
>>>
>>>
>>> *thanks *
>>> *prashant *
>>>
>>>
>>>
>>> On Mon, Jul 20, 2015 at 3:47 AM, <to...@163.com> wrote:
>>>
>>>> Hi, I restartd All hosts (one mgr and xenserver) again.
>>>>
>>>>
>>>> Following is the error log.
>>>>
>>>>
>>>> 2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
>>>> (StatsCollector-3:ctx-692a5392) Could not find exception:
>>>> com.cloud.exception.OperationTimedoutException in error code list for
>>>> exceptions
>>>> 2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed
>>>> out on null
>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Cancelling.
>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
>>>> (StatsCollector-3:ctx-692a5392) Unable to send storage pool command to
>>>> Pool[1|NetworkFilesystem] via 1
>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>> 3176445112179752972 to Host 1 timed out after 3600
>>>> at
>>>> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
>>>> at
>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:433)
>>>>
>>>> at
>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
>>>>
>>>> at
>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1000)
>>>>
>>>> at
>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
>>>>
>>>> at
>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
>>>>
>>>> at
>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
>>>>
>>>> at
>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>
>>>> at
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>
>>>> at
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>
>>>> at
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>
>>>> at
>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>
>>>> at
>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
>>>> (StatsCollector-3:ctx-692a5392) Unable to reach
>>>> Pool[1|NetworkFilesystem]
>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>> [StoragePool:1]
>>>> is unreachable: Unable to send command to the pool
>>>> at
>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1010)
>>>>
>>>> at
>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
>>>>
>>>> at
>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
>>>>
>>>> at
>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
>>>>
>>>> at
>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>
>>>> at
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>
>>>> at
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>
>>>> at
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>
>>>> at
>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>
>>>> at
>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> -----------
>>>> Cao Tong
>>>>
>>>>
>>>> On 07/20/2015 02:52 PM, tony_caotong@163.com wrote:
>>>>
>>>>> No, no one's IP was changed.
>>>>>
>>>>> 1. In xenserver I can not login systemvms using the internal IP like
>>>>> '169.254.1.112', There shoud be a bridge network for this right?
>>>>> it is
>>>>> gone.
>>>>>
>>>>> 2. I try to delete xenserver host from CS on web, it also failed with
>>>>> lots of logs like following, then memory is full and mangement
>>>>> down...
>>>>>
>>>>> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>> (AgentManager-Handler-15:null) Seq 1-7282039122481381399:
>>>>> Forwarding Seq
>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>> Ver: v1,
>>>>> Flags: 100111,
>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>> 192405008094602
>>>>> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>> (AgentManager-Handler-5:null) Seq 1-7282039122481381399:
>>>>> Forwarding Seq
>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>> Ver: v1,
>>>>> Flags: 100111,
>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>> 192405008094602
>>>>> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>> (AgentManager-Handler-1:null) Seq 1-7282039122481381399:
>>>>> Forwarding Seq
>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>> Ver: v1,
>>>>> Flags: 100111,
>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>> 192405008094602
>>>>> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>> (AgentManager-Handler-14:null) Seq 1-7282039122481381399:
>>>>> Forwarding Seq
>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>> Ver: v1,
>>>>> Flags: 100111,
>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>> 192405008094602
>>>>>
>>>>>
>>>>> I guess that, is there some service or daemons working for CS is
>>>>> not up
>>>>> on Xenserver ?
>>>>>
>>>>>
>>>>> -----------
>>>>> Cao Tong
>>>>> On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
>>>>>
>>>>>> Did the management server ip change?
>>>>>> management server ip in the configuration table is used my
>>>>>> systemvms.
>>>>>> select * from configuration where name like 'host';
>>>>>>
>>>>>> If it changed, correct the value in db and restart systemvms.
>>>>>>
>>>>>>
>>>>>> ~Rajani
>>>>>>
>>>>>> On Mon, Jul 20, 2015 at 11:56 AM,<to...@163.com> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>> I shutdown my cs-manager and xenserver last weekend, And now the
>>>>>>> ssvm
>>>>>>> and cpvm is disconnect, thost two was runing on xenserver. so What
>>>>>>> should i do right now ?
>>>>>>> Please anybody help me and thanks.
>>>>>>>
>>>>>>> In xenserver I found that the three system VMs are not running.
>>>>>>> my xenserver seems can not reconnect to CS-manager. and it seams
>>>>>>> not
>>>>>>> under control of CS.
>>>>>>>
>>>>>>>
>>>>>>> What is the right steps of shutdown all CS group machines and
>>>>>>> resume
>>>>>>> them?
>>>>>>> How can i let my xenserver reconnected ?
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> --
>>>>>>> -----------
>>>>>>> Cao Tong
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>
>>
>
>
Re: XenServer is disconnected after CS hosts shutdown
Posted by to...@163.com.
Hi,
After reinstall, I got the problem again
So, I will describe once again.
WHAT my environment looks like:
I have a ACS server host and a xenserver host, After both reboot, I can
not create a VM on xenserver through ACS.
A KVM and A NFS are running together in ACS manager host.
the status of new VM is always 'staring' on the WEB, but I can create
new VM using xencenter.
------------- ERR LOGS ----------
2015-07-22 15:56:56,357 DEBUG [c.c.s.StorageManagerImpl]
(StatsCollector-3:ctx-1aa2e8c9) Unable to send storage pool command to
Pool[4|NetworkFilesystem] via 4
com.cloud.exception.OperationTimedoutException: Commands
2829104990918803478 to Host 4 timed out after 3600
2015-07-22 15:56:56,358 INFO [c.c.s.StatsCollector]
(StatsCollector-3:ctx-1aa2e8c9) Unable to reach Pool[4|NetworkFilesystem]
com.cloud.exception.StorageUnavailableException: Resource
[StoragePool:4] is unreachable: Unable to send command to the pool
------------- and there are lots of DEBUG infos ------- repeat again
and again -----------
2015-07-22 15:36:12,887 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-14:null) Seq 4-8064821032713715922: Forwarding Seq
4-8064821032713715922: { Cmd , MgmtId: 227448510156211, via: 4, Ver:
v1, Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
} to 116784073679673
2015-07-22 15:36:12,889 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-10:null) Seq 4-8064821032713715883: Forwarding Seq
4-8064821032713715883: { Cmd , MgmtId: 227448510156211, via: 4, Ver:
v1, Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/5/af949612-838f-3a6d-931b-312e612db740.vhd","origUrl":"http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
5.6(64-bit) no GUI
(XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.0.0.100/storage/secondary","_role":"Image"}},"name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
5.6(64-bit) no GUI
(XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2df26406-31bf-3a95-8a61-f5008defd9a0","id":4,"poolType":"NetworkFilesystem","host":"10.0.0.100","path":"/storage/xen/primary","port":2049,"url":"NetworkFilesystem://10.0.0.100/storage/xen/primary/?ROLE=Primary&STOREUUID=2df26406-31bf-3a95-8a61-f5008defd9a0"}},"name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"executeInSequence":true,"options":{},"wait":10800}}]
} to 116784073679673
-----------------------------------------
Anyone have Any ideas? thanks.
-----------
Cao Tong
On 07/21/2015 06:14 PM, tony_caotong@163.com wrote:
>
> Thanks all,
>
> I have already reinstall my hosts for preparing a new clear
> environment to restart my research.
>
> -----------
> Cao Tong
>
> On 07/20/2015 09:24 PM, Prashant s wrote:
>> some questions :
>>
>> can you please tell ...
>>
>> 1. is your NFS storage or your primary Storage Repository in connected
>> mode with no red cross mark on them in xencenter.
>> 2. did you change any passwords on the xenservers ?
>> 3. is the cloudstack web ui up , can you login to the cloudstack web
>> page.
>> 4. *are the zone , pod, or clusters in unmanaged or disabled state ? *
>> *5. is all the hosts in connected state ? *
>> *6. did you run yum update on host reboot on the cs manager vm ? *
>> *7. system vms are stateless you can kill them and cs will recreate a
>> new
>> one .. so dont worry :-) *
>>
>>
>> *thanks *
>> *prashant *
>>
>>
>>
>> On Mon, Jul 20, 2015 at 3:47 AM, <to...@163.com> wrote:
>>
>>> Hi, I restartd All hosts (one mgr and xenserver) again.
>>>
>>>
>>> Following is the error log.
>>>
>>>
>>> 2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
>>> (StatsCollector-3:ctx-692a5392) Could not find exception:
>>> com.cloud.exception.OperationTimedoutException in error code list for
>>> exceptions
>>> 2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed out
>>> on null
>>> 2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Cancelling.
>>> 2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
>>> (StatsCollector-3:ctx-692a5392) Unable to send storage pool command to
>>> Pool[1|NetworkFilesystem] via 1
>>> com.cloud.exception.OperationTimedoutException: Commands
>>> 3176445112179752972 to Host 1 timed out after 3600
>>> at
>>> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
>>> at
>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:433)
>>>
>>> at
>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
>>>
>>> at
>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1000)
>>>
>>> at
>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
>>>
>>> at
>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
>>>
>>> at
>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
>>>
>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>
>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at
>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>
>>> at java.lang.Thread.run(Thread.java:745)
>>> 2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
>>> (StatsCollector-3:ctx-692a5392) Unable to reach
>>> Pool[1|NetworkFilesystem]
>>> com.cloud.exception.StorageUnavailableException: Resource
>>> [StoragePool:1]
>>> is unreachable: Unable to send command to the pool
>>> at
>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1010)
>>>
>>> at
>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
>>>
>>> at
>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
>>>
>>> at
>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
>>>
>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>
>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at
>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> -----------
>>> Cao Tong
>>>
>>>
>>> On 07/20/2015 02:52 PM, tony_caotong@163.com wrote:
>>>
>>>> No, no one's IP was changed.
>>>>
>>>> 1. In xenserver I can not login systemvms using the internal IP like
>>>> '169.254.1.112', There shoud be a bridge network for this right?
>>>> it is
>>>> gone.
>>>>
>>>> 2. I try to delete xenserver host from CS on web, it also failed with
>>>> lots of logs like following, then memory is full and mangement down...
>>>>
>>>> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>> (AgentManager-Handler-15:null) Seq 1-7282039122481381399:
>>>> Forwarding Seq
>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>> Ver: v1,
>>>> Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
>>>> } to
>>>> 192405008094602
>>>> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>> (AgentManager-Handler-5:null) Seq 1-7282039122481381399: Forwarding
>>>> Seq
>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>> Ver: v1,
>>>> Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
>>>> } to
>>>> 192405008094602
>>>> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>> (AgentManager-Handler-1:null) Seq 1-7282039122481381399: Forwarding
>>>> Seq
>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>> Ver: v1,
>>>> Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
>>>> } to
>>>> 192405008094602
>>>> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>> (AgentManager-Handler-14:null) Seq 1-7282039122481381399:
>>>> Forwarding Seq
>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>> Ver: v1,
>>>> Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
>>>> } to
>>>> 192405008094602
>>>>
>>>>
>>>> I guess that, is there some service or daemons working for CS is
>>>> not up
>>>> on Xenserver ?
>>>>
>>>>
>>>> -----------
>>>> Cao Tong
>>>> On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
>>>>
>>>>> Did the management server ip change?
>>>>> management server ip in the configuration table is used my systemvms.
>>>>> select * from configuration where name like 'host';
>>>>>
>>>>> If it changed, correct the value in db and restart systemvms.
>>>>>
>>>>>
>>>>> ~Rajani
>>>>>
>>>>> On Mon, Jul 20, 2015 at 11:56 AM,<to...@163.com> wrote:
>>>>>
>>>>> Hello,
>>>>>> I shutdown my cs-manager and xenserver last weekend, And now the
>>>>>> ssvm
>>>>>> and cpvm is disconnect, thost two was runing on xenserver. so What
>>>>>> should i do right now ?
>>>>>> Please anybody help me and thanks.
>>>>>>
>>>>>> In xenserver I found that the three system VMs are not running.
>>>>>> my xenserver seems can not reconnect to CS-manager. and it seams
>>>>>> not
>>>>>> under control of CS.
>>>>>>
>>>>>>
>>>>>> What is the right steps of shutdown all CS group machines and resume
>>>>>> them?
>>>>>> How can i let my xenserver reconnected ?
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> --
>>>>>> -----------
>>>>>> Cao Tong
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>
>
Re: XenServer is disconnected after CS hosts shutdown
Posted by to...@163.com.
Thanks all,
I have already reinstall my hosts for preparing a new clear environment
to restart my research.
-----------
Cao Tong
On 07/20/2015 09:24 PM, Prashant s wrote:
> some questions :
>
> can you please tell ...
>
> 1. is your NFS storage or your primary Storage Repository in connected
> mode with no red cross mark on them in xencenter.
> 2. did you change any passwords on the xenservers ?
> 3. is the cloudstack web ui up , can you login to the cloudstack web page.
> 4. *are the zone , pod, or clusters in unmanaged or disabled state ? *
> *5. is all the hosts in connected state ? *
> *6. did you run yum update on host reboot on the cs manager vm ? *
> *7. system vms are stateless you can kill them and cs will recreate a new
> one .. so dont worry :-) *
>
>
> *thanks *
> *prashant *
>
>
>
> On Mon, Jul 20, 2015 at 3:47 AM, <to...@163.com> wrote:
>
>> Hi, I restartd All hosts (one mgr and xenserver) again.
>>
>>
>> Following is the error log.
>>
>>
>> 2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
>> (StatsCollector-3:ctx-692a5392) Could not find exception:
>> com.cloud.exception.OperationTimedoutException in error code list for
>> exceptions
>> 2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed out on null
>> 2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Cancelling.
>> 2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
>> (StatsCollector-3:ctx-692a5392) Unable to send storage pool command to
>> Pool[1|NetworkFilesystem] via 1
>> com.cloud.exception.OperationTimedoutException: Commands
>> 3176445112179752972 to Host 1 timed out after 3600
>> at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
>> at
>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:433)
>> at
>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
>> at
>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1000)
>> at
>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
>> at
>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
>> at
>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.lang.Thread.run(Thread.java:745)
>> 2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
>> (StatsCollector-3:ctx-692a5392) Unable to reach Pool[1|NetworkFilesystem]
>> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
>> is unreachable: Unable to send command to the pool
>> at
>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1010)
>> at
>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
>> at
>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
>> at
>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> -----------
>> Cao Tong
>>
>>
>> On 07/20/2015 02:52 PM, tony_caotong@163.com wrote:
>>
>>> No, no one's IP was changed.
>>>
>>> 1. In xenserver I can not login systemvms using the internal IP like
>>> '169.254.1.112', There shoud be a bridge network for this right? it is
>>> gone.
>>>
>>> 2. I try to delete xenserver host from CS on web, it also failed with
>>> lots of logs like following, then memory is full and mangement down...
>>>
>>> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>> (AgentManager-Handler-15:null) Seq 1-7282039122481381399: Forwarding Seq
>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1, Ver: v1,
>>> Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>> 192405008094602
>>> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>> (AgentManager-Handler-5:null) Seq 1-7282039122481381399: Forwarding Seq
>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1, Ver: v1,
>>> Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>> 192405008094602
>>> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>> (AgentManager-Handler-1:null) Seq 1-7282039122481381399: Forwarding Seq
>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1, Ver: v1,
>>> Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>> 192405008094602
>>> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>> (AgentManager-Handler-14:null) Seq 1-7282039122481381399: Forwarding Seq
>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1, Ver: v1,
>>> Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>> 192405008094602
>>>
>>>
>>> I guess that, is there some service or daemons working for CS is not up
>>> on Xenserver ?
>>>
>>>
>>> -----------
>>> Cao Tong
>>> On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
>>>
>>>> Did the management server ip change?
>>>> management server ip in the configuration table is used my systemvms.
>>>> select * from configuration where name like 'host';
>>>>
>>>> If it changed, correct the value in db and restart systemvms.
>>>>
>>>>
>>>> ~Rajani
>>>>
>>>> On Mon, Jul 20, 2015 at 11:56 AM,<to...@163.com> wrote:
>>>>
>>>> Hello,
>>>>> I shutdown my cs-manager and xenserver last weekend, And now the ssvm
>>>>> and cpvm is disconnect, thost two was runing on xenserver. so What
>>>>> should i do right now ?
>>>>> Please anybody help me and thanks.
>>>>>
>>>>> In xenserver I found that the three system VMs are not running.
>>>>> my xenserver seems can not reconnect to CS-manager. and it seams not
>>>>> under control of CS.
>>>>>
>>>>>
>>>>> What is the right steps of shutdown all CS group machines and resume
>>>>> them?
>>>>> How can i let my xenserver reconnected ?
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> --
>>>>> -----------
>>>>> Cao Tong
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
Re: XenServer is disconnected after CS hosts shutdown
Posted by Prashant s <op...@gmail.com>.
some questions :
can you please tell ...
1. is your NFS storage or your primary Storage Repository in connected
mode with no red cross mark on them in xencenter.
2. did you change any passwords on the xenservers ?
3. is the cloudstack web ui up , can you login to the cloudstack web page.
4. *are the zone , pod, or clusters in unmanaged or disabled state ? *
*5. is all the hosts in connected state ? *
*6. did you run yum update on host reboot on the cs manager vm ? *
*7. system vms are stateless you can kill them and cs will recreate a new
one .. so dont worry :-) *
*thanks *
*prashant *
On Mon, Jul 20, 2015 at 3:47 AM, <to...@163.com> wrote:
>
> Hi, I restartd All hosts (one mgr and xenserver) again.
>
>
> Following is the error log.
>
>
> 2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
> (StatsCollector-3:ctx-692a5392) Could not find exception:
> com.cloud.exception.OperationTimedoutException in error code list for
> exceptions
> 2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed out on null
> 2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Cancelling.
> 2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
> (StatsCollector-3:ctx-692a5392) Unable to send storage pool command to
> Pool[1|NetworkFilesystem] via 1
> com.cloud.exception.OperationTimedoutException: Commands
> 3176445112179752972 to Host 1 timed out after 3600
> at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
> at
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:433)
> at
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1000)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
> at
> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.lang.Thread.run(Thread.java:745)
> 2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
> (StatsCollector-3:ctx-692a5392) Unable to reach Pool[1|NetworkFilesystem]
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
> is unreachable: Unable to send command to the pool
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1010)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
> at
> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> -----------
> Cao Tong
>
>
> On 07/20/2015 02:52 PM, tony_caotong@163.com wrote:
>
>>
>> No, no one's IP was changed.
>>
>> 1. In xenserver I can not login systemvms using the internal IP like
>> '169.254.1.112', There shoud be a bridge network for this right? it is
>> gone.
>>
>> 2. I try to delete xenserver host from CS on web, it also failed with
>> lots of logs like following, then memory is full and mangement down...
>>
>> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
>> (AgentManager-Handler-15:null) Seq 1-7282039122481381399: Forwarding Seq
>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1, Ver: v1,
>> Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>> 192405008094602
>> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
>> (AgentManager-Handler-5:null) Seq 1-7282039122481381399: Forwarding Seq
>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1, Ver: v1,
>> Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>> 192405008094602
>> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
>> (AgentManager-Handler-1:null) Seq 1-7282039122481381399: Forwarding Seq
>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1, Ver: v1,
>> Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>> 192405008094602
>> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
>> (AgentManager-Handler-14:null) Seq 1-7282039122481381399: Forwarding Seq
>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1, Ver: v1,
>> Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>> 192405008094602
>>
>>
>> I guess that, is there some service or daemons working for CS is not up
>> on Xenserver ?
>>
>>
>> -----------
>> Cao Tong
>> On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
>>
>>> Did the management server ip change?
>>> management server ip in the configuration table is used my systemvms.
>>> select * from configuration where name like 'host';
>>>
>>> If it changed, correct the value in db and restart systemvms.
>>>
>>>
>>> ~Rajani
>>>
>>> On Mon, Jul 20, 2015 at 11:56 AM,<to...@163.com> wrote:
>>>
>>> Hello,
>>>>
>>>> I shutdown my cs-manager and xenserver last weekend, And now the ssvm
>>>> and cpvm is disconnect, thost two was runing on xenserver. so What
>>>> should i do right now ?
>>>> Please anybody help me and thanks.
>>>>
>>>> In xenserver I found that the three system VMs are not running.
>>>> my xenserver seems can not reconnect to CS-manager. and it seams not
>>>> under control of CS.
>>>>
>>>>
>>>> What is the right steps of shutdown all CS group machines and resume
>>>> them?
>>>> How can i let my xenserver reconnected ?
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> --
>>>> -----------
>>>> Cao Tong
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>
Re: XenServer is disconnected after CS hosts shutdown
Posted by to...@163.com.
Hi, I restartd All hosts (one mgr and xenserver) again.
Following is the error log.
2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
(StatsCollector-3:ctx-692a5392) Could not find exception:
com.cloud.exception.OperationTimedoutException in error code list for
exceptions
2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
(StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed out on null
2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
(StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Cancelling.
2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
(StatsCollector-3:ctx-692a5392) Unable to send storage pool command to
Pool[1|NetworkFilesystem] via 1
com.cloud.exception.OperationTimedoutException: Commands
3176445112179752972 to Host 1 timed out after 3600
at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:433)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1000)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
at
com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.lang.Thread.run(Thread.java:745)
2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
(StatsCollector-3:ctx-692a5392) Unable to reach Pool[1|NetworkFilesystem]
com.cloud.exception.StorageUnavailableException: Resource
[StoragePool:1] is unreachable: Unable to send command to the pool
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1010)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
at
com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
-----------
Cao Tong
On 07/20/2015 02:52 PM, tony_caotong@163.com wrote:
>
> No, no one's IP was changed.
>
> 1. In xenserver I can not login systemvms using the internal IP like
> '169.254.1.112', There shoud be a bridge network for this right? it
> is gone.
>
> 2. I try to delete xenserver host from CS on web, it also failed with
> lots of logs like following, then memory is full and mangement down...
>
> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-15:null) Seq 1-7282039122481381399: Forwarding
> Seq 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
> Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to 192405008094602
> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-5:null) Seq 1-7282039122481381399: Forwarding
> Seq 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
> Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to 192405008094602
> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-1:null) Seq 1-7282039122481381399: Forwarding
> Seq 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
> Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to 192405008094602
> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-14:null) Seq 1-7282039122481381399: Forwarding
> Seq 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
> Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to 192405008094602
>
>
> I guess that, is there some service or daemons working for CS is not
> up on Xenserver ?
>
>
> -----------
> Cao Tong
> On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
>> Did the management server ip change?
>> management server ip in the configuration table is used my systemvms.
>> select * from configuration where name like 'host';
>>
>> If it changed, correct the value in db and restart systemvms.
>>
>>
>> ~Rajani
>>
>> On Mon, Jul 20, 2015 at 11:56 AM,<to...@163.com> wrote:
>>
>>> Hello,
>>>
>>> I shutdown my cs-manager and xenserver last weekend, And now the ssvm
>>> and cpvm is disconnect, thost two was runing on xenserver. so What
>>> should i do right now ?
>>> Please anybody help me and thanks.
>>>
>>> In xenserver I found that the three system VMs are not running.
>>> my xenserver seems can not reconnect to CS-manager. and it seams not
>>> under control of CS.
>>>
>>>
>>> What is the right steps of shutdown all CS group machines and resume them?
>>> How can i let my xenserver reconnected ?
>>>
>>>
>>> Thanks,
>>>
>>> --
>>> -----------
>>> Cao Tong
>>>
>>>
>>>
>>>
>>>
>
Re: XenServer is disconnected after CS hosts shutdown
Posted by to...@163.com.
No, no one's IP was changed.
1. In xenserver I can not login systemvms using the internal IP like
'169.254.1.112', There shoud be a bridge network for this right? it is
gone.
2. I try to delete xenserver host from CS on web, it also failed with
lots of logs like following, then memory is full and mangement down...
2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-15:null) Seq 1-7282039122481381399: Forwarding Seq
1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1, Ver:
v1, Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
} to 192405008094602
2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-5:null) Seq 1-7282039122481381399: Forwarding Seq
1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1, Ver:
v1, Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
} to 192405008094602
2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-1:null) Seq 1-7282039122481381399: Forwarding Seq
1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1, Ver:
v1, Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
} to 192405008094602
2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-14:null) Seq 1-7282039122481381399: Forwarding Seq
1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1, Ver:
v1, Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
} to 192405008094602
I guess that, is there some service or daemons working for CS is not up
on Xenserver ?
-----------
Cao Tong
On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
> Did the management server ip change?
> management server ip in the configuration table is used my systemvms.
> select * from configuration where name like 'host';
>
> If it changed, correct the value in db and restart systemvms.
>
>
> ~Rajani
>
> On Mon, Jul 20, 2015 at 11:56 AM, <to...@163.com> wrote:
>
>> Hello,
>>
>> I shutdown my cs-manager and xenserver last weekend, And now the ssvm
>> and cpvm is disconnect, thost two was runing on xenserver. so What
>> should i do right now ?
>> Please anybody help me and thanks.
>>
>> In xenserver I found that the three system VMs are not running.
>> my xenserver seems can not reconnect to CS-manager. and it seams not
>> under control of CS.
>>
>>
>> What is the right steps of shutdown all CS group machines and resume them?
>> How can i let my xenserver reconnected ?
>>
>>
>> Thanks,
>>
>> --
>> -----------
>> Cao Tong
>>
>>
>>
>>
>>
Re: XenServer is disconnected after CS hosts shutdown
Posted by Rajani Karuturi <ra...@apache.org>.
Did the management server ip change?
management server ip in the configuration table is used my systemvms.
select * from configuration where name like 'host';
If it changed, correct the value in db and restart systemvms.
~Rajani
On Mon, Jul 20, 2015 at 11:56 AM, <to...@163.com> wrote:
>
> Hello,
>
> I shutdown my cs-manager and xenserver last weekend, And now the ssvm
> and cpvm is disconnect, thost two was runing on xenserver. so What
> should i do right now ?
> Please anybody help me and thanks.
>
> In xenserver I found that the three system VMs are not running.
> my xenserver seems can not reconnect to CS-manager. and it seams not
> under control of CS.
>
>
> What is the right steps of shutdown all CS group machines and resume them?
> How can i let my xenserver reconnected ?
>
>
> Thanks,
>
> --
> -----------
> Cao Tong
>
>
>
>
>