You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by to...@163.com on 2015/08/12 10:41:20 UTC
Re: XenServer is disconnected after CS hosts shutdown
Hello,
Almost one month gone, my problem is still here. I really really need
someone to help me.
new Settings ACS 4.4.4 XS 6.2
After reboot, errors here:
2015-08-12 16:18:54,557 INFO [c.c.a.t.Request]
(StatsCollector-3:ctx-b287085a) not building log message for '[{}]',
_cmds.length == 1
2015-08-12 16:18:54,557 DEBUG [c.c.a.m.ClusteredAgentAttache]
(StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Forwarding
null to 191386435611186
2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-12:null) Seq 9-4450963806725603425: Routing from
249082151178140
2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-12:null) Seq 9-4450963806725603425: Link is closed
2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-12:null) Seq 9-4450963806725603425: MgmtId
249082151178140: Req: Resource [Host:9] is unreachable: Host 9: Link is
closed
2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-12:null) Seq 9--1: MgmtId 249082151178140: Req:
Routing to peer
2015-08-12 16:18:54,559 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentManager-Handler-15:null) Seq 9--1: MgmtId 249082151178140: Req:
Cancel request received
2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
(AgentManager-Handler-15:null) Seq 9-4450963806725603425: Cancelling.
2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
(StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Waiting some
more time because this is the current command
2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
(StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Waiting some
more time because this is the current command
2015-08-12 16:18:54,559 INFO [c.c.u.e.CSExceptionErrorCode]
(StatsCollector-3:ctx-b287085a) Could not find exception:
com.cloud.exception.OperationTimedoutException in error code list for
exceptions
2015-08-12 16:18:54,559 INFO [c.c.a.t.Request]
(StatsCollector-3:ctx-b287085a) not building log message for '[{}]',
_cmds.length == 1
2015-08-12 16:18:54,559 WARN [c.c.a.m.AgentAttache]
(StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Timed out on null
2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
(StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Cancelling.
2015-08-12 16:18:54,559 DEBUG [c.c.s.StorageManagerImpl]
(StatsCollector-3:ctx-b287085a) Unable to send storage pool command to
Pool[3|NetworkFilesystem] via 9
com.cloud.exception.OperationTimedoutException: Commands
4450963806725603425 to Host 9 timed out after 3600
at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:434)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:418)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:965)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:390)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:404)
at
com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
-----------
Cao Tong
On 07/30/2015 10:55 AM, tony_caotong@163.com wrote:
>
> Hi Yiping, Thanks for your reply first.
>
>
> My NFS server deployed in ACS manager host, another host is a single
> xenserver.
> A KVM environment deployed in ACS manager host too.
> there is three storages named, Kprimary, Xprimary, Secondary.
>
> First, I add KVM cluster with zone-wide Kprimary, it works fine even
> if after reboot.
> Second, I add Xenserver with Xprimary(cluster-wide), it still works
> fine.
> Then, I set zone to disable, shutdown all system VMs, disable those
> two hosts. finally shutdown xenserver host.
> When xenserver's shutdown finish, I stop those services by order (
> cloudstack-agent, cloudstack-management, libvirtd, nfs, rpcbind,
> mysqld).
> At last shutdown ACS host.
>
> The start process is totally reverse of this order.
>
> when done of starting, ACS says unable to send cmd to pool via host
> xenserver.
>
> I am sure nothing have change in my environment during reboot time.
> When I fix this problem, the only change is change Kprimary from
> zone-wide to cluster-wide.
>
> I guess that after reboot some status have been initial from
> beginning, ACS found that Xenserver host have two primary can be used.
> so it use the high priority one， and it is Kprimary.
>
> Whatever, Maybe it could help you peoples to get ACS better.
>
> BTW, Error logs attached some days before is already point out this is
> a storage problem
> like:
>
> Unable to send storage pool command
> to Pool[4|NetworkFilesystem] via 4
>
>
>
> -----------
> Cao Tong
>
> On 07/30/2015 12:44 AM, Yiping Zhang wrote:
>> Well, sometimes people can’t answer a question because of lack of
>> relevant information, or simply because no one has encountered a similar
>> situation before.
>>
>> Looking at your past messages on this thread, there were no mentions
>> about
>> primary storage. Obviously, your primary storage configuration had
>> changed
>> between the time you shut down CS manager and xenservers and the time
>> you
>> restarted them. That is the vital info the list didn’t know.
>>
>> To best of my knowledge, zone wide primary storage has never been
>> supported for Xen hypervisors.
>>
>> I do have to say that quite often CloudStack error messages are very
>> cryptic, do not provide enough *useful* information to help users
>> identify
>> and trouble shoot actual problems. Those stack trace output might be a
>> gold mine to developers, but they are utterly useless for end users.
>>
>> Just my $0.02
>>
>> Yiping
>>
>> On 7/28/15, 11:19 PM, "tony_caotong@163.com" <to...@163.com>
>> wrote:
>>
>>> Hi, Finally I resolved this problem by my self.
>>>
>>> * Primary Storage: A storage resource typically provided to a single
>>> cluster for the actual running of instance disk images. (Zone-wide
>>> primary storage is an option, though not typically used.)
>>>
>>> This line above is from
>>> http://docs.cloudstack.apache.org/en/master/concepts.html
>>>
>>> Because I have a Zone-wide primary storage, ACS can not find the
>>> correct
>>> primary which belong to XenServer cluster after reboot.
>>>
>>> Then I change the Zone-wide primary to cluster-wide, it resolved.
>>>
>>> Right now, I have two primary storage, one is kvm cluster-wide, another
>>> is xenserver cluster-wide.
>>>
>>> Above is for people who have the same problem oneday.
>>>
>>> by the way, I am very curious why I never receive replys from this a
>>> big
>>> community ?? of course except the very beginning.
>>>
>>> Is my English skill really really poor, result in no body can
>>> understood
>>> what language I am speaking ?
>>>
>>> -----------
>>> Cao Tong
>>>
>>> On 07/22/2015 09:03 PM, tony_caotong@163.com wrote:
>>>> Hey! help please...
>>>>
>>>> some news.
>>>> I think the cause is that the ACS host can't communicate with
>>>> XenServer host.
>>>> ACS continues outputing logs like this
>>>>
>>>> 2015-07-22 20:42:13,555 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>> (AgentManager-Handler-7:null) Seq 5-8174877748607582212: Forwarding
>>>> Seq 5-8174877748607582212: { Cmd , MgmtId: 279278805451459, via: 5,
>>>> Ver: v1, Flags: 100111,
>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>> 280345368052992
>>>>
>>>> I am not sure that if the ACS status is wrong or some services on
>>>> xenserver are not opend.
>>>>
>>>> on xenserver , I found *xenheartbeat.sh is not running.*
>>>> *(/bin/bash /opt/cloud/bin/xenheartbeat.sh
>>>> 00d8e0d0-8561-4b3d-9044-cbc496ff22cc 120 60)*
>>>>
>>>> As some operations about xenserver was pending， xenserver can not be
>>>> deleted from web UI.
>>>>
>>>> I got a temporary solution
>>>>
>>>> 1. delete jobs from DB cloud.vm_work_job.
>>>> 2. delete xenserver from DB cloud.host.
>>>> 3. add xenserver host back from web UI.
>>>>
>>>> then it works.
>>>>
>>>> Does anyone have a idea for this?
>>>>
>>>> Could anyone tell what things does ACS do on xenserver host when
>>>> adding a xenserver ?
>>>>
>>>> Thanks,
>>>>
>>>> -----------
>>>> Cao Tong
>>>>
>>>> On 07/22/2015 04:26 PM, tony_caotong@163.com wrote:
>>>>> @prashant, following it the answer of you questions
>>>>>
>>>>> 1. Yes, primary storage is connected fine for my xenserver.
>>>>>
>>>>> 2. No, Xenserver's password is not changed.
>>>>>
>>>>> 3. yes, web UI is fine, and I can login.
>>>>>
>>>>> 4. before reboot, I unmanaged and disabled resources, and after
>>>>> reboot I have enabled all of them.
>>>>>
>>>>> 5. hosts is states is UP.
>>>>>
>>>>> 6. No yum update in anywhere.
>>>>>
>>>>> 7. system VMs status is fine, i think.
>>>>>
>>>>> -----------
>>>>> Cao Tong
>>>>>
>>>>> On 07/22/2015 04:13 PM, tony_caotong@163.com wrote:
>>>>>> Hi,
>>>>>>
>>>>>> After reinstall, I got the problem again
>>>>>>
>>>>>> So, I will describe once again.
>>>>>>
>>>>>> WHAT my environment looks like:
>>>>>>
>>>>>> I have a ACS server host and a xenserver host, After both reboot, I
>>>>>> can not create a VM on xenserver through ACS.
>>>>>> A KVM and A NFS are running together in ACS manager host.
>>>>>>
>>>>>> the status of new VM is always 'staring' on the WEB, but I can
>>>>>> create new VM using xencenter.
>>>>>>
>>>>>> ------------- ERR LOGS ----------
>>>>>> 2015-07-22 15:56:56,357 DEBUG [c.c.s.StorageManagerImpl]
>>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to send storage pool command
>>>>>> to Pool[4|NetworkFilesystem] via 4
>>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>>> 2829104990918803478 to Host 4 timed out after 3600
>>>>>>
>>>>>> 2015-07-22 15:56:56,358 INFO [c.c.s.StatsCollector]
>>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to reach
>>>>>> Pool[4|NetworkFilesystem]
>>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>>> [StoragePool:4] is unreachable: Unable to send command to the pool
>>>>>>
>>>>>>
>>>>>> ------------- and there are lots of DEBUG infos ------- repeat
>>>>>> again and again -----------
>>>>>>
>>>>>> 2015-07-22 15:36:12,887 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>> (AgentManager-Handler-14:null) Seq 4-8064821032713715922: Forwarding
>>>>>> Seq 4-8064821032713715922: { Cmd , MgmtId: 227448510156211, via: 4,
>>>>>> Ver: v1, Flags: 100111,
>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>> 116784073679673
>>>>>> 2015-07-22 15:36:12,889 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>> (AgentManager-Handler-10:null) Seq 4-8064821032713715883: Forwarding
>>>>>> Seq 4-8064821032713715883: { Cmd , MgmtId: 227448510156211, via: 4,
>>>>>> Ver: v1, Flags: 100111,
>>>>>>
>>>>>> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap
>>>>>>
>>>>>> ache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/5
>>>>>>
>>>>>> /af949612-838f-3a6d-931b-312e612db740.vhd","origUrl":"http://download.c
>>>>>>
>>>>>> loud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-30
>>>>>>
>>>>>> 17-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksu
>>>>>>
>>>>>> m":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>>>>>
>>>>>>
>>>>>> 5.6(64-bit) no GUI
>>>>>>
>>>>>> (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"
>>>>>>
>>>>>> nfs://10.0.0.100/storage/secondary","_role":"Image"}},"name":"centos56-
>>>>>>
>>>>>> x86_64-xen","hypervisorType":"XenServer"}},"destTO":{"org.apache.clouds
>>>>>>
>>>>>> tack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com
>>>>>>
>>>>>> /templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-
>>>>>>
>>>>>> 8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905c
>>>>>>
>>>>>> ec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>>>>> 5.6(64-bit) no GUI
>>>>>>
>>>>>> (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.Primar
>>>>>>
>>>>>> yDataStoreTO":{"uuid":"2df26406-31bf-3a95-8a61-f5008defd9a0","id":4,"po
>>>>>>
>>>>>> olType":"NetworkFilesystem","host":"10.0.0.100","path":"/storage/xen/pr
>>>>>>
>>>>>> imary","port":2049,"url":"NetworkFilesystem://10.0.0.100/storage/xen/pr
>>>>>>
>>>>>> imary/?ROLE=Primary&STOREUUID=2df26406-31bf-3a95-8a61-f5008defd9a0"}},"
>>>>>>
>>>>>> name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"executeInSe
>>>>>>
>>>>>> quence":true,"options":{},"wait":10800}}]
>>>>>> } to 116784073679673
>>>>>>
>>>>>>
>>>>>> -----------------------------------------
>>>>>>
>>>>>> Anyone have Any ideas? thanks.
>>>>>>
>>>>>> -----------
>>>>>> Cao Tong
>>>>>>
>>>>>> On 07/21/2015 06:14 PM, tony_caotong@163.com wrote:
>>>>>>> Thanks all,
>>>>>>>
>>>>>>> I have already reinstall my hosts for preparing a new clear
>>>>>>> environment to restart my research.
>>>>>>>
>>>>>>> -----------
>>>>>>> Cao Tong
>>>>>>>
>>>>>>> On 07/20/2015 09:24 PM, Prashant s wrote:
>>>>>>>> some questions :
>>>>>>>>
>>>>>>>> can you please tell ...
>>>>>>>>
>>>>>>>> 1. is your NFS storage or your primary Storage Repository in
>>>>>>>> connected
>>>>>>>> mode with no red cross mark on them in xencenter.
>>>>>>>> 2. did you change any passwords on the xenservers ?
>>>>>>>> 3. is the cloudstack web ui up , can you login to the cloudstack
>>>>>>>> web page.
>>>>>>>> 4. *are the zone , pod, or clusters in unmanaged or disabled state
>>>>>>>> ? *
>>>>>>>> *5. is all the hosts in connected state ? *
>>>>>>>> *6. did you run yum update on host reboot on the cs manager vm
>>>>>>>> ? *
>>>>>>>> *7. system vms are stateless you can kill them and cs will
>>>>>>>> recreate a new
>>>>>>>> one .. so dont worry :-) *
>>>>>>>>
>>>>>>>>
>>>>>>>> *thanks *
>>>>>>>> *prashant *
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jul 20, 2015 at 3:47 AM, <to...@163.com> wrote:
>>>>>>>>
>>>>>>>>> Hi, I restartd All hosts (one mgr and xenserver) again.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Following is the error log.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
>>>>>>>>> (StatsCollector-3:ctx-692a5392) Could not find exception:
>>>>>>>>> com.cloud.exception.OperationTimedoutException in error code list
>>>>>>>>> for
>>>>>>>>> exceptions
>>>>>>>>> 2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
>>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed
>>>>>>>>> out on null
>>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
>>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972:
>>>>>>>>> Cancelling.
>>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
>>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to send storage pool
>>>>>>>>> command to
>>>>>>>>> Pool[1|NetworkFilesystem] via 1
>>>>>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>>>>>> 3176445112179752972 to Host 1 timed out after 3600
>>>>>>>>> at
>>>>>>>>> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:
>>>>>>>>>
>>>>>>>>> 433)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:
>>>>>>>>>
>>>>>>>>> 362)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>
>>>>>>>>> ava:1000)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>
>>>>>>>>> ava:392)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>
>>>>>>>>> ava:406)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC
>>>>>>>>>
>>>>>>>>> ollector.java:642)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
>>>>>>>>>
>>>>>>>>> anagedContextRunnable.java:49)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
>>>>>>>>>
>>>>>>>>> all(DefaultManagedContext.java:56)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
>>>>>>>>>
>>>>>>>>> lWithContext(DefaultManagedContext.java:103)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
>>>>>>>>>
>>>>>>>>> WithContext(DefaultManagedContext.java:53)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
>>>>>>>>>
>>>>>>>>> agedContextRunnable.java:46)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>>>>>
>>>>>>>>> 71)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>
>>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>
>>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>>>>>>>>
>>>>>>>>> .java:1145)
>>>>>>>>>
>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>> 2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
>>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to reach
>>>>>>>>> Pool[1|NetworkFilesystem]
>>>>>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>>>>>> [StoragePool:1]
>>>>>>>>> is unreachable: Unable to send command to the pool
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>
>>>>>>>>> ava:1010)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>
>>>>>>>>> ava:392)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>
>>>>>>>>> ava:406)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC
>>>>>>>>>
>>>>>>>>> ollector.java:642)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
>>>>>>>>>
>>>>>>>>> anagedContextRunnable.java:49)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
>>>>>>>>>
>>>>>>>>> all(DefaultManagedContext.java:56)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
>>>>>>>>>
>>>>>>>>> lWithContext(DefaultManagedContext.java:103)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
>>>>>>>>>
>>>>>>>>> WithContext(DefaultManagedContext.java:53)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
>>>>>>>>>
>>>>>>>>> agedContextRunnable.java:46)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>>>>>
>>>>>>>>> 71)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>
>>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>
>>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>>>>>>>>
>>>>>>>>> .java:1145)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto
>>>>>>>>>
>>>>>>>>> r.java:615)
>>>>>>>>>
>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>
>>>>>>>>> -----------
>>>>>>>>> Cao Tong
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 07/20/2015 02:52 PM, tony_caotong@163.com wrote:
>>>>>>>>>
>>>>>>>>>> No, no one's IP was changed.
>>>>>>>>>>
>>>>>>>>>> 1. In xenserver I can not login systemvms using the internal IP
>>>>>>>>>> like
>>>>>>>>>> '169.254.1.112', There shoud be a bridge network for this
>>>>>>>>>> right? it is
>>>>>>>>>> gone.
>>>>>>>>>>
>>>>>>>>>> 2. I try to delete xenserver host from CS on web, it also failed
>>>>>>>>>> with
>>>>>>>>>> lots of logs like following, then memory is full and mangement
>>>>>>>>>> down...
>>>>>>>>>>
>>>>>>>>>> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>> (AgentManager-Handler-15:null) Seq 1-7282039122481381399:
>>>>>>>>>> Forwarding Seq
>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>>>>> Ver: v1,
>>>>>>>>>> Flags: 100111,
>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>> 192405008094602
>>>>>>>>>> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>> (AgentManager-Handler-5:null) Seq 1-7282039122481381399:
>>>>>>>>>> Forwarding Seq
>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>>>>> Ver: v1,
>>>>>>>>>> Flags: 100111,
>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>> 192405008094602
>>>>>>>>>> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>> (AgentManager-Handler-1:null) Seq 1-7282039122481381399:
>>>>>>>>>> Forwarding Seq
>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>>>>> Ver: v1,
>>>>>>>>>> Flags: 100111,
>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>> 192405008094602
>>>>>>>>>> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>> (AgentManager-Handler-14:null) Seq 1-7282039122481381399:
>>>>>>>>>> Forwarding Seq
>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
>>>>>>>>>> Ver: v1,
>>>>>>>>>> Flags: 100111,
>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>> 192405008094602
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I guess that, is there some service or daemons working for CS
>>>>>>>>>> is not up
>>>>>>>>>> on Xenserver ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -----------
>>>>>>>>>> Cao Tong
>>>>>>>>>> On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
>>>>>>>>>>
>>>>>>>>>>> Did the management server ip change?
>>>>>>>>>>> management server ip in the configuration table is used my
>>>>>>>>>>> systemvms.
>>>>>>>>>>> select * from configuration where name like 'host';
>>>>>>>>>>>
>>>>>>>>>>> If it changed, correct the value in db and restart systemvms.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ~Rajani
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jul 20, 2015 at 11:56 AM,<to...@163.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>> I shutdown my cs-manager and xenserver last weekend, And now
>>>>>>>>>>>> the ssvm
>>>>>>>>>>>> and cpvm is disconnect, thost two was runing on xenserver. so
>>>>>>>>>>>> What
>>>>>>>>>>>> should i do right now ?
>>>>>>>>>>>> Please anybody help me and thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> In xenserver I found that the three system VMs are not
>>>>>>>>>>>> running.
>>>>>>>>>>>> my xenserver seems can not reconnect to CS-manager. and it
>>>>>>>>>>>> seams not
>>>>>>>>>>>> under control of CS.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> What is the right steps of shutdown all CS group machines and
>>>>>>>>>>>> resume
>>>>>>>>>>>> them?
>>>>>>>>>>>> How can i let my xenserver reconnected ?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> -----------
>>>>>>>>>>>> Cao Tong
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>
>
RE: XenServer is disconnected after CS hosts shutdown
Posted by Somesh Naidu <So...@citrix.com>.
> starting XS agent manually
Either unmanage/manage cluster or perform a force reconnect to the host.
Regards,
Somesh
-----Original Message-----
From: tony_caotong@163.com [mailto:tony_caotong@163.com]
Sent: Wednesday, August 12, 2015 5:14 AM
To: users@cloudstack.apache.org
Subject: Re: XenServer is disconnected after CS hosts shutdown
After I disabled and enabled XS cluster & primary. it works.
seems XS agent have started by ACS host. so what is the correct
behavior of starting XS agent manually ?
-----------
Cao Tong
On 08/12/2015 04:41 PM, tony_caotong@163.com wrote:
>
> Hello,
>
> Almost one month gone, my problem is still here. I really really need
> someone to help me.
>
> new Settings ACS 4.4.4 XS 6.2
> After reboot, errors here:
>
> 2015-08-12 16:18:54,557 INFO [c.c.a.t.Request]
> (StatsCollector-3:ctx-b287085a) not building log message for '[{}]',
> _cmds.length == 1
> 2015-08-12 16:18:54,557 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Forwarding
> null to 191386435611186
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-12:null) Seq 9-4450963806725603425: Routing from
> 249082151178140
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-12:null) Seq 9-4450963806725603425: Link is closed
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-12:null) Seq 9-4450963806725603425: MgmtId
> 249082151178140: Req: Resource [Host:9] is unreachable: Host 9: Link
> is closed
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-12:null) Seq 9--1: MgmtId 249082151178140: Req:
> Routing to peer
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-15:null) Seq 9--1: MgmtId 249082151178140: Req:
> Cancel request received
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
> (AgentManager-Handler-15:null) Seq 9-4450963806725603425: Cancelling.
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Waiting
> some more time because this is the current command
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Waiting
> some more time because this is the current command
> 2015-08-12 16:18:54,559 INFO [c.c.u.e.CSExceptionErrorCode]
> (StatsCollector-3:ctx-b287085a) Could not find exception:
> com.cloud.exception.OperationTimedoutException in error code list for
> exceptions
> 2015-08-12 16:18:54,559 INFO [c.c.a.t.Request]
> (StatsCollector-3:ctx-b287085a) not building log message for '[{}]',
> _cmds.length == 1
> 2015-08-12 16:18:54,559 WARN [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Timed out
> on null
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Cancelling.
> 2015-08-12 16:18:54,559 DEBUG [c.c.s.StorageManagerImpl]
> (StatsCollector-3:ctx-b287085a) Unable to send storage pool command to
> Pool[3|NetworkFilesystem] via 9
> com.cloud.exception.OperationTimedoutException: Commands
> 4450963806725603425 to Host 9 timed out after 3600
> at
> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:434)
> at
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:418)
> at
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:965)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:390)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:404)
> at
> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
>
>
>
> -----------
> Cao Tong
>
> On 07/30/2015 10:55 AM, tony_caotong@163.com wrote:
>>
>> Hi Yiping, Thanks for your reply first.
>>
>>
>> My NFS server deployed in ACS manager host, another host is a single
>> xenserver.
>> A KVM environment deployed in ACS manager host too.
>> there is three storages named, Kprimary, Xprimary, Secondary.
>>
>> First, I add KVM cluster with zone-wide Kprimary, it works fine even
>> if after reboot.
>> Second, I add Xenserver with Xprimary(cluster-wide), it still works
>> fine.
>> Then, I set zone to disable, shutdown all system VMs, disable
>> those two hosts. finally shutdown xenserver host.
>> When xenserver's shutdown finish, I stop those services by order (
>> cloudstack-agent, cloudstack-management, libvirtd, nfs, rpcbind,
>> mysqld).
>> At last shutdown ACS host.
>>
>> The start process is totally reverse of this order.
>>
>> when done of starting, ACS says unable to send cmd to pool via host
>> xenserver.
>>
>> I am sure nothing have change in my environment during reboot time.
>> When I fix this problem, the only change is change Kprimary from
>> zone-wide to cluster-wide.
>>
>> I guess that after reboot some status have been initial from
>> beginning, ACS found that Xenserver host have two primary can be used.
>> so it use the high priority one， and it is Kprimary.
>>
>> Whatever, Maybe it could help you peoples to get ACS better.
>>
>> BTW, Error logs attached some days before is already point out this
>> is a storage problem
>> like:
>>
>> Unable to send storage pool command
>> to Pool[4|NetworkFilesystem] via 4
>>
>>
>>
>> -----------
>> Cao Tong
>>
>> On 07/30/2015 12:44 AM, Yiping Zhang wrote:
>>> Well, sometimes people can’t answer a question because of lack of
>>> relevant information, or simply because no one has encountered a
>>> similar
>>> situation before.
>>>
>>> Looking at your past messages on this thread, there were no mentions
>>> about
>>> primary storage. Obviously, your primary storage configuration had
>>> changed
>>> between the time you shut down CS manager and xenservers and the
>>> time you
>>> restarted them. That is the vital info the list didn’t know.
>>>
>>> To best of my knowledge, zone wide primary storage has never been
>>> supported for Xen hypervisors.
>>>
>>> I do have to say that quite often CloudStack error messages are very
>>> cryptic, do not provide enough *useful* information to help users
>>> identify
>>> and trouble shoot actual problems. Those stack trace output might be a
>>> gold mine to developers, but they are utterly useless for end users.
>>>
>>> Just my $0.02
>>>
>>> Yiping
>>>
>>> On 7/28/15, 11:19 PM, "tony_caotong@163.com" <to...@163.com>
>>> wrote:
>>>
>>>> Hi, Finally I resolved this problem by my self.
>>>>
>>>> * Primary Storage: A storage resource typically provided to a single
>>>> cluster for the actual running of instance disk images. (Zone-wide
>>>> primary storage is an option, though not typically used.)
>>>>
>>>> This line above is from
>>>> http://docs.cloudstack.apache.org/en/master/concepts.html
>>>>
>>>> Because I have a Zone-wide primary storage, ACS can not find the
>>>> correct
>>>> primary which belong to XenServer cluster after reboot.
>>>>
>>>> Then I change the Zone-wide primary to cluster-wide, it resolved.
>>>>
>>>> Right now, I have two primary storage, one is kvm cluster-wide,
>>>> another
>>>> is xenserver cluster-wide.
>>>>
>>>> Above is for people who have the same problem oneday.
>>>>
>>>> by the way, I am very curious why I never receive replys from this
>>>> a big
>>>> community ?? of course except the very beginning.
>>>>
>>>> Is my English skill really really poor, result in no body can
>>>> understood
>>>> what language I am speaking ?
>>>>
>>>> -----------
>>>> Cao Tong
>>>>
>>>> On 07/22/2015 09:03 PM, tony_caotong@163.com wrote:
>>>>> Hey! help please...
>>>>>
>>>>> some news.
>>>>> I think the cause is that the ACS host can't communicate with
>>>>> XenServer host.
>>>>> ACS continues outputing logs like this
>>>>>
>>>>> 2015-07-22 20:42:13,555 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>> (AgentManager-Handler-7:null) Seq 5-8174877748607582212: Forwarding
>>>>> Seq 5-8174877748607582212: { Cmd , MgmtId: 279278805451459, via: 5,
>>>>> Ver: v1, Flags: 100111,
>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>> 280345368052992
>>>>>
>>>>> I am not sure that if the ACS status is wrong or some services on
>>>>> xenserver are not opend.
>>>>>
>>>>> on xenserver , I found *xenheartbeat.sh is not running.*
>>>>> *(/bin/bash /opt/cloud/bin/xenheartbeat.sh
>>>>> 00d8e0d0-8561-4b3d-9044-cbc496ff22cc 120 60)*
>>>>>
>>>>> As some operations about xenserver was pending， xenserver can not be
>>>>> deleted from web UI.
>>>>>
>>>>> I got a temporary solution
>>>>>
>>>>> 1. delete jobs from DB cloud.vm_work_job.
>>>>> 2. delete xenserver from DB cloud.host.
>>>>> 3. add xenserver host back from web UI.
>>>>>
>>>>> then it works.
>>>>>
>>>>> Does anyone have a idea for this?
>>>>>
>>>>> Could anyone tell what things does ACS do on xenserver host when
>>>>> adding a xenserver ?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -----------
>>>>> Cao Tong
>>>>>
>>>>> On 07/22/2015 04:26 PM, tony_caotong@163.com wrote:
>>>>>> @prashant, following it the answer of you questions
>>>>>>
>>>>>> 1. Yes, primary storage is connected fine for my xenserver.
>>>>>>
>>>>>> 2. No, Xenserver's password is not changed.
>>>>>>
>>>>>> 3. yes, web UI is fine, and I can login.
>>>>>>
>>>>>> 4. before reboot, I unmanaged and disabled resources, and after
>>>>>> reboot I have enabled all of them.
>>>>>>
>>>>>> 5. hosts is states is UP.
>>>>>>
>>>>>> 6. No yum update in anywhere.
>>>>>>
>>>>>> 7. system VMs status is fine, i think.
>>>>>>
>>>>>> -----------
>>>>>> Cao Tong
>>>>>>
>>>>>> On 07/22/2015 04:13 PM, tony_caotong@163.com wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> After reinstall, I got the problem again
>>>>>>>
>>>>>>> So, I will describe once again.
>>>>>>>
>>>>>>> WHAT my environment looks like:
>>>>>>>
>>>>>>> I have a ACS server host and a xenserver host, After both reboot, I
>>>>>>> can not create a VM on xenserver through ACS.
>>>>>>> A KVM and A NFS are running together in ACS manager host.
>>>>>>>
>>>>>>> the status of new VM is always 'staring' on the WEB, but I can
>>>>>>> create new VM using xencenter.
>>>>>>>
>>>>>>> ------------- ERR LOGS ----------
>>>>>>> 2015-07-22 15:56:56,357 DEBUG [c.c.s.StorageManagerImpl]
>>>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to send storage pool command
>>>>>>> to Pool[4|NetworkFilesystem] via 4
>>>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>>>> 2829104990918803478 to Host 4 timed out after 3600
>>>>>>>
>>>>>>> 2015-07-22 15:56:56,358 INFO [c.c.s.StatsCollector]
>>>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to reach
>>>>>>> Pool[4|NetworkFilesystem]
>>>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>>>> [StoragePool:4] is unreachable: Unable to send command to the pool
>>>>>>>
>>>>>>>
>>>>>>> ------------- and there are lots of DEBUG infos ------- repeat
>>>>>>> again and again -----------
>>>>>>>
>>>>>>> 2015-07-22 15:36:12,887 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>> (AgentManager-Handler-14:null) Seq 4-8064821032713715922:
>>>>>>> Forwarding
>>>>>>> Seq 4-8064821032713715922: { Cmd , MgmtId: 227448510156211,
>>>>>>> via: 4,
>>>>>>> Ver: v1, Flags: 100111,
>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>> 116784073679673
>>>>>>> 2015-07-22 15:36:12,889 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>> (AgentManager-Handler-10:null) Seq 4-8064821032713715883:
>>>>>>> Forwarding
>>>>>>> Seq 4-8064821032713715883: { Cmd , MgmtId: 227448510156211,
>>>>>>> via: 4,
>>>>>>> Ver: v1, Flags: 100111,
>>>>>>>
>>>>>>> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap
>>>>>>>
>>>>>>> ache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/5
>>>>>>>
>>>>>>> /af949612-838f-3a6d-931b-312e612db740.vhd","origUrl":"http://download.c
>>>>>>>
>>>>>>> loud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-30
>>>>>>>
>>>>>>> 17-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksu
>>>>>>>
>>>>>>> m":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>>>>>>
>>>>>>>
>>>>>>> 5.6(64-bit) no GUI
>>>>>>>
>>>>>>> (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"
>>>>>>>
>>>>>>> nfs://10.0.0.100/storage/secondary","_role":"Image"}},"name":"centos56-
>>>>>>>
>>>>>>> x86_64-xen","hypervisorType":"XenServer"}},"destTO":{"org.apache.clouds
>>>>>>>
>>>>>>> tack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com
>>>>>>>
>>>>>>> /templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-
>>>>>>>
>>>>>>> 8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905c
>>>>>>>
>>>>>>> ec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>>>>>> 5.6(64-bit) no GUI
>>>>>>>
>>>>>>> (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.Primar
>>>>>>>
>>>>>>> yDataStoreTO":{"uuid":"2df26406-31bf-3a95-8a61-f5008defd9a0","id":4,"po
>>>>>>>
>>>>>>> olType":"NetworkFilesystem","host":"10.0.0.100","path":"/storage/xen/pr
>>>>>>>
>>>>>>> imary","port":2049,"url":"NetworkFilesystem://10.0.0.100/storage/xen/pr
>>>>>>>
>>>>>>> imary/?ROLE=Primary&STOREUUID=2df26406-31bf-3a95-8a61-f5008defd9a0"}},"
>>>>>>>
>>>>>>> name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"executeInSe
>>>>>>>
>>>>>>> quence":true,"options":{},"wait":10800}}]
>>>>>>> } to 116784073679673
>>>>>>>
>>>>>>>
>>>>>>> -----------------------------------------
>>>>>>>
>>>>>>> Anyone have Any ideas? thanks.
>>>>>>>
>>>>>>> -----------
>>>>>>> Cao Tong
>>>>>>>
>>>>>>> On 07/21/2015 06:14 PM, tony_caotong@163.com wrote:
>>>>>>>> Thanks all,
>>>>>>>>
>>>>>>>> I have already reinstall my hosts for preparing a new clear
>>>>>>>> environment to restart my research.
>>>>>>>>
>>>>>>>> -----------
>>>>>>>> Cao Tong
>>>>>>>>
>>>>>>>> On 07/20/2015 09:24 PM, Prashant s wrote:
>>>>>>>>> some questions :
>>>>>>>>>
>>>>>>>>> can you please tell ...
>>>>>>>>>
>>>>>>>>> 1. is your NFS storage or your primary Storage Repository in
>>>>>>>>> connected
>>>>>>>>> mode with no red cross mark on them in xencenter.
>>>>>>>>> 2. did you change any passwords on the xenservers ?
>>>>>>>>> 3. is the cloudstack web ui up , can you login to the cloudstack
>>>>>>>>> web page.
>>>>>>>>> 4. *are the zone , pod, or clusters in unmanaged or disabled
>>>>>>>>> state
>>>>>>>>> ? *
>>>>>>>>> *5. is all the hosts in connected state ? *
>>>>>>>>> *6. did you run yum update on host reboot on the cs manager
>>>>>>>>> vm ? *
>>>>>>>>> *7. system vms are stateless you can kill them and cs will
>>>>>>>>> recreate a new
>>>>>>>>> one .. so dont worry :-) *
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *thanks *
>>>>>>>>> *prashant *
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jul 20, 2015 at 3:47 AM, <to...@163.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi, I restartd All hosts (one mgr and xenserver) again.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Following is the error log.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Could not find exception:
>>>>>>>>>> com.cloud.exception.OperationTimedoutException in error code
>>>>>>>>>> list
>>>>>>>>>> for
>>>>>>>>>> exceptions
>>>>>>>>>> 2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed
>>>>>>>>>> out on null
>>>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972:
>>>>>>>>>> Cancelling.
>>>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to send storage pool
>>>>>>>>>> command to
>>>>>>>>>> Pool[1|NetworkFilesystem] via 1
>>>>>>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>>>>>>> 3176445112179752972 to Host 1 timed out after 3600
>>>>>>>>>> at
>>>>>>>>>> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:
>>>>>>>>>>
>>>>>>>>>> 433)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:
>>>>>>>>>>
>>>>>>>>>> 362)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:1000)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:392)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:406)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC
>>>>>>>>>>
>>>>>>>>>> ollector.java:642)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
>>>>>>>>>>
>>>>>>>>>> anagedContextRunnable.java:49)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
>>>>>>>>>>
>>>>>>>>>> all(DefaultManagedContext.java:56)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
>>>>>>>>>>
>>>>>>>>>> lWithContext(DefaultManagedContext.java:103)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
>>>>>>>>>>
>>>>>>>>>> WithContext(DefaultManagedContext.java:53)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
>>>>>>>>>>
>>>>>>>>>> agedContextRunnable.java:46)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>>>>>>
>>>>>>>>>> 71)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>>
>>>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>>
>>>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>>>>>>>>>
>>>>>>>>>> .java:1145)
>>>>>>>>>>
>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>> 2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to reach
>>>>>>>>>> Pool[1|NetworkFilesystem]
>>>>>>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>>>>>>> [StoragePool:1]
>>>>>>>>>> is unreachable: Unable to send command to the pool
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:1010)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:392)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:406)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC
>>>>>>>>>>
>>>>>>>>>> ollector.java:642)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
>>>>>>>>>>
>>>>>>>>>> anagedContextRunnable.java:49)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
>>>>>>>>>>
>>>>>>>>>> all(DefaultManagedContext.java:56)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
>>>>>>>>>>
>>>>>>>>>> lWithContext(DefaultManagedContext.java:103)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
>>>>>>>>>>
>>>>>>>>>> WithContext(DefaultManagedContext.java:53)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
>>>>>>>>>>
>>>>>>>>>> agedContextRunnable.java:46)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>>>>>>
>>>>>>>>>> 71)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>>
>>>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>>
>>>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>>>>>>>>>
>>>>>>>>>> .java:1145)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto
>>>>>>>>>>
>>>>>>>>>> r.java:615)
>>>>>>>>>>
>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>>
>>>>>>>>>> -----------
>>>>>>>>>> Cao Tong
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 07/20/2015 02:52 PM, tony_caotong@163.com wrote:
>>>>>>>>>>
>>>>>>>>>>> No, no one's IP was changed.
>>>>>>>>>>>
>>>>>>>>>>> 1. In xenserver I can not login systemvms using the internal IP
>>>>>>>>>>> like
>>>>>>>>>>> '169.254.1.112', There shoud be a bridge network for this
>>>>>>>>>>> right? it is
>>>>>>>>>>> gone.
>>>>>>>>>>>
>>>>>>>>>>> 2. I try to delete xenserver host from CS on web, it also
>>>>>>>>>>> failed
>>>>>>>>>>> with
>>>>>>>>>>> lots of logs like following, then memory is full and mangement
>>>>>>>>>>> down...
>>>>>>>>>>>
>>>>>>>>>>> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-15:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411,
>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-5:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411,
>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-1:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411,
>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-14:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411,
>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I guess that, is there some service or daemons working for CS
>>>>>>>>>>> is not up
>>>>>>>>>>> on Xenserver ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -----------
>>>>>>>>>>> Cao Tong
>>>>>>>>>>> On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Did the management server ip change?
>>>>>>>>>>>> management server ip in the configuration table is used my
>>>>>>>>>>>> systemvms.
>>>>>>>>>>>> select * from configuration where name like 'host';
>>>>>>>>>>>>
>>>>>>>>>>>> If it changed, correct the value in db and restart systemvms.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ~Rajani
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jul 20, 2015 at 11:56 AM,<to...@163.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>> I shutdown my cs-manager and xenserver last weekend, And now
>>>>>>>>>>>>> the ssvm
>>>>>>>>>>>>> and cpvm is disconnect, thost two was runing on xenserver. so
>>>>>>>>>>>>> What
>>>>>>>>>>>>> should i do right now ?
>>>>>>>>>>>>> Please anybody help me and thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In xenserver I found that the three system VMs are not
>>>>>>>>>>>>> running.
>>>>>>>>>>>>> my xenserver seems can not reconnect to CS-manager. and it
>>>>>>>>>>>>> seams not
>>>>>>>>>>>>> under control of CS.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> What is the right steps of shutdown all CS group machines and
>>>>>>>>>>>>> resume
>>>>>>>>>>>>> them?
>>>>>>>>>>>>> How can i let my xenserver reconnected ?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> -----------
>>>>>>>>>>>>> Cao Tong
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>
>>
>
>
Re: XenServer is disconnected after CS hosts shutdown
Posted by to...@163.com.
After I disabled and enabled XS cluster & primary. it works.
seems XS agent have started by ACS host. so what is the correct
behavior of starting XS agent manually ?
-----------
Cao Tong
On 08/12/2015 04:41 PM, tony_caotong@163.com wrote:
>
> Hello,
>
> Almost one month gone, my problem is still here. I really really need
> someone to help me.
>
> new Settings ACS 4.4.4 XS 6.2
> After reboot, errors here:
>
> 2015-08-12 16:18:54,557 INFO [c.c.a.t.Request]
> (StatsCollector-3:ctx-b287085a) not building log message for '[{}]',
> _cmds.length == 1
> 2015-08-12 16:18:54,557 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Forwarding
> null to 191386435611186
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-12:null) Seq 9-4450963806725603425: Routing from
> 249082151178140
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (AgentManager-Handler-12:null) Seq 9-4450963806725603425: Link is closed
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-12:null) Seq 9-4450963806725603425: MgmtId
> 249082151178140: Req: Resource [Host:9] is unreachable: Host 9: Link
> is closed
> 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-12:null) Seq 9--1: MgmtId 249082151178140: Req:
> Routing to peer
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentManager-Handler-15:null) Seq 9--1: MgmtId 249082151178140: Req:
> Cancel request received
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
> (AgentManager-Handler-15:null) Seq 9-4450963806725603425: Cancelling.
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Waiting
> some more time because this is the current command
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Waiting
> some more time because this is the current command
> 2015-08-12 16:18:54,559 INFO [c.c.u.e.CSExceptionErrorCode]
> (StatsCollector-3:ctx-b287085a) Could not find exception:
> com.cloud.exception.OperationTimedoutException in error code list for
> exceptions
> 2015-08-12 16:18:54,559 INFO [c.c.a.t.Request]
> (StatsCollector-3:ctx-b287085a) not building log message for '[{}]',
> _cmds.length == 1
> 2015-08-12 16:18:54,559 WARN [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Timed out
> on null
> 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache]
> (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Cancelling.
> 2015-08-12 16:18:54,559 DEBUG [c.c.s.StorageManagerImpl]
> (StatsCollector-3:ctx-b287085a) Unable to send storage pool command to
> Pool[3|NetworkFilesystem] via 9
> com.cloud.exception.OperationTimedoutException: Commands
> 4450963806725603425 to Host 9 timed out after 3600
> at
> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:434)
> at
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:418)
> at
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:965)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:390)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:404)
> at
> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
>
>
>
> -----------
> Cao Tong
>
> On 07/30/2015 10:55 AM, tony_caotong@163.com wrote:
>>
>> Hi Yiping, Thanks for your reply first.
>>
>>
>> My NFS server deployed in ACS manager host, another host is a single
>> xenserver.
>> A KVM environment deployed in ACS manager host too.
>> there is three storages named, Kprimary, Xprimary, Secondary.
>>
>> First, I add KVM cluster with zone-wide Kprimary, it works fine even
>> if after reboot.
>> Second, I add Xenserver with Xprimary(cluster-wide), it still works
>> fine.
>> Then, I set zone to disable, shutdown all system VMs, disable
>> those two hosts. finally shutdown xenserver host.
>> When xenserver's shutdown finish, I stop those services by order (
>> cloudstack-agent, cloudstack-management, libvirtd, nfs, rpcbind,
>> mysqld).
>> At last shutdown ACS host.
>>
>> The start process is totally reverse of this order.
>>
>> when done of starting, ACS says unable to send cmd to pool via host
>> xenserver.
>>
>> I am sure nothing have change in my environment during reboot time.
>> When I fix this problem, the only change is change Kprimary from
>> zone-wide to cluster-wide.
>>
>> I guess that after reboot some status have been initial from
>> beginning, ACS found that Xenserver host have two primary can be used.
>> so it use the high priority one， and it is Kprimary.
>>
>> Whatever, Maybe it could help you peoples to get ACS better.
>>
>> BTW, Error logs attached some days before is already point out this
>> is a storage problem
>> like:
>>
>> Unable to send storage pool command
>> to Pool[4|NetworkFilesystem] via 4
>>
>>
>>
>> -----------
>> Cao Tong
>>
>> On 07/30/2015 12:44 AM, Yiping Zhang wrote:
>>> Well, sometimes people can’t answer a question because of lack of
>>> relevant information, or simply because no one has encountered a
>>> similar
>>> situation before.
>>>
>>> Looking at your past messages on this thread, there were no mentions
>>> about
>>> primary storage. Obviously, your primary storage configuration had
>>> changed
>>> between the time you shut down CS manager and xenservers and the
>>> time you
>>> restarted them. That is the vital info the list didn’t know.
>>>
>>> To best of my knowledge, zone wide primary storage has never been
>>> supported for Xen hypervisors.
>>>
>>> I do have to say that quite often CloudStack error messages are very
>>> cryptic, do not provide enough *useful* information to help users
>>> identify
>>> and trouble shoot actual problems. Those stack trace output might be a
>>> gold mine to developers, but they are utterly useless for end users.
>>>
>>> Just my $0.02
>>>
>>> Yiping
>>>
>>> On 7/28/15, 11:19 PM, "tony_caotong@163.com" <to...@163.com>
>>> wrote:
>>>
>>>> Hi, Finally I resolved this problem by my self.
>>>>
>>>> * Primary Storage: A storage resource typically provided to a single
>>>> cluster for the actual running of instance disk images. (Zone-wide
>>>> primary storage is an option, though not typically used.)
>>>>
>>>> This line above is from
>>>> http://docs.cloudstack.apache.org/en/master/concepts.html
>>>>
>>>> Because I have a Zone-wide primary storage, ACS can not find the
>>>> correct
>>>> primary which belong to XenServer cluster after reboot.
>>>>
>>>> Then I change the Zone-wide primary to cluster-wide, it resolved.
>>>>
>>>> Right now, I have two primary storage, one is kvm cluster-wide,
>>>> another
>>>> is xenserver cluster-wide.
>>>>
>>>> Above is for people who have the same problem oneday.
>>>>
>>>> by the way, I am very curious why I never receive replys from this
>>>> a big
>>>> community ?? of course except the very beginning.
>>>>
>>>> Is my English skill really really poor, result in no body can
>>>> understood
>>>> what language I am speaking ?
>>>>
>>>> -----------
>>>> Cao Tong
>>>>
>>>> On 07/22/2015 09:03 PM, tony_caotong@163.com wrote:
>>>>> Hey! help please...
>>>>>
>>>>> some news.
>>>>> I think the cause is that the ACS host can't communicate with
>>>>> XenServer host.
>>>>> ACS continues outputing logs like this
>>>>>
>>>>> 2015-07-22 20:42:13,555 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>> (AgentManager-Handler-7:null) Seq 5-8174877748607582212: Forwarding
>>>>> Seq 5-8174877748607582212: { Cmd , MgmtId: 279278805451459, via: 5,
>>>>> Ver: v1, Flags: 100111,
>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>> 280345368052992
>>>>>
>>>>> I am not sure that if the ACS status is wrong or some services on
>>>>> xenserver are not opend.
>>>>>
>>>>> on xenserver , I found *xenheartbeat.sh is not running.*
>>>>> *(/bin/bash /opt/cloud/bin/xenheartbeat.sh
>>>>> 00d8e0d0-8561-4b3d-9044-cbc496ff22cc 120 60)*
>>>>>
>>>>> As some operations about xenserver was pending， xenserver can not be
>>>>> deleted from web UI.
>>>>>
>>>>> I got a temporary solution
>>>>>
>>>>> 1. delete jobs from DB cloud.vm_work_job.
>>>>> 2. delete xenserver from DB cloud.host.
>>>>> 3. add xenserver host back from web UI.
>>>>>
>>>>> then it works.
>>>>>
>>>>> Does anyone have a idea for this?
>>>>>
>>>>> Could anyone tell what things does ACS do on xenserver host when
>>>>> adding a xenserver ?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -----------
>>>>> Cao Tong
>>>>>
>>>>> On 07/22/2015 04:26 PM, tony_caotong@163.com wrote:
>>>>>> @prashant, following it the answer of you questions
>>>>>>
>>>>>> 1. Yes, primary storage is connected fine for my xenserver.
>>>>>>
>>>>>> 2. No, Xenserver's password is not changed.
>>>>>>
>>>>>> 3. yes, web UI is fine, and I can login.
>>>>>>
>>>>>> 4. before reboot, I unmanaged and disabled resources, and after
>>>>>> reboot I have enabled all of them.
>>>>>>
>>>>>> 5. hosts is states is UP.
>>>>>>
>>>>>> 6. No yum update in anywhere.
>>>>>>
>>>>>> 7. system VMs status is fine, i think.
>>>>>>
>>>>>> -----------
>>>>>> Cao Tong
>>>>>>
>>>>>> On 07/22/2015 04:13 PM, tony_caotong@163.com wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> After reinstall, I got the problem again
>>>>>>>
>>>>>>> So, I will describe once again.
>>>>>>>
>>>>>>> WHAT my environment looks like:
>>>>>>>
>>>>>>> I have a ACS server host and a xenserver host, After both reboot, I
>>>>>>> can not create a VM on xenserver through ACS.
>>>>>>> A KVM and A NFS are running together in ACS manager host.
>>>>>>>
>>>>>>> the status of new VM is always 'staring' on the WEB, but I can
>>>>>>> create new VM using xencenter.
>>>>>>>
>>>>>>> ------------- ERR LOGS ----------
>>>>>>> 2015-07-22 15:56:56,357 DEBUG [c.c.s.StorageManagerImpl]
>>>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to send storage pool command
>>>>>>> to Pool[4|NetworkFilesystem] via 4
>>>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>>>> 2829104990918803478 to Host 4 timed out after 3600
>>>>>>>
>>>>>>> 2015-07-22 15:56:56,358 INFO [c.c.s.StatsCollector]
>>>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to reach
>>>>>>> Pool[4|NetworkFilesystem]
>>>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>>>> [StoragePool:4] is unreachable: Unable to send command to the pool
>>>>>>>
>>>>>>>
>>>>>>> ------------- and there are lots of DEBUG infos ------- repeat
>>>>>>> again and again -----------
>>>>>>>
>>>>>>> 2015-07-22 15:36:12,887 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>> (AgentManager-Handler-14:null) Seq 4-8064821032713715922:
>>>>>>> Forwarding
>>>>>>> Seq 4-8064821032713715922: { Cmd , MgmtId: 227448510156211,
>>>>>>> via: 4,
>>>>>>> Ver: v1, Flags: 100111,
>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>> 116784073679673
>>>>>>> 2015-07-22 15:36:12,889 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>> (AgentManager-Handler-10:null) Seq 4-8064821032713715883:
>>>>>>> Forwarding
>>>>>>> Seq 4-8064821032713715883: { Cmd , MgmtId: 227448510156211,
>>>>>>> via: 4,
>>>>>>> Ver: v1, Flags: 100111,
>>>>>>>
>>>>>>> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap
>>>>>>>
>>>>>>> ache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/5
>>>>>>>
>>>>>>> /af949612-838f-3a6d-931b-312e612db740.vhd","origUrl":"http://download.c
>>>>>>>
>>>>>>> loud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-30
>>>>>>>
>>>>>>> 17-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksu
>>>>>>>
>>>>>>> m":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>>>>>>
>>>>>>>
>>>>>>> 5.6(64-bit) no GUI
>>>>>>>
>>>>>>> (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"
>>>>>>>
>>>>>>> nfs://10.0.0.100/storage/secondary","_role":"Image"}},"name":"centos56-
>>>>>>>
>>>>>>> x86_64-xen","hypervisorType":"XenServer"}},"destTO":{"org.apache.clouds
>>>>>>>
>>>>>>> tack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com
>>>>>>>
>>>>>>> /templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5-
>>>>>>>
>>>>>>> 8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905c
>>>>>>>
>>>>>>> ec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS
>>>>>>> 5.6(64-bit) no GUI
>>>>>>>
>>>>>>> (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.Primar
>>>>>>>
>>>>>>> yDataStoreTO":{"uuid":"2df26406-31bf-3a95-8a61-f5008defd9a0","id":4,"po
>>>>>>>
>>>>>>> olType":"NetworkFilesystem","host":"10.0.0.100","path":"/storage/xen/pr
>>>>>>>
>>>>>>> imary","port":2049,"url":"NetworkFilesystem://10.0.0.100/storage/xen/pr
>>>>>>>
>>>>>>> imary/?ROLE=Primary&STOREUUID=2df26406-31bf-3a95-8a61-f5008defd9a0"}},"
>>>>>>>
>>>>>>> name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"executeInSe
>>>>>>>
>>>>>>> quence":true,"options":{},"wait":10800}}]
>>>>>>> } to 116784073679673
>>>>>>>
>>>>>>>
>>>>>>> -----------------------------------------
>>>>>>>
>>>>>>> Anyone have Any ideas? thanks.
>>>>>>>
>>>>>>> -----------
>>>>>>> Cao Tong
>>>>>>>
>>>>>>> On 07/21/2015 06:14 PM, tony_caotong@163.com wrote:
>>>>>>>> Thanks all,
>>>>>>>>
>>>>>>>> I have already reinstall my hosts for preparing a new clear
>>>>>>>> environment to restart my research.
>>>>>>>>
>>>>>>>> -----------
>>>>>>>> Cao Tong
>>>>>>>>
>>>>>>>> On 07/20/2015 09:24 PM, Prashant s wrote:
>>>>>>>>> some questions :
>>>>>>>>>
>>>>>>>>> can you please tell ...
>>>>>>>>>
>>>>>>>>> 1. is your NFS storage or your primary Storage Repository in
>>>>>>>>> connected
>>>>>>>>> mode with no red cross mark on them in xencenter.
>>>>>>>>> 2. did you change any passwords on the xenservers ?
>>>>>>>>> 3. is the cloudstack web ui up , can you login to the cloudstack
>>>>>>>>> web page.
>>>>>>>>> 4. *are the zone , pod, or clusters in unmanaged or disabled
>>>>>>>>> state
>>>>>>>>> ? *
>>>>>>>>> *5. is all the hosts in connected state ? *
>>>>>>>>> *6. did you run yum update on host reboot on the cs manager
>>>>>>>>> vm ? *
>>>>>>>>> *7. system vms are stateless you can kill them and cs will
>>>>>>>>> recreate a new
>>>>>>>>> one .. so dont worry :-) *
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *thanks *
>>>>>>>>> *prashant *
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jul 20, 2015 at 3:47 AM, <to...@163.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi, I restartd All hosts (one mgr and xenserver) again.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Following is the error log.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Could not find exception:
>>>>>>>>>> com.cloud.exception.OperationTimedoutException in error code
>>>>>>>>>> list
>>>>>>>>>> for
>>>>>>>>>> exceptions
>>>>>>>>>> 2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed
>>>>>>>>>> out on null
>>>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972:
>>>>>>>>>> Cancelling.
>>>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to send storage pool
>>>>>>>>>> command to
>>>>>>>>>> Pool[1|NetworkFilesystem] via 1
>>>>>>>>>> com.cloud.exception.OperationTimedoutException: Commands
>>>>>>>>>> 3176445112179752972 to Host 1 timed out after 3600
>>>>>>>>>> at
>>>>>>>>>> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:
>>>>>>>>>>
>>>>>>>>>> 433)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:
>>>>>>>>>>
>>>>>>>>>> 362)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:1000)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:392)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:406)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC
>>>>>>>>>>
>>>>>>>>>> ollector.java:642)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
>>>>>>>>>>
>>>>>>>>>> anagedContextRunnable.java:49)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
>>>>>>>>>>
>>>>>>>>>> all(DefaultManagedContext.java:56)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
>>>>>>>>>>
>>>>>>>>>> lWithContext(DefaultManagedContext.java:103)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
>>>>>>>>>>
>>>>>>>>>> WithContext(DefaultManagedContext.java:53)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
>>>>>>>>>>
>>>>>>>>>> agedContextRunnable.java:46)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>>>>>>
>>>>>>>>>> 71)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>>
>>>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>>
>>>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>>>>>>>>>
>>>>>>>>>> .java:1145)
>>>>>>>>>>
>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>> 2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
>>>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to reach
>>>>>>>>>> Pool[1|NetworkFilesystem]
>>>>>>>>>> com.cloud.exception.StorageUnavailableException: Resource
>>>>>>>>>> [StoragePool:1]
>>>>>>>>>> is unreachable: Unable to send command to the pool
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:1010)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:392)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j
>>>>>>>>>>
>>>>>>>>>> ava:406)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC
>>>>>>>>>>
>>>>>>>>>> ollector.java:642)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
>>>>>>>>>>
>>>>>>>>>> anagedContextRunnable.java:49)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
>>>>>>>>>>
>>>>>>>>>> all(DefaultManagedContext.java:56)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
>>>>>>>>>>
>>>>>>>>>> lWithContext(DefaultManagedContext.java:103)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
>>>>>>>>>>
>>>>>>>>>> WithContext(DefaultManagedContext.java:53)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
>>>>>>>>>>
>>>>>>>>>> agedContextRunnable.java:46)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>>>>>>
>>>>>>>>>> 71)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>>
>>>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>>>>
>>>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>>>>>>>>>
>>>>>>>>>> .java:1145)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto
>>>>>>>>>>
>>>>>>>>>> r.java:615)
>>>>>>>>>>
>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>>
>>>>>>>>>> -----------
>>>>>>>>>> Cao Tong
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 07/20/2015 02:52 PM, tony_caotong@163.com wrote:
>>>>>>>>>>
>>>>>>>>>>> No, no one's IP was changed.
>>>>>>>>>>>
>>>>>>>>>>> 1. In xenserver I can not login systemvms using the internal IP
>>>>>>>>>>> like
>>>>>>>>>>> '169.254.1.112', There shoud be a bridge network for this
>>>>>>>>>>> right? it is
>>>>>>>>>>> gone.
>>>>>>>>>>>
>>>>>>>>>>> 2. I try to delete xenserver host from CS on web, it also
>>>>>>>>>>> failed
>>>>>>>>>>> with
>>>>>>>>>>> lots of logs like following, then memory is full and mangement
>>>>>>>>>>> down...
>>>>>>>>>>>
>>>>>>>>>>> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-15:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411,
>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-5:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411,
>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-1:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411,
>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>>>>>>>>> (AgentManager-Handler-14:null) Seq 1-7282039122481381399:
>>>>>>>>>>> Forwarding Seq
>>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411,
>>>>>>>>>>> via: 1,
>>>>>>>>>>> Ver: v1,
>>>>>>>>>>> Flags: 100111,
>>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to
>>>>>>>>>>> 192405008094602
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I guess that, is there some service or daemons working for CS
>>>>>>>>>>> is not up
>>>>>>>>>>> on Xenserver ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -----------
>>>>>>>>>>> Cao Tong
>>>>>>>>>>> On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Did the management server ip change?
>>>>>>>>>>>> management server ip in the configuration table is used my
>>>>>>>>>>>> systemvms.
>>>>>>>>>>>> select * from configuration where name like 'host';
>>>>>>>>>>>>
>>>>>>>>>>>> If it changed, correct the value in db and restart systemvms.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ~Rajani
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jul 20, 2015 at 11:56 AM,<to...@163.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>> I shutdown my cs-manager and xenserver last weekend, And now
>>>>>>>>>>>>> the ssvm
>>>>>>>>>>>>> and cpvm is disconnect, thost two was runing on xenserver. so
>>>>>>>>>>>>> What
>>>>>>>>>>>>> should i do right now ?
>>>>>>>>>>>>> Please anybody help me and thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In xenserver I found that the three system VMs are not
>>>>>>>>>>>>> running.
>>>>>>>>>>>>> my xenserver seems can not reconnect to CS-manager. and it
>>>>>>>>>>>>> seams not
>>>>>>>>>>>>> under control of CS.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> What is the right steps of shutdown all CS group machines and
>>>>>>>>>>>>> resume
>>>>>>>>>>>>> them?
>>>>>>>>>>>>> How can i let my xenserver reconnected ?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> -----------
>>>>>>>>>>>>> Cao Tong
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>
>>
>
>