You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Pierre-Luc Dion <pd...@cloudops.com> on 2014/02/04 21:38:04 UTC
unable to remove xenserver host from a cluster
Using Cloudstack 4.2.1 managing a xenserver cluster (xenserver pool)
running on 6.2sp1
I've proceed with removing a host from a pool managed by cloudstack,I've
enable the maintenance mode in cloudstack for the host , make sure no
Instance where running on the host. The delete the host.
the end result now is that the host have been removed in the host inventory
in cloudstack, but it is now the pool master in the xenserver pool.
Also I'm now having this repeating stack trace :
2014-02-04 15:26:55,698 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-209:null) Vm cpu utilization 0.005
2014-02-04 15:26:55,705 WARN [xen.resource.CitrixResourceBase]
(DirectAgent-209:null) Error while collecting disk stats from :
You gave an invalid object reference. The object may have recently been
deleted. The class parameter gives the type of reference given, and the
handle parameter echoes the bad value given.
at com.xensource.xenapi.Types.checkResponse(Types.java:209)
at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
at com.xensource.xenapi.VBDMetrics.getIoReadKbs(VBDMetrics.java:210)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getVmStats(CitrixResourceBase.java:2791)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:2691)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:497)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
And here is the logs during the removal job:
2014-02-04 15:04:27,871 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager
Timer:null) Resetting hosts suitable for reconnect
2014-02-04 15:04:27,873 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager
Timer:null) Completed resetting hosts suitable for reconnect
2014-02-04 15:04:27,873 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager
Timer:null) Acquiring hosts for clusters already owned by this management
server
2014-02-04 15:04:27,874 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager
Timer:null) Completed acquiring hosts for clusters already owned by this
management server
2014-02-04 15:04:27,874 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager
Timer:null) Acquiring hosts for clusters not owned by any management server
2014-02-04 15:04:27,874 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager
Timer:null) Completed acquiring hosts for clusters not owned by any
management server
2014-02-04 15:04:33,848 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-178:null) XmlRpcException: Still waiting for the conversion to
the master Failed to create input stream: Read timed out
2014-02-04 15:04:38,870 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-178:null) Logging on as the slave to 172.16.20.36
2014-02-04 15:04:38,915 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-178:null) Slave logon successful. session=
com.xensource.xenapi.Session@4d81ff53
2014-02-04 15:04:38,923 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-178:null) Logging on as the master to 172.16.20.34
2014-02-04 15:04:38,968 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-178:null) Remove master connection through 172.16.20.36 for
pool(null)
2014-02-04 15:04:49,100 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-178:null) Logging on as the slave to 172.16.20.37
2014-02-04 15:04:49,154 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-178:null) Logging on as the slave to 172.16.20.36
2014-02-04 15:04:49,204 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-178:null) Logging on as the slave to 172.16.20.34
2014-02-04 15:04:49,245 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-178:null) Logging on as the slave to 172.16.20.36
2014-02-04 15:04:49,288 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-178:null) Logging on as the slave to 172.16.20.37
2014-02-04 15:04:49,333 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-225:null) Logging on as the slave to 172.16.20.34
2014-02-04 15:04:49,345 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-178:null) Session is invalid for method: pool.eject due to
null. Reconnecting...retry=1
2014-02-04 15:04:49,390 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-178:null) HostIsSlave Exception for method:
session.get_this_host due to null. Reconnecting...retry=1
2014-02-04 15:04:49,390 WARN [xen.resource.CitrixResourceBase]
(DirectAgent-178:null) Unable to eject host
6819db51-87b7-4ba2-a630-077e48c9d48f due to You cannot make regular API
calls directly on a slave. Please pass API calls via the master host.
2014-02-04 15:04:49,396 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-178:null) HostIsSlave Exception for method: host.destroy due
to null. Reconnecting...retry=1
2014-02-04 15:04:49,396 WARN [xen.resource.CitrixResourceBase]
(DirectAgent-178:null) Exception Unable to destroy host
6819db51-87b7-4ba2-a630-077e48c9d48f in xenserver database due to You
cannot make regular API calls directly on a slave. Please pass API calls
via the master host.
You cannot make regular API calls directly on a slave. Please pass API
calls via the master host.
at com.xensource.xenapi.Types.checkResponse(Types.java:262)
at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
at com.xensource.xenapi.Host.destroy(Host.java:1922)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:7898)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:559)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
2014-02-04 15:04:49,397 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-178:null) Seq 1-1530859468: Response Received:
2014-02-04 15:04:49,397 DEBUG [agent.transport.Request]
(DirectAgent-178:null) Seq 1-1530859468: Processing: { Ans: , MgmtId:
6794168500400, via: 1, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"Exception Unable
to destroy host 6819db51-87b7-4ba2-a630-077e48c9d48f in xenserver database
due to You cannot make regular API calls directly on a slave. Please pass
API calls via the master host.","wait":0}}] }
2014-02-04 15:04:49,397 DEBUG [agent.manager.AgentAttache]
(DirectAgent-178:null) Seq 1-1530859468: No more commands found
2014-02-04 15:04:49,397 DEBUG [agent.transport.Request]
(catalina-exec-22:null) Seq 1-1530859468: Received: { Ans: , MgmtId:
6794168500400, via: 1, Ver: v1, Flags: 110, { Answer } }
2014-02-04 15:04:49,397 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-22:null) Details from executing class
com.cloud.agent.api.PoolEjectCommand: Exception Unable to destroy host
6819db51-87b7-4ba2-a630-077e48c9d48f in xenserver database due to You
cannot make regular API calls directly on a slave. Please pass API calls
via the master host.
2014-02-04 15:04:49,397 WARN [xen.discoverer.XcpServerDiscoverer]
(catalina-exec-22:null) Eject Host: 4 from 1 failed due to Exception Unable
to destroy host 6819db51-87b7-4ba2-a630-077e48c9d48f in xenserver database
due to You cannot make regular API calls directly on a slave. Please pass
API calls via the master host.
2014-02-04 15:04:49,400 DEBUG [agent.transport.Request]
(catalina-exec-22:null) Seq 5-882053032: Sending { Cmd , MgmtId:
6794168500400, via: 5, Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.PoolEjectCommand":{"hostuuid":"571299d2-e0c8-4cc4-a730-8d9aaf9f0f81","wait":0}}]
}
2014-02-04 15:04:49,400 DEBUG [agent.transport.Request]
(catalina-exec-22:null) Seq 5-882053032: Executing: { Cmd , MgmtId:
6794168500400, via: 5, Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.PoolEjectCommand":{"hostuuid":"571299d2-e0c8-4cc4-a730-8d9aaf9f0f81","wait":0}}]
}
2014-02-04 15:04:49,400 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-448:null) Seq 5-882053032: Executing request
2014-02-04 15:04:49,416 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-225:null) Add master connection through 172.16.20.34 for
pool(null)
2014-02-04 15:04:49,609 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-363:null) Ping from 5
2014-02-04 15:04:49,648 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-300:null) Ping from 4
2014-02-04 15:04:49,649 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-358:null) Ping from 1
2014-02-04 15:04:49,665 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-31:null) Seq 4-557056006: Response Received:
2014-02-04 15:04:49,666 DEBUG [agent.transport.Request]
(DirectAgent-31:null) Seq 4-557056006: Processing: { Ans: , MgmtId:
6794168500400, via: 4, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2014-02-04 15:04:49,666 DEBUG [agent.manager.AgentAttache]
(DirectAgent-31:null) Seq 4-557056006: Unable to find listener.
2014-02-04 15:04:49,678 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-477:null) Seq 1-1530855430: Response Received:
2014-02-04 15:04:49,678 DEBUG [agent.transport.Request]
(DirectAgent-477:null) Seq 1-1530855430: Processing: { Ans: , MgmtId:
6794168500400, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2014-02-04 15:04:49,684 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-382:null) Seq 5-882049030: Response Received:
2014-02-04 15:04:49,684 DEBUG [agent.transport.Request]
(DirectAgent-382:null) Seq 5-882049030: Processing: { Ans: , MgmtId:
6794168500400, via: 5, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.ClusterSyncAnswer":{"_clusterId":1,"_newStates":{},"_isExecuted":false,"result":true,"wait":0}}]
}
2014-02-04 15:04:49,770 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-448:null) New master for the XenPool is
571299d2-e0c8-4cc4-a730-8d9aaf9f0f81 : 172.16.20.37
2014-02-04 15:04:49,773 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-448:null) Designating the new master to 172.16.20.37
2014-02-04 15:04:49,773 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-448:null) Designating the new master to 172.16.20.37
2014-02-04 15:04:49,859 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-225:null) Vm cpu utilization 0.005
2014-02-04 15:04:49,996 WARN [xen.resource.CitrixResourceBase]
(DirectAgent-225:null) Error while collecting disk stats from :
You gave an invalid object reference. The object may have recently been
deleted. The class parameter gives the type of reference given, and the
handle parameter echoes the bad value given.
at com.xensource.xenapi.Types.checkResponse(Types.java:209)
at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
at com.xensource.xenapi.VBDMetrics.getIoReadKbs(VBDMetrics.java:210)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getVmStats(CitrixResourceBase.java:2791)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:2691)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:497)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Should I fill a jira bug for this ?
Thanks,
Pierre-Luc Dion
Architecte de Solution Cloud | Cloud Solutions Architect
514-447-3456, 1101
- - -
*CloudOps*420 rue Guy
Montréal QC H3J 1S6
www.cloudops.com
@CloudOps_