You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Simon Weller (JIRA)" <ji...@apache.org> on 2016/03/03 23:54:18 UTC
[jira] [Commented] (CLOUDSTACK-9285) Cloudstack 4.8 can't connect
to XEN and KVM hosts
[ https://issues.apache.org/jira/browse/CLOUDSTACK-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178769#comment-15178769 ]
Simon Weller commented on CLOUDSTACK-9285:
------------------------------------------
Are you using a load balancer between your management server and the host agent?
We are seeing this with haproxy between our agents and management server. If we down all the management servers, the agents connection is cut to the load balancer and the agent throws a "Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions."
We're digging into this right now.
> Cloudstack 4.8 can't connect to XEN and KVM hosts
> -------------------------------------------------
>
> Key: CLOUDSTACK-9285
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9285
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: KVM, Xen
> Affects Versions: 4.8.0
> Environment: CentOS 7
> Reporter: Jens Fettig
> Priority: Blocker
>
> We used Cloudstack 4.7 in our testing environment. For future uses we updated to Cloudstack 4.8 for some tests. But after the update from 4.7 to 4.8 cloudstack has some problems to connect to the hosts again. Here are the error logs.
> {code:title=XEN Server from the managment-server.log:|borderStyle=solid}
> 2016-02-15 11:22:03,476 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Loading directly connected host 13(xen5)
> 2016-02-15 11:22:06,204 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-aaf933f3) (logid:d157e37d) AutoScaling Monitor is running...
> 2016-02-15 11:22:06,206 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-8e2fc084) (logid:379074cb) VmStatsCollector is running...
> 2016-02-15 11:22:06,217 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-cc1cf960) (logid:b6397e2e) HostStatsCollector is running...
> 2016-02-15 11:22:06,298 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-efda9c07) (logid:5731423e) StorageCollector is running...
> 2016-02-15 11:22:06,302 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-efda9c07) (logid:5731423e) There is no secondary storage VM for secondary storage host cdsdev-secondary
> 2016-02-15 11:22:11,018 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) Begin cleanup expired async-jobs
> 2016-02-15 11:22:11,024 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) End cleanup expired async-jobs
> 2016-02-15 11:22:11,055 DEBUG [c.c.h.x.r.XenServerConnectionPool] (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Unable to create master connection to host(192.168.0.97) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
> 2016-02-15 11:22:11,055 DEBUG [c.c.h.Status] (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 13, name = xen5]
> 2016-02-15 11:22:11,101 WARN [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) can not load directly connected host 13(xen5) due to
> com.cloud.utils.exception.CloudRuntimeException: Unable to create master connection to host(192.168.0.97) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
> at com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:163)
> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:523)
> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.configure(CitrixResourceBase.java:827)
> at com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:158)
> at com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:697)
> at com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:217)
> at com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:182)
> at com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:96)
> at com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:233)
> at org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> Caused by: The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
> at com.xensource.xenapi.Types.checkResponse(Types.java:322)
> at com.xensource.xenapi.Connection.dispatch(Connection.java:395)
> at com.xensource.xenapi.Session.loginWithPassword(Session.java:587)
> ... 19 more
> {code}
> This seems to be the problem:
> Unable to create master connection to host(192.168.0.97) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
> I updated the vhd-utils like in the upgrade gourde from 4.6:
> http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.7.0/upgrade/upgrade-4.6.html
> But maybe a need a new vhd-utils file for 4.8, because this one was for 4.7?
> {code:title=KVM from the agent.log:|borderStyle=solid}
> 2016-02-15 10:46:20,971 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
> 2016-02-15 10:46:20,973 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.8.0
> 2016-02-15 10:46:20,974 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> 2016-02-15 10:46:20,977 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage
> 2016-02-15 10:46:20,978 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm
> 2016-02-15 10:46:20,988 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> 2016-02-15 10:46:20,999 INFO [cloud.agent.AgentShell] (main:null) (logid:) Preferring IPv4 address family for agent connection
> 2016-02-15 10:46:21,095 INFO [cloud.agent.Agent] (main:null) (logid:) id is 17
> 2016-02-15 10:46:21,116 INFO [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one
> 2016-02-15 10:46:21,373 INFO [org.reflections.Reflections] (main:null) (logid:) Reflections took 66 ms to scan 1 urls, producing 7 keys and 10 values
> 2016-02-15 10:46:21,514 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) No libvirt.vif.driver specified. Defaults to BridgeVifDriver.
> 2016-02-15 10:46:21,543 INFO [cloud.agent.Agent] (main:null) (logid:) Agent [id = 17 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 : host = 192.168.0.72 : port = 8250
> 2016-02-15 10:46:21,549 INFO [utils.nio.NioClient] (main:null) (logid:) Connecting to 192.168.0.72:8250
> 2016-02-15 10:46:21,609 ERROR [utils.nio.NioConnection] (main:null) (logid:) Unable to initialize the threads.
> java.io.IOException: Connection closed with -1 on reading size.
> at com.cloud.utils.nio.Link.doHandshake(Link.java:513)
> at com.cloud.utils.nio.NioClient.init(NioClient.java:80)
> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:228)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.start(AgentShell.java:461)
> at com.cloud.agent.AgentShell.main(AgentShell.java:496)
> 2016-02-15 10:46:21,611 INFO [utils.exception.CSExceptionErrorCode] (main:null) (logid:) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
> 2016-02-15 10:46:21,612 ERROR [cloud.agent.AgentShell] (main:null) (logid:) Unable to start agent:
> com.cloud.utils.exception.CloudRuntimeException: Unable to start the connection!
> at com.cloud.agent.Agent.start(Agent.java:230)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.start(AgentShell.java:461)
> at com.cloud.agent.AgentShell.main(AgentShell.java:496)
> Caused by: com.cloud.utils.exception.NioConnectionException: Connection closed with -1 on reading size.
> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:94)
> at com.cloud.agent.Agent.start(Agent.java:228)
> ... 5 more
> Caused by: java.io.IOException: Connection closed with -1 on reading size.
> at com.cloud.utils.nio.Link.doHandshake(Link.java:513)
> at com.cloud.utils.nio.NioClient.init(NioClient.java:80)
> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
> ... 6 more
> {code}
> cloudstack-agent on the kvm host and the cloudstack-management on the cloudstack server is running. Everything on version 4.8.0.
> Maybe there are some bugs or did I something wrong?
> Greetings J. Fettig
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)