You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by F5 <f5...@gmail.com> on 2020/06/23 12:57:10 UTC
Help CloudStack 4.1.3.1
Hello
I have a problem with my ACS 4.13.1
When restarting cloudstack-management it simply takes 1 hour for the agent to start.
In the Agent Logs it appears as connected as soon as it starts.
But on the host you get Alert.
Has anyone been through this and know the solution?
__________________________________________________________
KVM Host Information
qemu
qemu-kvm-common-1.5.3-173.el7_8.3.x86_64
qemu-kvm-1.5.3-173.el7_8.3.x86_64
qemu-img-1.5.3-173.el7_8.3.x86_64
libvirt
libvirt-4.5.0-33.el7_8.1.x86_64
Java
java-1.8.0
________________________________________________________________
Logs Agent:
2020-06-23 09:40:56,591 INFO [cloud.agent.Agent] (Agent Timer:null) (logid:) Reconnecting to host:172.26.0.209
2020-06-23 09:40:56,592 INFO [utils.nio.NioClient] (Agent Timer:null) (logid:) Connecting to 172.26.0.209:8250
2020-06-23 09:40:56,593 INFO [utils.nio.Link] (Agent Timer:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2020-06-23 09:40:56,827 INFO [utils.nio.NioClient] (Agent Timer:null) (logid:) SSL: Handshake done
2020-06-23 09:40:56,827 INFO [utils.nio.NioClient] (Agent Timer:null) (logid:) Connected to 172.26.0.209:8250
2020-06-23 09:40:56,873 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool 1a26d75e-d63a-4255-961a-8878f526ffc5 (Filesystem) in libvirt
2020-06-23 09:40:56,873 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2020-06-23 09:40:56,874 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Found existing defined storage pool 1a26d75e-d63a-4255-961a-8878f526ffc5, using it.
2020-06-23 09:40:56,874 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Trying to fetch storage pool 1a26d75e-d63a-4255-961a-8878f526ffc5 from libvirt
2020-06-23 09:41:01,828 INFO [cloud.agent.Agent] (Agent Timer:null) (logid:) Connected to the host: 172.26.0.209
2020-06-23 09:46:56,897 INFO [cloud.agent.Agent] (Agent Timer:null) (logid:) The startup command is now cancelled
2020-06-23 09:46:56,897 INFO [cloud.agent.Agent] (Agent Timer:null) (logid:) Lost connection to host: 172.26.0.209. Attempting reconnection while we still have 0 commands in progress.
2020-06-23 09:46:56,898 INFO [utils.nio.NioClient] (Agent Timer:null) (logid:) NioClient connection closed
2020-06-23 09:46:56,898 INFO [cloud.agent.Agent] (Agent Timer:null) (logid:) Reconnecting to host:172.26.0.209
2020-06-23 09:46:56,898 INFO [utils.nio.NioClient] (Agent Timer:null) (logid:) Connecting to 172.26.0.209:8250
2020-06-23 09:46:56,899 INFO [utils.nio.Link] (Agent Timer:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2020-06-23 09:46:57,107 INFO [utils.nio.NioClient] (Agent Timer:null) (logid:) SSL: Handshake done
2020-06-23 09:46:57,107 INFO [utils.nio.NioClient] (Agent Timer:null) (logid:) Connected to 172.26.0.209:8250
2020-06-23 09:46:57,151 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool 1a26d75e-d63a-4255-961a-8878f526ffc5 (Filesystem) in libvirt
2020-06-23 09:46:57,151 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2020-06-23 09:46:57,152 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Found existing defined storage pool 1a26d75e-d63a-4255-961a-8878f526ffc5, using it.
2020-06-23 09:46:57,152 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Trying to fetch storage pool 1a26d75e-d63a-4255-961a-8878f526ffc5 from libvirt
2020-06-23 09:47:02,108 INFO [cloud.agent.Agent] (Agent Timer:null) (logid:) Connected to the host: 172.26.0.209
2020-06-23 09:52:57,175 INFO [cloud.agent.Agent] (Agent Timer:null) (logid:) The startup command is now cancelled
2020-06-23 09:52:57,175 INFO [cloud.agent.Agent] (Agent Timer:null) (logid:) Lost connection to host: 172.26.0.209. Attempting reconnection while we still have 0 commands in progress.
2020-06-23 09:52:57,176 INFO [utils.nio.NioClient] (Agent Timer:null) (logid:) NioClient connection closed
2020-06-23 09:52:57,176 INFO [cloud.agent.Agent] (Agent Timer:null) (logid:) Reconnecting to host:172.26.0.209
2020-06-23 09:52:57,176 INFO [utils.nio.NioClient] (Agent Timer:null) (logid:) Connecting to 172.26.0.209:8250
2020-06-23 09:52:57,178 INFO [utils.nio.Link] (Agent Timer:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2020-06-23 09:52:57,488 INFO [utils.nio.NioClient] (Agent Timer:null) (logid:) SSL: Handshake done
2020-06-23 09:52:57,488 INFO [utils.nio.NioClient] (Agent Timer:null) (logid:) Connected to 172.26.0.209:8250
2020-06-23 09:52:57,535 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool 1a26d75e-d63a-4255-961a-8878f526ffc5 (Filesystem) in libvirt
2020-06-23 09:52:57,539 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2020-06-23 09:52:57,540 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Found existing defined storage pool 1a26d75e-d63a-4255-961a-8878f526ffc5, using it.
2020-06-23 09:52:57,540 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Trying to fetch storage pool 1a26d75e-d63a-4255-961a-8878f526ffc5 from libvirt
2020-06-23 09:53:02,489 INFO [cloud.agent.Agent] (Agent Timer:null) (logid:) Connected to the host: 172.26.0.209
Re: Help CloudStack 4.1.3.1
Posted by "Riepl, Gregor (SWISS TXT)" <Gr...@swisstxt.ch>.
Seems to be this line: https://github.com/apache/cloudstack/blob/4.13.1.0/server/src/main/java/com/cloud/resource/ResourceManagerImpl.java#L1742
Judging from the surrounding code, the NPE happens at the .getName() call.
Could it be that you haven't got a datacenter assigned to the pod?
________________________________
From: F5 <f5...@gmail.com>
Sent: 23 June 2020 17:34
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Help CloudStack 4.1.3.1
Logs Management :
2020-06-23 12:31:21,977 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-33:ctx-fcb0622c) (logid:152ccaa9) Failed to handle host connection:
java.lang.NullPointerException
at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1742)
at com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:2172)
at sun.reflect.GeneratedMethodAccessor191.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy179.createHostVOForConnectedAgent(Unknown Source)
at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1090)
at com.cloud.agent.manager.AgentManagerImpl.access$000(AgentManagerImpl.java:126)
at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1187)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-06-23 12:31:21,977 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-33:ctx-fcb0622c) (logid:152ccaa9) Failed to send ready command:java.nio.channels.ClosedChannelException
2020-06-23 12:31:21,977 WARN [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-33:ctx-fcb0622c) (logid:152ccaa9) Unable to create attache for agent: Seq -1-0: { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":1,"cpus":4,"speed":3100,"memory":7115526144,"dom0MinMemory":1073741824,"poolSync":false,"supportsClonedVolumes":false,"caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"Host.OS.Kernel.Version":"3.10.0-1062.12.1.el7.x86_64","com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS.Version":"7.8.2003","secured":"false","Host.OS":"CentOS"},"hostTags":[],"groupDetails":{},"type":"Routing","dataCenter":"default","pod":"default","cluster":"default","guid":"4d7578e0-ddf1-4504-ba0c-e8e4a95c8caa-LibvirtComputingResource","name":"kvm-lab-01","version":"4.13.1.0","iqn":"iqn.1994-05.com.redhat:87b9f1e86e51","publicIpAddress":"172.26.0.222","publicNetmask":"255.255.255.0","publicMacAddress":"5c:f3:fc:fd:8f:ed","privateIpAddress":"172.26.0.222","privateMacAddress":"5c:f3:fc:fd:8f:ed","privateNetmask":"255.255.255.0","storageIpAddress":"172.26.0.222","storageNetmask":"255.255.255.0","storageMacAddress":"5c:f3:fc:fd:8f:ed","resourceName":"LibvirtComputingResource","gatewayIpAddress":"172.26.0.1","msHostList":"172.26.0.209","wait":0}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":0,"poolInfo":{"uuid":"fdf04200-8e76-4fd6-9045-18c1f7b8924a","host":"172.26.0.222","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":53660876800,"availableBytes":47240916992},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"default","pod":"default","guid":"4d7578e0-ddf1-4504-ba0c-e8e4a95c8caa-LibvirtComputingResource","name":"kvm-lab-01","version":"4.13.1.0","resourceName":"LibvirtComputingResource","msHostList":"172.26.0.209","wait":0}}] }
2020-06-23 12:31:22,392 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-58-thread-1:null) (logid:) A client/agent attempting connection from address=172.26.0.222 has presented these certificate(s):
Certificate [1] :
Serial: 6fd751bd03918fb6
Not Before:Mon Jun 22 20:37:38 BRT 2020
Not After:Wed Jun 23 08:37:38 BRT 2021
Signature Algorithm:SHA256withRSA
Version:3
Subject DN:C=cloudstack, O=cloudstack, OU=cloudstack, CN=kvm-lab-01
Issuer DN:CN=ca.cloudstack.apache.org
Alternative Names:[[7, fe80:0:0:0:5ef3:fcff:fefd:8fed], [7, 2804:eb0:0:5:5ef3:fcff:fefd:8fee], [7, fe80:0:0:0:5ef3:fcff:fefd:8fee], [7, 172.26.0.222], [7, fe80:0:0:0:2cc7:7ff:fe95:9f51], [2, 172.26.0.222]]
Certificate [2] :
Serial: 69c381745233fbac
Not Before:Tue May 19 08:44:10 BRT 2020
Not After:Thu May 12 20:44:10 BRT 2050
Signature Algorithm:SHA256withRSA
Version:3
Subject DN:CN=ca.cloudstack.apache.org
Issuer DN:CN=ca.cloudstack.apache.org
Alternative Names:null
Re: Help CloudStack 4.1.3.1
Posted by F5 <f5...@gmail.com>.
Logs Management :
2020-06-23 12:31:21,977 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-33:ctx-fcb0622c) (logid:152ccaa9) Failed to handle host connection:
java.lang.NullPointerException
at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1742)
at com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:2172)
at sun.reflect.GeneratedMethodAccessor191.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy179.createHostVOForConnectedAgent(Unknown Source)
at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1090)
at com.cloud.agent.manager.AgentManagerImpl.access$000(AgentManagerImpl.java:126)
at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1187)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-06-23 12:31:21,977 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-33:ctx-fcb0622c) (logid:152ccaa9) Failed to send ready command:java.nio.channels.ClosedChannelException
2020-06-23 12:31:21,977 WARN [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-33:ctx-fcb0622c) (logid:152ccaa9) Unable to create attache for agent: Seq -1-0: { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":1,"cpus":4,"speed":3100,"memory":7115526144,"dom0MinMemory":1073741824,"poolSync":false,"supportsClonedVolumes":false,"caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"Host.OS.Kernel.Version":"3.10.0-1062.12.1.el7.x86_64","com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS.Version":"7.8.2003","secured":"false","Host.OS":"CentOS"},"hostTags":[],"groupDetails":{},"type":"Routing","dataCenter":"default","pod":"default","cluster":"default","guid":"4d7578e0-ddf1-4504-ba0c-e8e4a95c8caa-LibvirtComputingResource","name":"kvm-lab-01","version":"4.13.1.0","iqn":"iqn.1994-05.com.redhat:87b9f1e86e51","publicIpAddress":"172.26.0.222","publicNetmask":"255.255.255.0","publicMacAddress":"5c:f3:fc:fd:8f:ed","privateIpAddress":"172.26.0.222","privateMacAddress":"5c:f3:fc:fd:8f:ed","privateNetmask":"255.255.255.0","storageIpAddress":"172.26.0.222","storageNetmask":"255.255.255.0","storageMacAddress":"5c:f3:fc:fd:8f:ed","resourceName":"LibvirtComputingResource","gatewayIpAddress":"172.26.0.1","msHostList":"172.26.0.209","wait":0}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":0,"poolInfo":{"uuid":"fdf04200-8e76-4fd6-9045-18c1f7b8924a","host":"172.26.0.222","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":53660876800,"availableBytes":47240916992},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"default","pod":"default","guid":"4d7578e0-ddf1-4504-ba0c-e8e4a95c8caa-LibvirtComputingResource","name":"kvm-lab-01","version":"4.13.1.0","resourceName":"LibvirtComputingResource","msHostList":"172.26.0.209","wait":0}}] }
2020-06-23 12:31:22,392 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-58-thread-1:null) (logid:) A client/agent attempting connection from address=172.26.0.222 has presented these certificate(s):
Certificate [1] :
Serial: 6fd751bd03918fb6
Not Before:Mon Jun 22 20:37:38 BRT 2020
Not After:Wed Jun 23 08:37:38 BRT 2021
Signature Algorithm:SHA256withRSA
Version:3
Subject DN:C=cloudstack, O=cloudstack, OU=cloudstack, CN=kvm-lab-01
Issuer DN:CN=ca.cloudstack.apache.org
Alternative Names:[[7, fe80:0:0:0:5ef3:fcff:fefd:8fed], [7, 2804:eb0:0:5:5ef3:fcff:fefd:8fee], [7, fe80:0:0:0:5ef3:fcff:fefd:8fee], [7, 172.26.0.222], [7, fe80:0:0:0:2cc7:7ff:fe95:9f51], [2, 172.26.0.222]]
Certificate [2] :
Serial: 69c381745233fbac
Not Before:Tue May 19 08:44:10 BRT 2020
Not After:Thu May 12 20:44:10 BRT 2050
Signature Algorithm:SHA256withRSA
Version:3
Subject DN:CN=ca.cloudstack.apache.org
Issuer DN:CN=ca.cloudstack.apache.org
Alternative Names:null