You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@cloudstack.apache.org by Jeremy Hansen <je...@skidrow.la.INVALID> on 2021/12/13 05:45:31 UTC

Issues with libvirt after upgrading to RockyLinux 8.5

Testing on an unused compute node, I tested upgrading to RockyLinux 8.5.

I’m running Cloudstack 4.15.0.0.

Trying to bring up cloudstack agent, I’m seeing some issues communicating with libvirt:

2021-12-12 21:25:03,992 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool d1b1e853-1c30-473d-badc-6c30318aa5b0 (Filesystem) in libvirt
2021-12-12 21:25:03,997 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2021-12-12 21:25:04,000 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Found existing defined storage pool d1b1e853-1c30-473d-badc-6c30318aa5b0, using it.

Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data: Input/output error
Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data: Input/output error
Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data: Input/output error
Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data: Input/output error
Dec 12 21:24:26 cm02 libvirtd[269244]: Cannot recv data: Input/output error

Libvirt version is: libvirt-6.0.0-37.module+el8.5.0+670+c4aa478c.x86_64

Functional hosts that have yet to be upgraded are using:

libvirt-6.0.0-35.module_el8.4.0+783+f8734d30.x86_64

My libvirtd.conf looks like this:

listen_tcp=0
listen_tls=1
tcp_port="16509"
auth_tcp="none"
mdns_adv = 0
key_file="/etc/pki/libvirt/private/serverkey.pem"
cert_file="/etc/pki/libvirt/servercert.pem"
ca_file="/etc/pki/CA/cacert.pem"
tls_port="16514"
auth_tls=“none"

2021-12-12 21:43:42,841 ERROR [kvm.resource.LibvirtComputingResource] (main:null) (logid:) uefi properties file not found due to: Unable to find file uefi.properties.
2021-12-12 21:43:42,901 INFO [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one
2021-12-12 21:43:43,127 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) No libvirt.vif.driver specified. Defaults to BridgeVifDriver.
2021-12-12 21:43:43,296 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) iscsi session clean up is disabled
2021-12-12 21:43:43,312 INFO [cloud.agent.Agent] (main:null) (logid:) Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 : host = 192.168.30.59 : port = 8250
2021-12-12 21:43:43,321 INFO [utils.nio.NioClient] (main:null) (logid:) Connecting to 192.168.30.59:8250
2021-12-12 21:43:43,325 INFO [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:) SSL: Handshake done
2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:) Connected to 192.168.30.59:8250
2021-12-12 21:43:43,925 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
2021-12-12 21:43:43,929 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2021-12-12 21:43:43,932 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Found existing defined storage pool 18796842-a137-475d-9799-9874240e3c0c, using it.
2021-12-12 21:43:43,933 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Trying to fetch storage pool 18796842-a137-475d-9799-9874240e3c0c from libvirt
2021-12-12 21:43:43,985 INFO [cloud.serializer.GsonHelper] (Agent-Handler-1:null) (logid:) Default Builder inited.
2021-12-12 21:43:44,020 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Proccess agent startup answer, agent id = 0
2021-12-12 21:43:44,022 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Set agent id 0
2021-12-12 21:43:44,028 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Startup Response Received: agent id = 0
2021-12-12 21:43:44,031 INFO [cloud.agent.Agent] (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
2021-12-12 21:43:55,682 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2021-12-12 21:43:55,688 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.15.0.0
2021-12-12 21:43:55,690 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2021-12-12 21:43:55,709 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage
2021-12-12 21:43:55,711 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm
2021-12-12 21:43:55,714 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2021-12-12 21:43:55,728 INFO [cloud.agent.AgentShell] (main:null) (logid:) Using default Java settings for IPv6 preference for agent connection
2021-12-12 21:43:55,840 INFO [cloud.agent.Agent] (main:null) (logid:) id is 0
2021-12-12 21:43:55,853 ERROR [kvm.resource.LibvirtComputingResource] (main:null) (logid:) uefi properties file not found due to: Unable to find file uefi.properties.
2021-12-12 21:43:55,909 INFO [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one
2021-12-12 21:43:56,145 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) No libvirt.vif.driver specified. Defaults to BridgeVifDriver.
2021-12-12 21:43:56,307 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) iscsi session clean up is disabled
2021-12-12 21:43:56,322 INFO [cloud.agent.Agent] (main:null) (logid:) Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 : host = 192.168.30.59 : port = 8250
2021-12-12 21:43:56,331 INFO [utils.nio.NioClient] (main:null) (logid:) Connecting to 192.168.30.59:8250
2021-12-12 21:43:56,335 INFO [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:) SSL: Handshake done
2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:) Connected to 192.168.30.59:8250
2021-12-12 21:43:56,941 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
2021-12-12 21:43:56,945 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Found existing defined storage pool 18796842-a137-475d-9799-9874240e3c0c, using it.
2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Trying to fetch storage pool 18796842-a137-475d-9799-9874240e3c0c from libvirt
2021-12-12 21:43:56,998 INFO [cloud.serializer.GsonHelper] (Agent-Handler-1:null) (logid:) Default Builder inited.
2021-12-12 21:43:57,031 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Proccess agent startup answer, agent id = 0
2021-12-12 21:43:57,033 INFO [cloud.agent.Agent] (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
2021-12-12 21:43:57,033 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Set agent id 0
2021-12-12 21:43:57,040 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Startup Response Received: agent id = 0

Any ideas? I’m working toward upgrading the entire cluster along with Cloudstack itself.

-jeremy

Re: Issues with libvirt after upgrading to RockyLinux 8.5

Posted by Jeremy Hansen <je...@skidrow.la.INVALID>.

Faking the releases file helped me get past this.

Thanks

> On Thursday, Dec 16, 2021 at 1:32 AM, Jeremy Hansen <jeremy@skidrow.la (mailto:jeremy@skidrow.la)> wrote:
> Thank you. I thought about trying that.
>
> If I upgrade everything to Rocky, and I’m on 4.16, should I expect it to work? Will I have to re-add compute nodes to update the db with Rocky as the OS?
>
> I expect a lot of people might want to transition to Rocky since CentOS is going EOL after this month I believe. At this point, Rocky and CentOS are mostly the same thing. This complicates things a bit.
>
> Thanks
> -jeremy
>
>
>
> > On Thursday, Dec 16, 2021 at 1:09 AM, Andrija Panic <andrija.panic@gmail.com (mailto:andrija.panic@gmail.com)> wrote:
> > Can't add host: 192.168.30.54 with hostOS: Rocky into a cluster,in which
> > there are CentOS hosts added
> >
> >
> > Try changing the content of the /etc/*release file - to match the file of
> > the CentOS nodes - and then play with restarting agent, and observe if you
> > would hit the same error or not - I've seen this issue (RHEL inside CentOS
> > cluster, due to bad /etc/rhel-release file content), but I've not tested if
> > this "fix" works or not.
> >
> > Best,
> >
> > On Thu, 16 Dec 2021 at 08:39, Jeremy Hansen <je...@skidrow.la.invalid>
> > wrote:
> >
> > > But if I convert all the hosts to Rocky and upgrade to 4.16, I should be
> > > ok?
> > >
> > > Thanks
> > >
> > >
> > >
> > > On Wednesday, Dec 15, 2021 at 11:17 PM, Slavka Peleva <
> > > slavkap@storpool.com.INVALID> wrote:
> > > Sorry, I didn't pay attention to your CS version. After the upgrade, I
> > > think you will have the same problem. Because in the DB, there is
> > > information about host/hosts on this cluster that is/are with CentOS.
> > >
> > > Best regards,
> > > Slavka
> > >
> > > On Thu, Dec 16, 2021 at 8:49 AM Jeremy Hansen <je...@skidrow.la.invalid>
> > > wrote:
> > >
> > > I noticed in the compatibility matrix that Rocky isn’t supported until
> > > 4.16.0.0. If I upgrade Cloudstack first, would this help or is it still
> > > going to complain about the centos/rocky mix? If I convert all my existing
> > > nodes to Rocky, which is the plan anyway, will this go away? Shouldn’t
> > > CentOS and Rocky be considered that same thing… sort of…?
> > >
> > > Thanks
> > > -jeremy
> > >
> > >
> > >
> > >
> > > On Wednesday, Dec 15, 2021 at 10:43 PM, Slavka Peleva <
> > > slavkap@storpool.com.INVALID> wrote:
> > > Hi Jeremy,
> > >
> > > It will help if you have another cluster for Rocky Linux. Hosts need to be
> > > of the same OS, it's not possible to mix OSes in the same cluster.
> > >
> > > Best regards,
> > > Slavka
> > >
> > > On Thu, Dec 16, 2021 at 4:08 AM Jeremy Hansen <je...@skidrow.la.invalid>
> > > wrote:
> > >
> > > Any tips on how I would troubleshoot this? I’ve tried downgrading libvirt
> > > and qemu and ca-certificates to the same version as the other functional
> > > nodes. That didn’t seem to help. This is obviously an ssl issue but I
> > > don’t really know what to do about it.
> > >
> > > 2021-12-15 18:04:14,438 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) Agent started
> > > 2021-12-15 18:04:14,444 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) Implementation Version is 4.15.0.0
> > > 2021-12-15 18:04:14,447 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> > > 2021-12-15 18:04:14,466 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) Defaulting to using properties file for storage
> > > 2021-12-15 18:04:14,467 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) Defaulting to the constant time backoff algorithm
> > > 2021-12-15 18:04:14,471 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> > > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> > > 2021-12-15 18:04:14,485 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) Using default Java settings for IPv6 preference for agent
> > > connection
> > > 2021-12-15 18:04:14,592 INFO [cloud.agent.Agent] (main:null) (logid:) id
> > > is 0
> > > 2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource]
> > > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > > file uefi.properties.
> > > 2021-12-15 18:04:14,663 INFO [kvm.resource.LibvirtConnection] (main:null)
> > > (logid:) No existing libvirtd connection found. Opening a new one
> > > 2021-12-15 18:04:14,890 INFO [kvm.resource.LibvirtComputingResource]
> > > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > > BridgeVifDriver.
> > > 2021-12-15 18:04:15,086 INFO [kvm.resource.LibvirtComputingResource]
> > > (main:null) (logid:) iscsi session clean up is disabled
> > > 2021-12-15 18:04:15,129 INFO [cloud.agent.Agent] (main:null) (logid:)
> > > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > > workers = 5 : host = 192.168.30.59 : port = 8250
> > > 2021-12-15 18:04:15,139 INFO [utils.nio.NioClient] (main:null) (logid:)
> > > Connecting to 192.168.30.59:8250
> > > 2021-12-15 18:04:15,153 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > > file found: /etc/cloudstack/agent/agent.properties
> > > 2021-12-15 18:04:15,919 INFO [utils.nio.NioClient] (main:null) (logid:)
> > > SSL: Handshake done
> > > 2021-12-15 18:04:15,920 INFO [utils.nio.NioClient] (main:null) (logid:)
> > > Connected to 192.168.30.59:8250
> > > 2021-12-15 18:04:16,057 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > > 2021-12-15 18:04:16,062 ERROR [kvm.resource.LibvirtConnection]
> > > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> > > invalid
> > > connection pointer in virConnectGetVersion
> > > 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > > 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > > 2021-12-15 18:04:16,151 INFO [cloud.serializer.GsonHelper]
> > > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > > 2021-12-15 18:04:16,272 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > > (logid:) Proccess agent startup answer, agent id = 0
> > > 2021-12-15 18:04:16,273 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > > (logid:) Set agent id 0
> > > 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > > (logid:) Startup Response Received: agent id = 0
> > > 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent]
> > > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> > >
> > > I also noticed this:
> > >
> > > 2021-12-15 18:05:34,542 DEBUG [c.c.a.m.AgentManagerImpl]
> > > (AgentConnectTaskPool-19617:ctx-e0dfdb18) (logid:d3930423) Failed to
> > > handle
> > > host connection:
> > > java.lang.IllegalArgumentException: Can't add host: 192.168.30.54 with
> > > hostOS: Rocky into a cluster,in which there are CentOS hosts added
> > > at
> > > com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.createHostVOForConnectedAgent(LibvirtServerDiscoverer.java:456)
> > >
> > >
> > > at
> > > com.cloud.resource.ResourceManagerImpl.dispatchToStateAdapters(ResourceManagerImpl.java:1719)
> > >
> > >
> > > at
> > > com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1946)
> > >
> > >
> > > at
> > > com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:2278)
> > >
> > >
> > > at jdk.internal.reflect.GeneratedMethodAccessor510.invoke(Unknown Source)
> > > at
> > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >
> > >
> > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > at
> > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> > >
> > >
> > > at
> > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> > >
> > >
> > > at
> > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > >
> > >
> > > at
> > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
> > >
> > >
> > > at
> > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> > >
> > >
> > > at
> > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> > >
> > >
> > > at com.sun.proxy.$Proxy188.createHostVOForConnectedAgent(Unknown Source)
> > > at
> > > com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1097)
> > >
> > >
> > > at
> > > com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1194)
> > >
> > >
> > > at
> > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > >
> > >
> > > at
> > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > >
> > >
> > > at
> > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > >
> > >
> > > at
> > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > >
> > >
> > > at
> > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > >
> > >
> > > at
> > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > >
> > >
> > > at
> > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > >
> > >
> > > at java.base/java.lang.Thread.run(Thread.java:829)
> > >
> > > The fact that this host has been converted to Rocky Linux is causing an
> > > issue? What’s the work around for this?
> > >
> > > Thanks
> > > -jeremy
> > >
> > >
> > >
> > > On Monday, Dec 13, 2021 at 12:19 AM, Jeremy Hansen <je...@skidrow.la>
> > > wrote:
> > > It doesn’t error out but there is purposely no VMs running on these hosts
> > > to test the upgrade of the underlying distro before having to stop any
> > > active VMs.
> > >
> > > [jeremy@cm02 ~]$ sudo virsh list
> > > Id Name State
> > > --------------------
> > >
> > > [jeremy@cm02 ~]$
> > >
> > >
> > > -jeremy
> > >
> > >
> > >
> > >
> > > On Monday, Dec 13, 2021 at 12:06 AM, Wei ZHOU <us...@gmail.com>
> > > wrote:
> > > Hi,
> > >
> > > Do virsh commands e.g. "virsh list" work ?
> > >
> > > -Wei
> > >
> > > On Mon, 13 Dec 2021 at 06:46, Jeremy Hansen <je...@skidrow.la.invalid>
> > > wrote:
> > >
> > > Testing on an unused compute node, I tested upgrading to RockyLinux 8.5.
> > >
> > > I’m running Cloudstack 4.15.0.0.
> > >
> > > Trying to bring up cloudstack agent, I’m seeing some issues communicating
> > > with libvirt:
> > >
> > > 2021-12-12 21:25:03,992 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > > d1b1e853-1c30-473d-badc-6c30318aa5b0 (Filesystem) in libvirt
> > > 2021-12-12 21:25:03,997 ERROR [kvm.resource.LibvirtConnection]
> > > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> > > invalid
> > > connection pointer in virConnectGetVersion
> > > 2021-12-12 21:25:04,000 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > > d1b1e853-1c30-473d-badc-6c30318aa5b0, using it.
> > >
> > > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> > > Input/output error
> > > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> > > Input/output error
> > > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> > > Input/output error
> > > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> > > Input/output error
> > > Dec 12 21:24:26 cm02 libvirtd[269244]: Cannot recv data: Input/output
> > > error
> > >
> > > Libvirt version is: libvirt-6.0.0-37.module+el8.5.0+670+c4aa478c.x86_64
> > >
> > > Functional hosts that have yet to be upgraded are using:
> > >
> > > libvirt-6.0.0-35.module_el8.4.0+783+f8734d30.x86_64
> > >
> > >
> > > My libvirtd.conf looks like this:
> > >
> > > listen_tcp=0
> > > listen_tls=1
> > > tcp_port="16509"
> > > auth_tcp="none"
> > > mdns_adv = 0
> > > key_file="/etc/pki/libvirt/private/serverkey.pem"
> > > cert_file="/etc/pki/libvirt/servercert.pem"
> > > ca_file="/etc/pki/CA/cacert.pem"
> > > tls_port="16514"
> > > auth_tls=“none"
> > >
> > > 2021-12-12 21:43:42,841 ERROR [kvm.resource.LibvirtComputingResource]
> > > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > > file uefi.properties.
> > > 2021-12-12 21:43:42,901 INFO [kvm.resource.LibvirtConnection] (main:null)
> > > (logid:) No existing libvirtd connection found. Opening a new one
> > > 2021-12-12 21:43:43,127 INFO [kvm.resource.LibvirtComputingResource]
> > > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > > BridgeVifDriver.
> > > 2021-12-12 21:43:43,296 INFO [kvm.resource.LibvirtComputingResource]
> > > (main:null) (logid:) iscsi session clean up is disabled
> > > 2021-12-12 21:43:43,312 INFO [cloud.agent.Agent] (main:null) (logid:)
> > > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > > workers = 5 : host = 192.168.30.59 : port = 8250
> > > 2021-12-12 21:43:43,321 INFO [utils.nio.NioClient] (main:null) (logid:)
> > > Connecting to 192.168.30.59:8250
> > > 2021-12-12 21:43:43,325 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > > file found: /etc/cloudstack/agent/agent.properties
> > > 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> > > SSL: Handshake done
> > > 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> > > Connected to 192.168.30.59:8250
> > > 2021-12-12 21:43:43,925 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > > 2021-12-12 21:43:43,929 ERROR [kvm.resource.LibvirtConnection]
> > > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> > > invalid
> > > connection pointer in virConnectGetVersion
> > > 2021-12-12 21:43:43,932 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > > 2021-12-12 21:43:43,933 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > > 2021-12-12 21:43:43,985 INFO [cloud.serializer.GsonHelper]
> > > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > > 2021-12-12 21:43:44,020 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > > (logid:) Proccess agent startup answer, agent id = 0
> > > 2021-12-12 21:43:44,022 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > > (logid:) Set agent id 0
> > > 2021-12-12 21:43:44,028 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > > (logid:) Startup Response Received: agent id = 0
> > > 2021-12-12 21:43:44,031 INFO [cloud.agent.Agent]
> > > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> > > 2021-12-12 21:43:55,682 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) Agent started
> > > 2021-12-12 21:43:55,688 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) Implementation Version is 4.15.0.0
> > > 2021-12-12 21:43:55,690 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> > > 2021-12-12 21:43:55,709 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) Defaulting to using properties file for storage
> > > 2021-12-12 21:43:55,711 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) Defaulting to the constant time backoff algorithm
> > > 2021-12-12 21:43:55,714 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> > > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> > > 2021-12-12 21:43:55,728 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) Using default Java settings for IPv6 preference for agent
> > > connection
> > > 2021-12-12 21:43:55,840 INFO [cloud.agent.Agent] (main:null) (logid:) id
> > > is 0
> > > 2021-12-12 21:43:55,853 ERROR [kvm.resource.LibvirtComputingResource]
> > > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > > file uefi.properties.
> > > 2021-12-12 21:43:55,909 INFO [kvm.resource.LibvirtConnection] (main:null)
> > > (logid:) No existing libvirtd connection found. Opening a new one
> > > 2021-12-12 21:43:56,145 INFO [kvm.resource.LibvirtComputingResource]
> > > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > > BridgeVifDriver.
> > > 2021-12-12 21:43:56,307 INFO [kvm.resource.LibvirtComputingResource]
> > > (main:null) (logid:) iscsi session clean up is disabled
> > > 2021-12-12 21:43:56,322 INFO [cloud.agent.Agent] (main:null) (logid:)
> > > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > > workers = 5 : host = 192.168.30.59 : port = 8250
> > > 2021-12-12 21:43:56,331 INFO [utils.nio.NioClient] (main:null) (logid:)
> > > Connecting to 192.168.30.59:8250
> > > 2021-12-12 21:43:56,335 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > > file found: /etc/cloudstack/agent/agent.properties
> > > 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> > > SSL: Handshake done
> > > 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> > > Connected to 192.168.30.59:8250
> > > 2021-12-12 21:43:56,941 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > > 2021-12-12 21:43:56,945 ERROR [kvm.resource.LibvirtConnection]
> > > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> > > invalid
> > > connection pointer in virConnectGetVersion
> > > 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > > 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > > 2021-12-12 21:43:56,998 INFO [cloud.serializer.GsonHelper]
> > > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > > 2021-12-12 21:43:57,031 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > > (logid:) Proccess agent startup answer, agent id = 0
> > > 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent]
> > > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> > > 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > > (logid:) Set agent id 0
> > > 2021-12-12 21:43:57,040 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > > (logid:) Startup Response Received: agent id = 0
> > >
> > > Any ideas? I’m working toward upgrading the entire cluster along with
> > > Cloudstack itself.
> > >
> > > -jeremy
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> > --
> >
> > Andrija Panić

Re: Issues with libvirt after upgrading to RockyLinux 8.5

Posted by Jeremy Hansen <je...@skidrow.la.INVALID>.

Thank you. I thought about trying that.

If I upgrade everything to Rocky, and I’m on 4.16, should I expect it to work? Will I have to re-add compute nodes to update the db with Rocky as the OS?

I expect a lot of people might want to transition to Rocky since CentOS is going EOL after this month I believe. At this point, Rocky and CentOS are mostly the same thing. This complicates things a bit.

Thanks
-jeremy

> On Thursday, Dec 16, 2021 at 1:09 AM, Andrija Panic <andrija.panic@gmail.com (mailto:andrija.panic@gmail.com)> wrote:
> Can't add host: 192.168.30.54 with hostOS: Rocky into a cluster,in which
> there are CentOS hosts added
>
>
> Try changing the content of the /etc/*release file - to match the file of
> the CentOS nodes - and then play with restarting agent, and observe if you
> would hit the same error or not - I've seen this issue (RHEL inside CentOS
> cluster, due to bad /etc/rhel-release file content), but I've not tested if
> this "fix" works or not.
>
> Best,
>
> On Thu, 16 Dec 2021 at 08:39, Jeremy Hansen <je...@skidrow.la.invalid>
> wrote:
>
> > But if I convert all the hosts to Rocky and upgrade to 4.16, I should be
> > ok?
> >
> > Thanks
> >
> >
> >
> > On Wednesday, Dec 15, 2021 at 11:17 PM, Slavka Peleva <
> > slavkap@storpool.com.INVALID> wrote:
> > Sorry, I didn't pay attention to your CS version. After the upgrade, I
> > think you will have the same problem. Because in the DB, there is
> > information about host/hosts on this cluster that is/are with CentOS.
> >
> > Best regards,
> > Slavka
> >
> > On Thu, Dec 16, 2021 at 8:49 AM Jeremy Hansen <je...@skidrow.la.invalid>
> > wrote:
> >
> > I noticed in the compatibility matrix that Rocky isn’t supported until
> > 4.16.0.0. If I upgrade Cloudstack first, would this help or is it still
> > going to complain about the centos/rocky mix? If I convert all my existing
> > nodes to Rocky, which is the plan anyway, will this go away? Shouldn’t
> > CentOS and Rocky be considered that same thing… sort of…?
> >
> > Thanks
> > -jeremy
> >
> >
> >
> >
> > On Wednesday, Dec 15, 2021 at 10:43 PM, Slavka Peleva <
> > slavkap@storpool.com.INVALID> wrote:
> > Hi Jeremy,
> >
> > It will help if you have another cluster for Rocky Linux. Hosts need to be
> > of the same OS, it's not possible to mix OSes in the same cluster.
> >
> > Best regards,
> > Slavka
> >
> > On Thu, Dec 16, 2021 at 4:08 AM Jeremy Hansen <je...@skidrow.la.invalid>
> > wrote:
> >
> > Any tips on how I would troubleshoot this? I’ve tried downgrading libvirt
> > and qemu and ca-certificates to the same version as the other functional
> > nodes. That didn’t seem to help. This is obviously an ssl issue but I
> > don’t really know what to do about it.
> >
> > 2021-12-15 18:04:14,438 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Agent started
> > 2021-12-15 18:04:14,444 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Implementation Version is 4.15.0.0
> > 2021-12-15 18:04:14,447 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> > 2021-12-15 18:04:14,466 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to using properties file for storage
> > 2021-12-15 18:04:14,467 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to the constant time backoff algorithm
> > 2021-12-15 18:04:14,471 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> > 2021-12-15 18:04:14,485 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Using default Java settings for IPv6 preference for agent
> > connection
> > 2021-12-15 18:04:14,592 INFO [cloud.agent.Agent] (main:null) (logid:) id
> > is 0
> > 2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > file uefi.properties.
> > 2021-12-15 18:04:14,663 INFO [kvm.resource.LibvirtConnection] (main:null)
> > (logid:) No existing libvirtd connection found. Opening a new one
> > 2021-12-15 18:04:14,890 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > BridgeVifDriver.
> > 2021-12-15 18:04:15,086 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) iscsi session clean up is disabled
> > 2021-12-15 18:04:15,129 INFO [cloud.agent.Agent] (main:null) (logid:)
> > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > workers = 5 : host = 192.168.30.59 : port = 8250
> > 2021-12-15 18:04:15,139 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connecting to 192.168.30.59:8250
> > 2021-12-15 18:04:15,153 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > file found: /etc/cloudstack/agent/agent.properties
> > 2021-12-15 18:04:15,919 INFO [utils.nio.NioClient] (main:null) (logid:)
> > SSL: Handshake done
> > 2021-12-15 18:04:15,920 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connected to 192.168.30.59:8250
> > 2021-12-15 18:04:16,057 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > 2021-12-15 18:04:16,062 ERROR [kvm.resource.LibvirtConnection]
> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> > invalid
> > connection pointer in virConnectGetVersion
> > 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > 2021-12-15 18:04:16,151 INFO [cloud.serializer.GsonHelper]
> > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > 2021-12-15 18:04:16,272 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Proccess agent startup answer, agent id = 0
> > 2021-12-15 18:04:16,273 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Set agent id 0
> > 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Startup Response Received: agent id = 0
> > 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent]
> > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> >
> > I also noticed this:
> >
> > 2021-12-15 18:05:34,542 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentConnectTaskPool-19617:ctx-e0dfdb18) (logid:d3930423) Failed to
> > handle
> > host connection:
> > java.lang.IllegalArgumentException: Can't add host: 192.168.30.54 with
> > hostOS: Rocky into a cluster,in which there are CentOS hosts added
> > at
> > com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.createHostVOForConnectedAgent(LibvirtServerDiscoverer.java:456)
> >
> >
> > at
> > com.cloud.resource.ResourceManagerImpl.dispatchToStateAdapters(ResourceManagerImpl.java:1719)
> >
> >
> > at
> > com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1946)
> >
> >
> > at
> > com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:2278)
> >
> >
> > at jdk.internal.reflect.GeneratedMethodAccessor510.invoke(Unknown Source)
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >
> >
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> >
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> >
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >
> >
> > at
> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
> >
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> >
> >
> > at
> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> >
> >
> > at com.sun.proxy.$Proxy188.createHostVOForConnectedAgent(Unknown Source)
> > at
> > com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1097)
> >
> >
> > at
> > com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1194)
> >
> >
> > at
> > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >
> >
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >
> >
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >
> >
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >
> >
> > at
> > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >
> >
> > at java.base/java.lang.Thread.run(Thread.java:829)
> >
> > The fact that this host has been converted to Rocky Linux is causing an
> > issue? What’s the work around for this?
> >
> > Thanks
> > -jeremy
> >
> >
> >
> > On Monday, Dec 13, 2021 at 12:19 AM, Jeremy Hansen <je...@skidrow.la>
> > wrote:
> > It doesn’t error out but there is purposely no VMs running on these hosts
> > to test the upgrade of the underlying distro before having to stop any
> > active VMs.
> >
> > [jeremy@cm02 ~]$ sudo virsh list
> > Id Name State
> > --------------------
> >
> > [jeremy@cm02 ~]$
> >
> >
> > -jeremy
> >
> >
> >
> >
> > On Monday, Dec 13, 2021 at 12:06 AM, Wei ZHOU <us...@gmail.com>
> > wrote:
> > Hi,
> >
> > Do virsh commands e.g. "virsh list" work ?
> >
> > -Wei
> >
> > On Mon, 13 Dec 2021 at 06:46, Jeremy Hansen <je...@skidrow.la.invalid>
> > wrote:
> >
> > Testing on an unused compute node, I tested upgrading to RockyLinux 8.5.
> >
> > I’m running Cloudstack 4.15.0.0.
> >
> > Trying to bring up cloudstack agent, I’m seeing some issues communicating
> > with libvirt:
> >
> > 2021-12-12 21:25:03,992 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > d1b1e853-1c30-473d-badc-6c30318aa5b0 (Filesystem) in libvirt
> > 2021-12-12 21:25:03,997 ERROR [kvm.resource.LibvirtConnection]
> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> > invalid
> > connection pointer in virConnectGetVersion
> > 2021-12-12 21:25:04,000 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > d1b1e853-1c30-473d-badc-6c30318aa5b0, using it.
> >
> > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> > Input/output error
> > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> > Input/output error
> > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> > Input/output error
> > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> > Input/output error
> > Dec 12 21:24:26 cm02 libvirtd[269244]: Cannot recv data: Input/output
> > error
> >
> > Libvirt version is: libvirt-6.0.0-37.module+el8.5.0+670+c4aa478c.x86_64
> >
> > Functional hosts that have yet to be upgraded are using:
> >
> > libvirt-6.0.0-35.module_el8.4.0+783+f8734d30.x86_64
> >
> >
> > My libvirtd.conf looks like this:
> >
> > listen_tcp=0
> > listen_tls=1
> > tcp_port="16509"
> > auth_tcp="none"
> > mdns_adv = 0
> > key_file="/etc/pki/libvirt/private/serverkey.pem"
> > cert_file="/etc/pki/libvirt/servercert.pem"
> > ca_file="/etc/pki/CA/cacert.pem"
> > tls_port="16514"
> > auth_tls=“none"
> >
> > 2021-12-12 21:43:42,841 ERROR [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > file uefi.properties.
> > 2021-12-12 21:43:42,901 INFO [kvm.resource.LibvirtConnection] (main:null)
> > (logid:) No existing libvirtd connection found. Opening a new one
> > 2021-12-12 21:43:43,127 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > BridgeVifDriver.
> > 2021-12-12 21:43:43,296 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) iscsi session clean up is disabled
> > 2021-12-12 21:43:43,312 INFO [cloud.agent.Agent] (main:null) (logid:)
> > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > workers = 5 : host = 192.168.30.59 : port = 8250
> > 2021-12-12 21:43:43,321 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connecting to 192.168.30.59:8250
> > 2021-12-12 21:43:43,325 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > file found: /etc/cloudstack/agent/agent.properties
> > 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> > SSL: Handshake done
> > 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connected to 192.168.30.59:8250
> > 2021-12-12 21:43:43,925 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > 2021-12-12 21:43:43,929 ERROR [kvm.resource.LibvirtConnection]
> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> > invalid
> > connection pointer in virConnectGetVersion
> > 2021-12-12 21:43:43,932 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > 2021-12-12 21:43:43,933 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > 2021-12-12 21:43:43,985 INFO [cloud.serializer.GsonHelper]
> > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > 2021-12-12 21:43:44,020 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Proccess agent startup answer, agent id = 0
> > 2021-12-12 21:43:44,022 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Set agent id 0
> > 2021-12-12 21:43:44,028 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Startup Response Received: agent id = 0
> > 2021-12-12 21:43:44,031 INFO [cloud.agent.Agent]
> > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> > 2021-12-12 21:43:55,682 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Agent started
> > 2021-12-12 21:43:55,688 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Implementation Version is 4.15.0.0
> > 2021-12-12 21:43:55,690 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> > 2021-12-12 21:43:55,709 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to using properties file for storage
> > 2021-12-12 21:43:55,711 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to the constant time backoff algorithm
> > 2021-12-12 21:43:55,714 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> > 2021-12-12 21:43:55,728 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Using default Java settings for IPv6 preference for agent
> > connection
> > 2021-12-12 21:43:55,840 INFO [cloud.agent.Agent] (main:null) (logid:) id
> > is 0
> > 2021-12-12 21:43:55,853 ERROR [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > file uefi.properties.
> > 2021-12-12 21:43:55,909 INFO [kvm.resource.LibvirtConnection] (main:null)
> > (logid:) No existing libvirtd connection found. Opening a new one
> > 2021-12-12 21:43:56,145 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > BridgeVifDriver.
> > 2021-12-12 21:43:56,307 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) iscsi session clean up is disabled
> > 2021-12-12 21:43:56,322 INFO [cloud.agent.Agent] (main:null) (logid:)
> > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > workers = 5 : host = 192.168.30.59 : port = 8250
> > 2021-12-12 21:43:56,331 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connecting to 192.168.30.59:8250
> > 2021-12-12 21:43:56,335 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > file found: /etc/cloudstack/agent/agent.properties
> > 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> > SSL: Handshake done
> > 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connected to 192.168.30.59:8250
> > 2021-12-12 21:43:56,941 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > 2021-12-12 21:43:56,945 ERROR [kvm.resource.LibvirtConnection]
> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> > invalid
> > connection pointer in virConnectGetVersion
> > 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > 2021-12-12 21:43:56,998 INFO [cloud.serializer.GsonHelper]
> > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > 2021-12-12 21:43:57,031 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Proccess agent startup answer, agent id = 0
> > 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent]
> > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> > 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Set agent id 0
> > 2021-12-12 21:43:57,040 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Startup Response Received: agent id = 0
> >
> > Any ideas? I’m working toward upgrading the entire cluster along with
> > Cloudstack itself.
> >
> > -jeremy
> >
> >
> >
> >
> >
> >
> >
>
> --
>
> Andrija Panić

Re: Issues with libvirt after upgrading to RockyLinux 8.5

Posted by Andrija Panic <an...@gmail.com>.

 Can't add host: 192.168.30.54 with hostOS: Rocky into a cluster,in which
there are CentOS hosts added


Try changing the content of the /etc/*release file - to match the file of
the CentOS nodes - and then play with restarting agent, and observe if you
would hit the same error or not - I've seen this issue (RHEL inside CentOS
cluster, due to bad /etc/rhel-release file content), but I've not tested if
this "fix" works or not.

Best,

On Thu, 16 Dec 2021 at 08:39, Jeremy Hansen <je...@skidrow.la.invalid>
wrote:

> But if I convert all the hosts to Rocky and upgrade to 4.16, I should be
> ok?
>
> Thanks
>
>
>
> On Wednesday, Dec 15, 2021 at 11:17 PM, Slavka Peleva <
> slavkap@storpool.com.INVALID> wrote:
> Sorry, I didn't pay attention to your CS version. After the upgrade, I
> think you will have the same problem. Because in the DB, there is
> information about host/hosts on this cluster that is/are with CentOS.
>
> Best regards,
> Slavka
>
> On Thu, Dec 16, 2021 at 8:49 AM Jeremy Hansen <je...@skidrow.la.invalid>
> wrote:
>
> I noticed in the compatibility matrix that Rocky isn’t supported until
> 4.16.0.0. If I upgrade Cloudstack first, would this help or is it still
> going to complain about the centos/rocky mix? If I convert all my existing
> nodes to Rocky, which is the plan anyway, will this go away? Shouldn’t
> CentOS and Rocky be considered that same thing… sort of…?
>
> Thanks
> -jeremy
>
>
>
>
> On Wednesday, Dec 15, 2021 at 10:43 PM, Slavka Peleva <
> slavkap@storpool.com.INVALID> wrote:
> Hi Jeremy,
>
> It will help if you have another cluster for Rocky Linux. Hosts need to be
> of the same OS, it's not possible to mix OSes in the same cluster.
>
> Best regards,
> Slavka
>
> On Thu, Dec 16, 2021 at 4:08 AM Jeremy Hansen <je...@skidrow.la.invalid>
> wrote:
>
> Any tips on how I would troubleshoot this? I’ve tried downgrading libvirt
> and qemu and ca-certificates to the same version as the other functional
> nodes. That didn’t seem to help. This is obviously an ssl issue but I
> don’t really know what to do about it.
>
> 2021-12-15 18:04:14,438 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Agent started
> 2021-12-15 18:04:14,444 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Implementation Version is 4.15.0.0
> 2021-12-15 18:04:14,447 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> 2021-12-15 18:04:14,466 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to using properties file for storage
> 2021-12-15 18:04:14,467 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to the constant time backoff algorithm
> 2021-12-15 18:04:14,471 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> 2021-12-15 18:04:14,485 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Using default Java settings for IPv6 preference for agent
> connection
> 2021-12-15 18:04:14,592 INFO [cloud.agent.Agent] (main:null) (logid:) id
> is 0
> 2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) uefi properties file not found due to: Unable to find
> file uefi.properties.
> 2021-12-15 18:04:14,663 INFO [kvm.resource.LibvirtConnection] (main:null)
> (logid:) No existing libvirtd connection found. Opening a new one
> 2021-12-15 18:04:14,890 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> BridgeVifDriver.
> 2021-12-15 18:04:15,086 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) iscsi session clean up is disabled
> 2021-12-15 18:04:15,129 INFO [cloud.agent.Agent] (main:null) (logid:)
> Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> workers = 5 : host = 192.168.30.59 : port = 8250
> 2021-12-15 18:04:15,139 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connecting to 192.168.30.59:8250
> 2021-12-15 18:04:15,153 INFO [utils.nio.Link] (main:null) (logid:) Conf
> file found: /etc/cloudstack/agent/agent.properties
> 2021-12-15 18:04:15,919 INFO [utils.nio.NioClient] (main:null) (logid:)
> SSL: Handshake done
> 2021-12-15 18:04:15,920 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connected to 192.168.30.59:8250
> 2021-12-15 18:04:16,057 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> 2021-12-15 18:04:16,062 ERROR [kvm.resource.LibvirtConnection]
> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> invalid
> connection pointer in virConnectGetVersion
> 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> 18796842-a137-475d-9799-9874240e3c0c, using it.
> 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> 18796842-a137-475d-9799-9874240e3c0c from libvirt
> 2021-12-15 18:04:16,151 INFO [cloud.serializer.GsonHelper]
> (Agent-Handler-1:null) (logid:) Default Builder inited.
> 2021-12-15 18:04:16,272 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Proccess agent startup answer, agent id = 0
> 2021-12-15 18:04:16,273 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Set agent id 0
> 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Startup Response Received: agent id = 0
> 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent]
> (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
>
> I also noticed this:
>
> 2021-12-15 18:05:34,542 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentConnectTaskPool-19617:ctx-e0dfdb18) (logid:d3930423) Failed to
> handle
> host connection:
> java.lang.IllegalArgumentException: Can't add host: 192.168.30.54 with
> hostOS: Rocky into a cluster,in which there are CentOS hosts added
> at
> com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.createHostVOForConnectedAgent(LibvirtServerDiscoverer.java:456)
>
>
> at
> com.cloud.resource.ResourceManagerImpl.dispatchToStateAdapters(ResourceManagerImpl.java:1719)
>
>
> at
> com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1946)
>
>
> at
> com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:2278)
>
>
> at jdk.internal.reflect.GeneratedMethodAccessor510.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>
>
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
>
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>
>
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>
>
> at com.sun.proxy.$Proxy188.createHostVOForConnectedAgent(Unknown Source)
> at
> com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1097)
>
>
> at
> com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1194)
>
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
>
> at java.base/java.lang.Thread.run(Thread.java:829)
>
> The fact that this host has been converted to Rocky Linux is causing an
> issue? What’s the work around for this?
>
> Thanks
> -jeremy
>
>
>
> On Monday, Dec 13, 2021 at 12:19 AM, Jeremy Hansen <je...@skidrow.la>
> wrote:
> It doesn’t error out but there is purposely no VMs running on these hosts
> to test the upgrade of the underlying distro before having to stop any
> active VMs.
>
> [jeremy@cm02 ~]$ sudo virsh list
> Id Name State
> --------------------
>
> [jeremy@cm02 ~]$
>
>
> -jeremy
>
>
>
>
> On Monday, Dec 13, 2021 at 12:06 AM, Wei ZHOU <us...@gmail.com>
> wrote:
> Hi,
>
> Do virsh commands e.g. "virsh list" work ?
>
> -Wei
>
> On Mon, 13 Dec 2021 at 06:46, Jeremy Hansen <je...@skidrow.la.invalid>
> wrote:
>
> Testing on an unused compute node, I tested upgrading to RockyLinux 8.5.
>
> I’m running Cloudstack 4.15.0.0.
>
> Trying to bring up cloudstack agent, I’m seeing some issues communicating
> with libvirt:
>
> 2021-12-12 21:25:03,992 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> d1b1e853-1c30-473d-badc-6c30318aa5b0 (Filesystem) in libvirt
> 2021-12-12 21:25:03,997 ERROR [kvm.resource.LibvirtConnection]
> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> invalid
> connection pointer in virConnectGetVersion
> 2021-12-12 21:25:04,000 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> d1b1e853-1c30-473d-badc-6c30318aa5b0, using it.
>
> Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> Input/output error
> Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> Input/output error
> Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> Input/output error
> Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> Input/output error
> Dec 12 21:24:26 cm02 libvirtd[269244]: Cannot recv data: Input/output
> error
>
> Libvirt version is: libvirt-6.0.0-37.module+el8.5.0+670+c4aa478c.x86_64
>
> Functional hosts that have yet to be upgraded are using:
>
> libvirt-6.0.0-35.module_el8.4.0+783+f8734d30.x86_64
>
>
> My libvirtd.conf looks like this:
>
> listen_tcp=0
> listen_tls=1
> tcp_port="16509"
> auth_tcp="none"
> mdns_adv = 0
> key_file="/etc/pki/libvirt/private/serverkey.pem"
> cert_file="/etc/pki/libvirt/servercert.pem"
> ca_file="/etc/pki/CA/cacert.pem"
> tls_port="16514"
> auth_tls=“none"
>
> 2021-12-12 21:43:42,841 ERROR [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) uefi properties file not found due to: Unable to find
> file uefi.properties.
> 2021-12-12 21:43:42,901 INFO [kvm.resource.LibvirtConnection] (main:null)
> (logid:) No existing libvirtd connection found. Opening a new one
> 2021-12-12 21:43:43,127 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> BridgeVifDriver.
> 2021-12-12 21:43:43,296 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) iscsi session clean up is disabled
> 2021-12-12 21:43:43,312 INFO [cloud.agent.Agent] (main:null) (logid:)
> Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> workers = 5 : host = 192.168.30.59 : port = 8250
> 2021-12-12 21:43:43,321 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connecting to 192.168.30.59:8250
> 2021-12-12 21:43:43,325 INFO [utils.nio.Link] (main:null) (logid:) Conf
> file found: /etc/cloudstack/agent/agent.properties
> 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> SSL: Handshake done
> 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connected to 192.168.30.59:8250
> 2021-12-12 21:43:43,925 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> 2021-12-12 21:43:43,929 ERROR [kvm.resource.LibvirtConnection]
> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> invalid
> connection pointer in virConnectGetVersion
> 2021-12-12 21:43:43,932 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> 18796842-a137-475d-9799-9874240e3c0c, using it.
> 2021-12-12 21:43:43,933 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> 18796842-a137-475d-9799-9874240e3c0c from libvirt
> 2021-12-12 21:43:43,985 INFO [cloud.serializer.GsonHelper]
> (Agent-Handler-1:null) (logid:) Default Builder inited.
> 2021-12-12 21:43:44,020 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Proccess agent startup answer, agent id = 0
> 2021-12-12 21:43:44,022 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Set agent id 0
> 2021-12-12 21:43:44,028 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Startup Response Received: agent id = 0
> 2021-12-12 21:43:44,031 INFO [cloud.agent.Agent]
> (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> 2021-12-12 21:43:55,682 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Agent started
> 2021-12-12 21:43:55,688 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Implementation Version is 4.15.0.0
> 2021-12-12 21:43:55,690 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> 2021-12-12 21:43:55,709 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to using properties file for storage
> 2021-12-12 21:43:55,711 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to the constant time backoff algorithm
> 2021-12-12 21:43:55,714 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> 2021-12-12 21:43:55,728 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Using default Java settings for IPv6 preference for agent
> connection
> 2021-12-12 21:43:55,840 INFO [cloud.agent.Agent] (main:null) (logid:) id
> is 0
> 2021-12-12 21:43:55,853 ERROR [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) uefi properties file not found due to: Unable to find
> file uefi.properties.
> 2021-12-12 21:43:55,909 INFO [kvm.resource.LibvirtConnection] (main:null)
> (logid:) No existing libvirtd connection found. Opening a new one
> 2021-12-12 21:43:56,145 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> BridgeVifDriver.
> 2021-12-12 21:43:56,307 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) iscsi session clean up is disabled
> 2021-12-12 21:43:56,322 INFO [cloud.agent.Agent] (main:null) (logid:)
> Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> workers = 5 : host = 192.168.30.59 : port = 8250
> 2021-12-12 21:43:56,331 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connecting to 192.168.30.59:8250
> 2021-12-12 21:43:56,335 INFO [utils.nio.Link] (main:null) (logid:) Conf
> file found: /etc/cloudstack/agent/agent.properties
> 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> SSL: Handshake done
> 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connected to 192.168.30.59:8250
> 2021-12-12 21:43:56,941 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> 2021-12-12 21:43:56,945 ERROR [kvm.resource.LibvirtConnection]
> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> invalid
> connection pointer in virConnectGetVersion
> 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> 18796842-a137-475d-9799-9874240e3c0c, using it.
> 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> 18796842-a137-475d-9799-9874240e3c0c from libvirt
> 2021-12-12 21:43:56,998 INFO [cloud.serializer.GsonHelper]
> (Agent-Handler-1:null) (logid:) Default Builder inited.
> 2021-12-12 21:43:57,031 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Proccess agent startup answer, agent id = 0
> 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent]
> (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Set agent id 0
> 2021-12-12 21:43:57,040 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Startup Response Received: agent id = 0
>
> Any ideas? I’m working toward upgrading the entire cluster along with
> Cloudstack itself.
>
> -jeremy
>
>
>
>
>
>
>

-- 

Andrija Panić

Re: Issues with libvirt after upgrading to RockyLinux 8.5

Posted by Jeremy Hansen <je...@skidrow.la.INVALID>.

But if I convert all the hosts to Rocky and upgrade to 4.16, I should be ok?

Thanks

> On Wednesday, Dec 15, 2021 at 11:17 PM, Slavka Peleva <slavkap@storpool.com.INVALID (mailto:slavkap@storpool.com.INVALID)> wrote:
> Sorry, I didn't pay attention to your CS version. After the upgrade, I
> think you will have the same problem. Because in the DB, there is
> information about host/hosts on this cluster that is/are with CentOS.
>
> Best regards,
> Slavka
>
> On Thu, Dec 16, 2021 at 8:49 AM Jeremy Hansen <je...@skidrow.la.invalid>
> wrote:
>
> > I noticed in the compatibility matrix that Rocky isn’t supported until
> > 4.16.0.0. If I upgrade Cloudstack first, would this help or is it still
> > going to complain about the centos/rocky mix? If I convert all my existing
> > nodes to Rocky, which is the plan anyway, will this go away? Shouldn’t
> > CentOS and Rocky be considered that same thing… sort of…?
> >
> > Thanks
> > -jeremy
> >
> >
> >
> >
> > On Wednesday, Dec 15, 2021 at 10:43 PM, Slavka Peleva <
> > slavkap@storpool.com.INVALID> wrote:
> > Hi Jeremy,
> >
> > It will help if you have another cluster for Rocky Linux. Hosts need to be
> > of the same OS, it's not possible to mix OSes in the same cluster.
> >
> > Best regards,
> > Slavka
> >
> > On Thu, Dec 16, 2021 at 4:08 AM Jeremy Hansen <je...@skidrow.la.invalid>
> > wrote:
> >
> > Any tips on how I would troubleshoot this? I’ve tried downgrading libvirt
> > and qemu and ca-certificates to the same version as the other functional
> > nodes. That didn’t seem to help. This is obviously an ssl issue but I
> > don’t really know what to do about it.
> >
> > 2021-12-15 18:04:14,438 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Agent started
> > 2021-12-15 18:04:14,444 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Implementation Version is 4.15.0.0
> > 2021-12-15 18:04:14,447 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> > 2021-12-15 18:04:14,466 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to using properties file for storage
> > 2021-12-15 18:04:14,467 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to the constant time backoff algorithm
> > 2021-12-15 18:04:14,471 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> > 2021-12-15 18:04:14,485 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Using default Java settings for IPv6 preference for agent
> > connection
> > 2021-12-15 18:04:14,592 INFO [cloud.agent.Agent] (main:null) (logid:) id
> > is 0
> > 2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > file uefi.properties.
> > 2021-12-15 18:04:14,663 INFO [kvm.resource.LibvirtConnection] (main:null)
> > (logid:) No existing libvirtd connection found. Opening a new one
> > 2021-12-15 18:04:14,890 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > BridgeVifDriver.
> > 2021-12-15 18:04:15,086 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) iscsi session clean up is disabled
> > 2021-12-15 18:04:15,129 INFO [cloud.agent.Agent] (main:null) (logid:)
> > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > workers = 5 : host = 192.168.30.59 : port = 8250
> > 2021-12-15 18:04:15,139 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connecting to 192.168.30.59:8250
> > 2021-12-15 18:04:15,153 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > file found: /etc/cloudstack/agent/agent.properties
> > 2021-12-15 18:04:15,919 INFO [utils.nio.NioClient] (main:null) (logid:)
> > SSL: Handshake done
> > 2021-12-15 18:04:15,920 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connected to 192.168.30.59:8250
> > 2021-12-15 18:04:16,057 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > 2021-12-15 18:04:16,062 ERROR [kvm.resource.LibvirtConnection]
> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> > invalid
> > connection pointer in virConnectGetVersion
> > 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > 2021-12-15 18:04:16,151 INFO [cloud.serializer.GsonHelper]
> > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > 2021-12-15 18:04:16,272 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Proccess agent startup answer, agent id = 0
> > 2021-12-15 18:04:16,273 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Set agent id 0
> > 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Startup Response Received: agent id = 0
> > 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent]
> > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> >
> > I also noticed this:
> >
> > 2021-12-15 18:05:34,542 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentConnectTaskPool-19617:ctx-e0dfdb18) (logid:d3930423) Failed to
> > handle
> > host connection:
> > java.lang.IllegalArgumentException: Can't add host: 192.168.30.54 with
> > hostOS: Rocky into a cluster,in which there are CentOS hosts added
> > at
> > com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.createHostVOForConnectedAgent(LibvirtServerDiscoverer.java:456)
> >
> > at
> > com.cloud.resource.ResourceManagerImpl.dispatchToStateAdapters(ResourceManagerImpl.java:1719)
> >
> > at
> > com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1946)
> >
> > at
> > com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:2278)
> >
> > at jdk.internal.reflect.GeneratedMethodAccessor510.invoke(Unknown Source)
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >
> > at
> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
> >
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> >
> > at
> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> >
> > at com.sun.proxy.$Proxy188.createHostVOForConnectedAgent(Unknown Source)
> > at
> > com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1097)
> >
> > at
> > com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1194)
> >
> > at
> > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >
> > at
> > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >
> > at java.base/java.lang.Thread.run(Thread.java:829)
> >
> > The fact that this host has been converted to Rocky Linux is causing an
> > issue? What’s the work around for this?
> >
> > Thanks
> > -jeremy
> >
> >
> >
> > On Monday, Dec 13, 2021 at 12:19 AM, Jeremy Hansen <je...@skidrow.la>
> > wrote:
> > It doesn’t error out but there is purposely no VMs running on these hosts
> > to test the upgrade of the underlying distro before having to stop any
> > active VMs.
> >
> > [jeremy@cm02 ~]$ sudo virsh list
> > Id Name State
> > --------------------
> >
> > [jeremy@cm02 ~]$
> >
> >
> > -jeremy
> >
> >
> >
> >
> > On Monday, Dec 13, 2021 at 12:06 AM, Wei ZHOU <us...@gmail.com>
> > wrote:
> > Hi,
> >
> > Do virsh commands e.g. "virsh list" work ?
> >
> > -Wei
> >
> > On Mon, 13 Dec 2021 at 06:46, Jeremy Hansen <je...@skidrow.la.invalid>
> > wrote:
> >
> > Testing on an unused compute node, I tested upgrading to RockyLinux 8.5.
> >
> > I’m running Cloudstack 4.15.0.0.
> >
> > Trying to bring up cloudstack agent, I’m seeing some issues communicating
> > with libvirt:
> >
> > 2021-12-12 21:25:03,992 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > d1b1e853-1c30-473d-badc-6c30318aa5b0 (Filesystem) in libvirt
> > 2021-12-12 21:25:03,997 ERROR [kvm.resource.LibvirtConnection]
> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> > invalid
> > connection pointer in virConnectGetVersion
> > 2021-12-12 21:25:04,000 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > d1b1e853-1c30-473d-badc-6c30318aa5b0, using it.
> >
> > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> > Input/output error
> > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> > Input/output error
> > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> > Input/output error
> > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> > Input/output error
> > Dec 12 21:24:26 cm02 libvirtd[269244]: Cannot recv data: Input/output
> > error
> >
> > Libvirt version is: libvirt-6.0.0-37.module+el8.5.0+670+c4aa478c.x86_64
> >
> > Functional hosts that have yet to be upgraded are using:
> >
> > libvirt-6.0.0-35.module_el8.4.0+783+f8734d30.x86_64
> >
> >
> > My libvirtd.conf looks like this:
> >
> > listen_tcp=0
> > listen_tls=1
> > tcp_port="16509"
> > auth_tcp="none"
> > mdns_adv = 0
> > key_file="/etc/pki/libvirt/private/serverkey.pem"
> > cert_file="/etc/pki/libvirt/servercert.pem"
> > ca_file="/etc/pki/CA/cacert.pem"
> > tls_port="16514"
> > auth_tls=“none"
> >
> > 2021-12-12 21:43:42,841 ERROR [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > file uefi.properties.
> > 2021-12-12 21:43:42,901 INFO [kvm.resource.LibvirtConnection] (main:null)
> > (logid:) No existing libvirtd connection found. Opening a new one
> > 2021-12-12 21:43:43,127 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > BridgeVifDriver.
> > 2021-12-12 21:43:43,296 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) iscsi session clean up is disabled
> > 2021-12-12 21:43:43,312 INFO [cloud.agent.Agent] (main:null) (logid:)
> > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > workers = 5 : host = 192.168.30.59 : port = 8250
> > 2021-12-12 21:43:43,321 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connecting to 192.168.30.59:8250
> > 2021-12-12 21:43:43,325 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > file found: /etc/cloudstack/agent/agent.properties
> > 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> > SSL: Handshake done
> > 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connected to 192.168.30.59:8250
> > 2021-12-12 21:43:43,925 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > 2021-12-12 21:43:43,929 ERROR [kvm.resource.LibvirtConnection]
> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> > invalid
> > connection pointer in virConnectGetVersion
> > 2021-12-12 21:43:43,932 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > 2021-12-12 21:43:43,933 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > 2021-12-12 21:43:43,985 INFO [cloud.serializer.GsonHelper]
> > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > 2021-12-12 21:43:44,020 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Proccess agent startup answer, agent id = 0
> > 2021-12-12 21:43:44,022 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Set agent id 0
> > 2021-12-12 21:43:44,028 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Startup Response Received: agent id = 0
> > 2021-12-12 21:43:44,031 INFO [cloud.agent.Agent]
> > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> > 2021-12-12 21:43:55,682 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Agent started
> > 2021-12-12 21:43:55,688 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Implementation Version is 4.15.0.0
> > 2021-12-12 21:43:55,690 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> > 2021-12-12 21:43:55,709 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to using properties file for storage
> > 2021-12-12 21:43:55,711 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to the constant time backoff algorithm
> > 2021-12-12 21:43:55,714 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> > 2021-12-12 21:43:55,728 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Using default Java settings for IPv6 preference for agent
> > connection
> > 2021-12-12 21:43:55,840 INFO [cloud.agent.Agent] (main:null) (logid:) id
> > is 0
> > 2021-12-12 21:43:55,853 ERROR [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > file uefi.properties.
> > 2021-12-12 21:43:55,909 INFO [kvm.resource.LibvirtConnection] (main:null)
> > (logid:) No existing libvirtd connection found. Opening a new one
> > 2021-12-12 21:43:56,145 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > BridgeVifDriver.
> > 2021-12-12 21:43:56,307 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) iscsi session clean up is disabled
> > 2021-12-12 21:43:56,322 INFO [cloud.agent.Agent] (main:null) (logid:)
> > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > workers = 5 : host = 192.168.30.59 : port = 8250
> > 2021-12-12 21:43:56,331 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connecting to 192.168.30.59:8250
> > 2021-12-12 21:43:56,335 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > file found: /etc/cloudstack/agent/agent.properties
> > 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> > SSL: Handshake done
> > 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connected to 192.168.30.59:8250
> > 2021-12-12 21:43:56,941 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > 2021-12-12 21:43:56,945 ERROR [kvm.resource.LibvirtConnection]
> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> > invalid
> > connection pointer in virConnectGetVersion
> > 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > 2021-12-12 21:43:56,998 INFO [cloud.serializer.GsonHelper]
> > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > 2021-12-12 21:43:57,031 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Proccess agent startup answer, agent id = 0
> > 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent]
> > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> > 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Set agent id 0
> > 2021-12-12 21:43:57,040 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Startup Response Received: agent id = 0
> >
> > Any ideas? I’m working toward upgrading the entire cluster along with
> > Cloudstack itself.
> >
> > -jeremy
> >
> >
> >
> >
> >
> >

Re: Issues with libvirt after upgrading to RockyLinux 8.5

Posted by Slavka Peleva <sl...@storpool.com.INVALID>.

Sorry, I didn't pay attention to your CS version. After the upgrade, I
think you will have the same problem. Because in the DB, there is
information about host/hosts on this cluster that is/are with CentOS.

Best regards,
Slavka

On Thu, Dec 16, 2021 at 8:49 AM Jeremy Hansen <je...@skidrow.la.invalid>
wrote:

> I noticed in the compatibility matrix that Rocky isn’t supported until
> 4.16.0.0.  If I upgrade Cloudstack first, would this help or is it still
> going to complain about the centos/rocky mix?  If I convert all my existing
> nodes to Rocky, which is the plan anyway, will this go away?  Shouldn’t
> CentOS and Rocky be considered that same thing… sort of…?
>
> Thanks
> -jeremy
>
>
>
>
> On Wednesday, Dec 15, 2021 at 10:43 PM, Slavka Peleva <
> slavkap@storpool.com.INVALID> wrote:
> Hi Jeremy,
>
> It will help if you have another cluster for Rocky Linux. Hosts need to be
> of the same OS, it's not possible to mix OSes in the same cluster.
>
> Best regards,
> Slavka
>
> On Thu, Dec 16, 2021 at 4:08 AM Jeremy Hansen <je...@skidrow.la.invalid>
> wrote:
>
> Any tips on how I would troubleshoot this? I’ve tried downgrading libvirt
> and qemu and ca-certificates to the same version as the other functional
> nodes. That didn’t seem to help. This is obviously an ssl issue but I
> don’t really know what to do about it.
>
> 2021-12-15 18:04:14,438 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Agent started
> 2021-12-15 18:04:14,444 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Implementation Version is 4.15.0.0
> 2021-12-15 18:04:14,447 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> 2021-12-15 18:04:14,466 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to using properties file for storage
> 2021-12-15 18:04:14,467 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to the constant time backoff algorithm
> 2021-12-15 18:04:14,471 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> 2021-12-15 18:04:14,485 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Using default Java settings for IPv6 preference for agent
> connection
> 2021-12-15 18:04:14,592 INFO [cloud.agent.Agent] (main:null) (logid:) id
> is 0
> 2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) uefi properties file not found due to: Unable to find
> file uefi.properties.
> 2021-12-15 18:04:14,663 INFO [kvm.resource.LibvirtConnection] (main:null)
> (logid:) No existing libvirtd connection found. Opening a new one
> 2021-12-15 18:04:14,890 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> BridgeVifDriver.
> 2021-12-15 18:04:15,086 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) iscsi session clean up is disabled
> 2021-12-15 18:04:15,129 INFO [cloud.agent.Agent] (main:null) (logid:)
> Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> workers = 5 : host = 192.168.30.59 : port = 8250
> 2021-12-15 18:04:15,139 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connecting to 192.168.30.59:8250
> 2021-12-15 18:04:15,153 INFO [utils.nio.Link] (main:null) (logid:) Conf
> file found: /etc/cloudstack/agent/agent.properties
> 2021-12-15 18:04:15,919 INFO [utils.nio.NioClient] (main:null) (logid:)
> SSL: Handshake done
> 2021-12-15 18:04:15,920 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connected to 192.168.30.59:8250
> 2021-12-15 18:04:16,057 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> 2021-12-15 18:04:16,062 ERROR [kvm.resource.LibvirtConnection]
> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> invalid
> connection pointer in virConnectGetVersion
> 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> 18796842-a137-475d-9799-9874240e3c0c, using it.
> 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> 18796842-a137-475d-9799-9874240e3c0c from libvirt
> 2021-12-15 18:04:16,151 INFO [cloud.serializer.GsonHelper]
> (Agent-Handler-1:null) (logid:) Default Builder inited.
> 2021-12-15 18:04:16,272 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Proccess agent startup answer, agent id = 0
> 2021-12-15 18:04:16,273 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Set agent id 0
> 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Startup Response Received: agent id = 0
> 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent]
> (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
>
> I also noticed this:
>
> 2021-12-15 18:05:34,542 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentConnectTaskPool-19617:ctx-e0dfdb18) (logid:d3930423) Failed to
> handle
> host connection:
> java.lang.IllegalArgumentException: Can't add host: 192.168.30.54 with
> hostOS: Rocky into a cluster,in which there are CentOS hosts added
> at
> com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.createHostVOForConnectedAgent(LibvirtServerDiscoverer.java:456)
>
> at
> com.cloud.resource.ResourceManagerImpl.dispatchToStateAdapters(ResourceManagerImpl.java:1719)
>
> at
> com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1946)
>
> at
> com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:2278)
>
> at jdk.internal.reflect.GeneratedMethodAccessor510.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>
> at com.sun.proxy.$Proxy188.createHostVOForConnectedAgent(Unknown Source)
> at
> com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1097)
>
> at
> com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1194)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> at java.base/java.lang.Thread.run(Thread.java:829)
>
> The fact that this host has been converted to Rocky Linux is causing an
> issue? What’s the work around for this?
>
> Thanks
> -jeremy
>
>
>
> On Monday, Dec 13, 2021 at 12:19 AM, Jeremy Hansen <je...@skidrow.la>
> wrote:
> It doesn’t error out but there is purposely no VMs running on these hosts
> to test the upgrade of the underlying distro before having to stop any
> active VMs.
>
> [jeremy@cm02 ~]$ sudo virsh list
> Id Name State
> --------------------
>
> [jeremy@cm02 ~]$
>
>
> -jeremy
>
>
>
>
> On Monday, Dec 13, 2021 at 12:06 AM, Wei ZHOU <us...@gmail.com>
> wrote:
> Hi,
>
> Do virsh commands e.g. "virsh list" work ?
>
> -Wei
>
> On Mon, 13 Dec 2021 at 06:46, Jeremy Hansen <je...@skidrow.la.invalid>
> wrote:
>
> Testing on an unused compute node, I tested upgrading to RockyLinux 8.5.
>
> I’m running Cloudstack 4.15.0.0.
>
> Trying to bring up cloudstack agent, I’m seeing some issues communicating
> with libvirt:
>
> 2021-12-12 21:25:03,992 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> d1b1e853-1c30-473d-badc-6c30318aa5b0 (Filesystem) in libvirt
> 2021-12-12 21:25:03,997 ERROR [kvm.resource.LibvirtConnection]
> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> invalid
> connection pointer in virConnectGetVersion
> 2021-12-12 21:25:04,000 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> d1b1e853-1c30-473d-badc-6c30318aa5b0, using it.
>
> Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> Input/output error
> Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> Input/output error
> Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> Input/output error
> Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> Input/output error
> Dec 12 21:24:26 cm02 libvirtd[269244]: Cannot recv data: Input/output
> error
>
> Libvirt version is: libvirt-6.0.0-37.module+el8.5.0+670+c4aa478c.x86_64
>
> Functional hosts that have yet to be upgraded are using:
>
> libvirt-6.0.0-35.module_el8.4.0+783+f8734d30.x86_64
>
>
> My libvirtd.conf looks like this:
>
> listen_tcp=0
> listen_tls=1
> tcp_port="16509"
> auth_tcp="none"
> mdns_adv = 0
> key_file="/etc/pki/libvirt/private/serverkey.pem"
> cert_file="/etc/pki/libvirt/servercert.pem"
> ca_file="/etc/pki/CA/cacert.pem"
> tls_port="16514"
> auth_tls=“none"
>
> 2021-12-12 21:43:42,841 ERROR [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) uefi properties file not found due to: Unable to find
> file uefi.properties.
> 2021-12-12 21:43:42,901 INFO [kvm.resource.LibvirtConnection] (main:null)
> (logid:) No existing libvirtd connection found. Opening a new one
> 2021-12-12 21:43:43,127 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> BridgeVifDriver.
> 2021-12-12 21:43:43,296 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) iscsi session clean up is disabled
> 2021-12-12 21:43:43,312 INFO [cloud.agent.Agent] (main:null) (logid:)
> Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> workers = 5 : host = 192.168.30.59 : port = 8250
> 2021-12-12 21:43:43,321 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connecting to 192.168.30.59:8250
> 2021-12-12 21:43:43,325 INFO [utils.nio.Link] (main:null) (logid:) Conf
> file found: /etc/cloudstack/agent/agent.properties
> 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> SSL: Handshake done
> 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connected to 192.168.30.59:8250
> 2021-12-12 21:43:43,925 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> 2021-12-12 21:43:43,929 ERROR [kvm.resource.LibvirtConnection]
> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> invalid
> connection pointer in virConnectGetVersion
> 2021-12-12 21:43:43,932 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> 18796842-a137-475d-9799-9874240e3c0c, using it.
> 2021-12-12 21:43:43,933 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> 18796842-a137-475d-9799-9874240e3c0c from libvirt
> 2021-12-12 21:43:43,985 INFO [cloud.serializer.GsonHelper]
> (Agent-Handler-1:null) (logid:) Default Builder inited.
> 2021-12-12 21:43:44,020 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Proccess agent startup answer, agent id = 0
> 2021-12-12 21:43:44,022 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Set agent id 0
> 2021-12-12 21:43:44,028 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Startup Response Received: agent id = 0
> 2021-12-12 21:43:44,031 INFO [cloud.agent.Agent]
> (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> 2021-12-12 21:43:55,682 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Agent started
> 2021-12-12 21:43:55,688 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Implementation Version is 4.15.0.0
> 2021-12-12 21:43:55,690 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> 2021-12-12 21:43:55,709 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to using properties file for storage
> 2021-12-12 21:43:55,711 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to the constant time backoff algorithm
> 2021-12-12 21:43:55,714 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> 2021-12-12 21:43:55,728 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Using default Java settings for IPv6 preference for agent
> connection
> 2021-12-12 21:43:55,840 INFO [cloud.agent.Agent] (main:null) (logid:) id
> is 0
> 2021-12-12 21:43:55,853 ERROR [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) uefi properties file not found due to: Unable to find
> file uefi.properties.
> 2021-12-12 21:43:55,909 INFO [kvm.resource.LibvirtConnection] (main:null)
> (logid:) No existing libvirtd connection found. Opening a new one
> 2021-12-12 21:43:56,145 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> BridgeVifDriver.
> 2021-12-12 21:43:56,307 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) iscsi session clean up is disabled
> 2021-12-12 21:43:56,322 INFO [cloud.agent.Agent] (main:null) (logid:)
> Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> workers = 5 : host = 192.168.30.59 : port = 8250
> 2021-12-12 21:43:56,331 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connecting to 192.168.30.59:8250
> 2021-12-12 21:43:56,335 INFO [utils.nio.Link] (main:null) (logid:) Conf
> file found: /etc/cloudstack/agent/agent.properties
> 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> SSL: Handshake done
> 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connected to 192.168.30.59:8250
> 2021-12-12 21:43:56,941 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> 2021-12-12 21:43:56,945 ERROR [kvm.resource.LibvirtConnection]
> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> invalid
> connection pointer in virConnectGetVersion
> 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> 18796842-a137-475d-9799-9874240e3c0c, using it.
> 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> 18796842-a137-475d-9799-9874240e3c0c from libvirt
> 2021-12-12 21:43:56,998 INFO [cloud.serializer.GsonHelper]
> (Agent-Handler-1:null) (logid:) Default Builder inited.
> 2021-12-12 21:43:57,031 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Proccess agent startup answer, agent id = 0
> 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent]
> (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Set agent id 0
> 2021-12-12 21:43:57,040 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Startup Response Received: agent id = 0
>
> Any ideas? I’m working toward upgrading the entire cluster along with
> Cloudstack itself.
>
> -jeremy
>
>
>
>
>
>

Re: Issues with libvirt after upgrading to RockyLinux 8.5

Posted by Jeremy Hansen <je...@skidrow.la.INVALID>.

I noticed in the compatibility matrix that Rocky isn’t supported until 4.16.0.0. If I upgrade Cloudstack first, would this help or is it still going to complain about the centos/rocky mix? If I convert all my existing nodes to Rocky, which is the plan anyway, will this go away? Shouldn’t CentOS and Rocky be considered that same thing… sort of…?

Thanks
-jeremy

> On Wednesday, Dec 15, 2021 at 10:43 PM, Slavka Peleva <slavkap@storpool.com.INVALID (mailto:slavkap@storpool.com.INVALID)> wrote:
> Hi Jeremy,
>
> It will help if you have another cluster for Rocky Linux. Hosts need to be
> of the same OS, it's not possible to mix OSes in the same cluster.
>
> Best regards,
> Slavka
>
> On Thu, Dec 16, 2021 at 4:08 AM Jeremy Hansen <je...@skidrow.la.invalid>
> wrote:
>
> > Any tips on how I would troubleshoot this? I’ve tried downgrading libvirt
> > and qemu and ca-certificates to the same version as the other functional
> > nodes. That didn’t seem to help. This is obviously an ssl issue but I
> > don’t really know what to do about it.
> >
> > 2021-12-15 18:04:14,438 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Agent started
> > 2021-12-15 18:04:14,444 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Implementation Version is 4.15.0.0
> > 2021-12-15 18:04:14,447 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> > 2021-12-15 18:04:14,466 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to using properties file for storage
> > 2021-12-15 18:04:14,467 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to the constant time backoff algorithm
> > 2021-12-15 18:04:14,471 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> > 2021-12-15 18:04:14,485 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Using default Java settings for IPv6 preference for agent
> > connection
> > 2021-12-15 18:04:14,592 INFO [cloud.agent.Agent] (main:null) (logid:) id
> > is 0
> > 2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > file uefi.properties.
> > 2021-12-15 18:04:14,663 INFO [kvm.resource.LibvirtConnection] (main:null)
> > (logid:) No existing libvirtd connection found. Opening a new one
> > 2021-12-15 18:04:14,890 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > BridgeVifDriver.
> > 2021-12-15 18:04:15,086 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) iscsi session clean up is disabled
> > 2021-12-15 18:04:15,129 INFO [cloud.agent.Agent] (main:null) (logid:)
> > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > workers = 5 : host = 192.168.30.59 : port = 8250
> > 2021-12-15 18:04:15,139 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connecting to 192.168.30.59:8250
> > 2021-12-15 18:04:15,153 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > file found: /etc/cloudstack/agent/agent.properties
> > 2021-12-15 18:04:15,919 INFO [utils.nio.NioClient] (main:null) (logid:)
> > SSL: Handshake done
> > 2021-12-15 18:04:15,920 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connected to 192.168.30.59:8250
> > 2021-12-15 18:04:16,057 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > 2021-12-15 18:04:16,062 ERROR [kvm.resource.LibvirtConnection]
> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid
> > connection pointer in virConnectGetVersion
> > 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > 2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > 2021-12-15 18:04:16,151 INFO [cloud.serializer.GsonHelper]
> > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > 2021-12-15 18:04:16,272 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Proccess agent startup answer, agent id = 0
> > 2021-12-15 18:04:16,273 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Set agent id 0
> > 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Startup Response Received: agent id = 0
> > 2021-12-15 18:04:16,289 INFO [cloud.agent.Agent]
> > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> >
> > I also noticed this:
> >
> > 2021-12-15 18:05:34,542 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentConnectTaskPool-19617:ctx-e0dfdb18) (logid:d3930423) Failed to handle
> > host connection:
> > java.lang.IllegalArgumentException: Can't add host: 192.168.30.54 with
> > hostOS: Rocky into a cluster,in which there are CentOS hosts added
> > at
> > com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.createHostVOForConnectedAgent(LibvirtServerDiscoverer.java:456)
> > at
> > com.cloud.resource.ResourceManagerImpl.dispatchToStateAdapters(ResourceManagerImpl.java:1719)
> > at
> > com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1946)
> > at
> > com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:2278)
> > at jdk.internal.reflect.GeneratedMethodAccessor510.invoke(Unknown Source)
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > at
> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
> > at
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> > at
> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> > at com.sun.proxy.$Proxy188.createHostVOForConnectedAgent(Unknown Source)
> > at
> > com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1097)
> > at
> > com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1194)
> > at
> > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > at
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > at
> > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> >
> > The fact that this host has been converted to Rocky Linux is causing an
> > issue? What’s the work around for this?
> >
> > Thanks
> > -jeremy
> >
> >
> >
> > On Monday, Dec 13, 2021 at 12:19 AM, Jeremy Hansen <je...@skidrow.la>
> > wrote:
> > It doesn’t error out but there is purposely no VMs running on these hosts
> > to test the upgrade of the underlying distro before having to stop any
> > active VMs.
> >
> > [jeremy@cm02 ~]$ sudo virsh list
> > Id Name State
> > --------------------
> >
> > [jeremy@cm02 ~]$
> >
> >
> > -jeremy
> >
> >
> >
> >
> > On Monday, Dec 13, 2021 at 12:06 AM, Wei ZHOU <us...@gmail.com>
> > wrote:
> > Hi,
> >
> > Do virsh commands e.g. "virsh list" work ?
> >
> > -Wei
> >
> > On Mon, 13 Dec 2021 at 06:46, Jeremy Hansen <je...@skidrow.la.invalid>
> > wrote:
> >
> > Testing on an unused compute node, I tested upgrading to RockyLinux 8.5.
> >
> > I’m running Cloudstack 4.15.0.0.
> >
> > Trying to bring up cloudstack agent, I’m seeing some issues communicating
> > with libvirt:
> >
> > 2021-12-12 21:25:03,992 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > d1b1e853-1c30-473d-badc-6c30318aa5b0 (Filesystem) in libvirt
> > 2021-12-12 21:25:03,997 ERROR [kvm.resource.LibvirtConnection]
> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> > invalid
> > connection pointer in virConnectGetVersion
> > 2021-12-12 21:25:04,000 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > d1b1e853-1c30-473d-badc-6c30318aa5b0, using it.
> >
> > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> > Input/output error
> > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> > Input/output error
> > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> > Input/output error
> > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> > Input/output error
> > Dec 12 21:24:26 cm02 libvirtd[269244]: Cannot recv data: Input/output
> > error
> >
> > Libvirt version is: libvirt-6.0.0-37.module+el8.5.0+670+c4aa478c.x86_64
> >
> > Functional hosts that have yet to be upgraded are using:
> >
> > libvirt-6.0.0-35.module_el8.4.0+783+f8734d30.x86_64
> >
> >
> > My libvirtd.conf looks like this:
> >
> > listen_tcp=0
> > listen_tls=1
> > tcp_port="16509"
> > auth_tcp="none"
> > mdns_adv = 0
> > key_file="/etc/pki/libvirt/private/serverkey.pem"
> > cert_file="/etc/pki/libvirt/servercert.pem"
> > ca_file="/etc/pki/CA/cacert.pem"
> > tls_port="16514"
> > auth_tls=“none"
> >
> > 2021-12-12 21:43:42,841 ERROR [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > file uefi.properties.
> > 2021-12-12 21:43:42,901 INFO [kvm.resource.LibvirtConnection] (main:null)
> > (logid:) No existing libvirtd connection found. Opening a new one
> > 2021-12-12 21:43:43,127 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > BridgeVifDriver.
> > 2021-12-12 21:43:43,296 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) iscsi session clean up is disabled
> > 2021-12-12 21:43:43,312 INFO [cloud.agent.Agent] (main:null) (logid:)
> > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > workers = 5 : host = 192.168.30.59 : port = 8250
> > 2021-12-12 21:43:43,321 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connecting to 192.168.30.59:8250
> > 2021-12-12 21:43:43,325 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > file found: /etc/cloudstack/agent/agent.properties
> > 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> > SSL: Handshake done
> > 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connected to 192.168.30.59:8250
> > 2021-12-12 21:43:43,925 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > 2021-12-12 21:43:43,929 ERROR [kvm.resource.LibvirtConnection]
> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> > invalid
> > connection pointer in virConnectGetVersion
> > 2021-12-12 21:43:43,932 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > 2021-12-12 21:43:43,933 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > 2021-12-12 21:43:43,985 INFO [cloud.serializer.GsonHelper]
> > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > 2021-12-12 21:43:44,020 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Proccess agent startup answer, agent id = 0
> > 2021-12-12 21:43:44,022 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Set agent id 0
> > 2021-12-12 21:43:44,028 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Startup Response Received: agent id = 0
> > 2021-12-12 21:43:44,031 INFO [cloud.agent.Agent]
> > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> > 2021-12-12 21:43:55,682 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Agent started
> > 2021-12-12 21:43:55,688 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Implementation Version is 4.15.0.0
> > 2021-12-12 21:43:55,690 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> > 2021-12-12 21:43:55,709 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to using properties file for storage
> > 2021-12-12 21:43:55,711 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to the constant time backoff algorithm
> > 2021-12-12 21:43:55,714 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> > 2021-12-12 21:43:55,728 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Using default Java settings for IPv6 preference for agent
> > connection
> > 2021-12-12 21:43:55,840 INFO [cloud.agent.Agent] (main:null) (logid:) id
> > is 0
> > 2021-12-12 21:43:55,853 ERROR [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > file uefi.properties.
> > 2021-12-12 21:43:55,909 INFO [kvm.resource.LibvirtConnection] (main:null)
> > (logid:) No existing libvirtd connection found. Opening a new one
> > 2021-12-12 21:43:56,145 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > BridgeVifDriver.
> > 2021-12-12 21:43:56,307 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) iscsi session clean up is disabled
> > 2021-12-12 21:43:56,322 INFO [cloud.agent.Agent] (main:null) (logid:)
> > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > workers = 5 : host = 192.168.30.59 : port = 8250
> > 2021-12-12 21:43:56,331 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connecting to 192.168.30.59:8250
> > 2021-12-12 21:43:56,335 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > file found: /etc/cloudstack/agent/agent.properties
> > 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> > SSL: Handshake done
> > 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connected to 192.168.30.59:8250
> > 2021-12-12 21:43:56,941 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > 2021-12-12 21:43:56,945 ERROR [kvm.resource.LibvirtConnection]
> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> > invalid
> > connection pointer in virConnectGetVersion
> > 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > 2021-12-12 21:43:56,998 INFO [cloud.serializer.GsonHelper]
> > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > 2021-12-12 21:43:57,031 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Proccess agent startup answer, agent id = 0
> > 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent]
> > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> > 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Set agent id 0
> > 2021-12-12 21:43:57,040 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Startup Response Received: agent id = 0
> >
> > Any ideas? I’m working toward upgrading the entire cluster along with
> > Cloudstack itself.
> >
> > -jeremy
> >
> >
> >
> >
> >

Re: Issues with libvirt after upgrading to RockyLinux 8.5

Posted by Slavka Peleva <sl...@storpool.com.INVALID>.

Hi Jeremy,

It will help if you have another cluster for Rocky Linux. Hosts need to be
of the same OS, it's not possible to mix OSes in the same cluster.

Best regards,
Slavka

On Thu, Dec 16, 2021 at 4:08 AM Jeremy Hansen <je...@skidrow.la.invalid>
wrote:

> Any tips on how I would troubleshoot this?  I’ve tried downgrading libvirt
> and qemu and ca-certificates to the same version as the other functional
> nodes.  That didn’t seem to help.  This is obviously an ssl issue but I
> don’t really know what to do about it.
>
> 2021-12-15 18:04:14,438 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) Agent started
> 2021-12-15 18:04:14,444 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) Implementation Version is 4.15.0.0
> 2021-12-15 18:04:14,447 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> 2021-12-15 18:04:14,466 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to using properties file for storage
> 2021-12-15 18:04:14,467 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to the constant time backoff algorithm
> 2021-12-15 18:04:14,471 INFO  [cloud.utils.LogUtils] (main:null) (logid:)
> log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> 2021-12-15 18:04:14,485 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) Using default Java settings for IPv6 preference for agent
> connection
> 2021-12-15 18:04:14,592 INFO  [cloud.agent.Agent] (main:null) (logid:) id
> is 0
> 2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) uefi properties file not found due to: Unable to find
> file uefi.properties.
> 2021-12-15 18:04:14,663 INFO  [kvm.resource.LibvirtConnection] (main:null)
> (logid:) No existing libvirtd connection found. Opening a new one
> 2021-12-15 18:04:14,890 INFO  [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> BridgeVifDriver.
> 2021-12-15 18:04:15,086 INFO  [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) iscsi session clean up is disabled
> 2021-12-15 18:04:15,129 INFO  [cloud.agent.Agent] (main:null) (logid:)
> Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> workers = 5 : host = 192.168.30.59 : port = 8250
> 2021-12-15 18:04:15,139 INFO  [utils.nio.NioClient] (main:null) (logid:)
> Connecting to 192.168.30.59:8250
> 2021-12-15 18:04:15,153 INFO  [utils.nio.Link] (main:null) (logid:) Conf
> file found: /etc/cloudstack/agent/agent.properties
> 2021-12-15 18:04:15,919 INFO  [utils.nio.NioClient] (main:null) (logid:)
> SSL: Handshake done
> 2021-12-15 18:04:15,920 INFO  [utils.nio.NioClient] (main:null) (logid:)
> Connected to 192.168.30.59:8250
> 2021-12-15 18:04:16,057 INFO  [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> 2021-12-15 18:04:16,062 ERROR [kvm.resource.LibvirtConnection]
> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid
> connection pointer in virConnectGetVersion
> 2021-12-15 18:04:16,066 INFO  [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> 18796842-a137-475d-9799-9874240e3c0c, using it.
> 2021-12-15 18:04:16,066 INFO  [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> 18796842-a137-475d-9799-9874240e3c0c from libvirt
> 2021-12-15 18:04:16,151 INFO  [cloud.serializer.GsonHelper]
> (Agent-Handler-1:null) (logid:) Default Builder inited.
> 2021-12-15 18:04:16,272 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Proccess agent startup answer, agent id = 0
> 2021-12-15 18:04:16,273 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Set agent id 0
> 2021-12-15 18:04:16,289 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Startup Response Received: agent id = 0
> 2021-12-15 18:04:16,289 INFO  [cloud.agent.Agent]
> (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
>
> I also noticed this:
>
> 2021-12-15 18:05:34,542 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentConnectTaskPool-19617:ctx-e0dfdb18) (logid:d3930423) Failed to handle
> host connection:
> java.lang.IllegalArgumentException: Can't add host: 192.168.30.54 with
> hostOS: Rocky into a cluster,in which there are CentOS hosts added
> at
> com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.createHostVOForConnectedAgent(LibvirtServerDiscoverer.java:456)
> at
> com.cloud.resource.ResourceManagerImpl.dispatchToStateAdapters(ResourceManagerImpl.java:1719)
> at
> com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1946)
> at
> com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:2278)
> at jdk.internal.reflect.GeneratedMethodAccessor510.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> at com.sun.proxy.$Proxy188.createHostVOForConnectedAgent(Unknown Source)
> at
> com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1097)
> at
> com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1194)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
>
> The fact that this host has been converted to Rocky Linux is causing an
> issue?  What’s the work around for this?
>
> Thanks
> -jeremy
>
>
>
> On Monday, Dec 13, 2021 at 12:19 AM, Jeremy Hansen <je...@skidrow.la>
> wrote:
> It doesn’t error out but there is purposely no VMs running on these hosts
> to test the upgrade of the underlying distro before having to stop any
> active VMs.
>
> [jeremy@cm02 ~]$ sudo virsh list
>  Id   Name   State
> --------------------
>
> [jeremy@cm02 ~]$
>
>
> -jeremy
>
>
>
>
> On Monday, Dec 13, 2021 at 12:06 AM, Wei ZHOU <us...@gmail.com>
> wrote:
> Hi,
>
> Do virsh commands e.g. "virsh list" work ?
>
> -Wei
>
> On Mon, 13 Dec 2021 at 06:46, Jeremy Hansen <je...@skidrow.la.invalid>
> wrote:
>
> Testing on an unused compute node, I tested upgrading to RockyLinux 8.5.
>
> I’m running Cloudstack 4.15.0.0.
>
> Trying to bring up cloudstack agent, I’m seeing some issues communicating
> with libvirt:
>
> 2021-12-12 21:25:03,992 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> d1b1e853-1c30-473d-badc-6c30318aa5b0 (Filesystem) in libvirt
> 2021-12-12 21:25:03,997 ERROR [kvm.resource.LibvirtConnection]
> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> invalid
> connection pointer in virConnectGetVersion
> 2021-12-12 21:25:04,000 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> d1b1e853-1c30-473d-badc-6c30318aa5b0, using it.
>
> Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> Input/output error
> Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> Input/output error
> Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> Input/output error
> Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> Input/output error
> Dec 12 21:24:26 cm02 libvirtd[269244]: Cannot recv data: Input/output
> error
>
> Libvirt version is: libvirt-6.0.0-37.module+el8.5.0+670+c4aa478c.x86_64
>
> Functional hosts that have yet to be upgraded are using:
>
> libvirt-6.0.0-35.module_el8.4.0+783+f8734d30.x86_64
>
>
> My libvirtd.conf looks like this:
>
> listen_tcp=0
> listen_tls=1
> tcp_port="16509"
> auth_tcp="none"
> mdns_adv = 0
> key_file="/etc/pki/libvirt/private/serverkey.pem"
> cert_file="/etc/pki/libvirt/servercert.pem"
> ca_file="/etc/pki/CA/cacert.pem"
> tls_port="16514"
> auth_tls=“none"
>
> 2021-12-12 21:43:42,841 ERROR [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) uefi properties file not found due to: Unable to find
> file uefi.properties.
> 2021-12-12 21:43:42,901 INFO [kvm.resource.LibvirtConnection] (main:null)
> (logid:) No existing libvirtd connection found. Opening a new one
> 2021-12-12 21:43:43,127 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> BridgeVifDriver.
> 2021-12-12 21:43:43,296 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) iscsi session clean up is disabled
> 2021-12-12 21:43:43,312 INFO [cloud.agent.Agent] (main:null) (logid:)
> Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> workers = 5 : host = 192.168.30.59 : port = 8250
> 2021-12-12 21:43:43,321 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connecting to 192.168.30.59:8250
> 2021-12-12 21:43:43,325 INFO [utils.nio.Link] (main:null) (logid:) Conf
> file found: /etc/cloudstack/agent/agent.properties
> 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> SSL: Handshake done
> 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connected to 192.168.30.59:8250
> 2021-12-12 21:43:43,925 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> 2021-12-12 21:43:43,929 ERROR [kvm.resource.LibvirtConnection]
> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> invalid
> connection pointer in virConnectGetVersion
> 2021-12-12 21:43:43,932 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> 18796842-a137-475d-9799-9874240e3c0c, using it.
> 2021-12-12 21:43:43,933 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> 18796842-a137-475d-9799-9874240e3c0c from libvirt
> 2021-12-12 21:43:43,985 INFO [cloud.serializer.GsonHelper]
> (Agent-Handler-1:null) (logid:) Default Builder inited.
> 2021-12-12 21:43:44,020 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Proccess agent startup answer, agent id = 0
> 2021-12-12 21:43:44,022 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Set agent id 0
> 2021-12-12 21:43:44,028 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Startup Response Received: agent id = 0
> 2021-12-12 21:43:44,031 INFO [cloud.agent.Agent]
> (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> 2021-12-12 21:43:55,682 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Agent started
> 2021-12-12 21:43:55,688 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Implementation Version is 4.15.0.0
> 2021-12-12 21:43:55,690 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> 2021-12-12 21:43:55,709 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to using properties file for storage
> 2021-12-12 21:43:55,711 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to the constant time backoff algorithm
> 2021-12-12 21:43:55,714 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> 2021-12-12 21:43:55,728 INFO [cloud.agent.AgentShell] (main:null)
> (logid:) Using default Java settings for IPv6 preference for agent
> connection
> 2021-12-12 21:43:55,840 INFO [cloud.agent.Agent] (main:null) (logid:) id
> is 0
> 2021-12-12 21:43:55,853 ERROR [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) uefi properties file not found due to: Unable to find
> file uefi.properties.
> 2021-12-12 21:43:55,909 INFO [kvm.resource.LibvirtConnection] (main:null)
> (logid:) No existing libvirtd connection found. Opening a new one
> 2021-12-12 21:43:56,145 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> BridgeVifDriver.
> 2021-12-12 21:43:56,307 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) iscsi session clean up is disabled
> 2021-12-12 21:43:56,322 INFO [cloud.agent.Agent] (main:null) (logid:)
> Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> workers = 5 : host = 192.168.30.59 : port = 8250
> 2021-12-12 21:43:56,331 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connecting to 192.168.30.59:8250
> 2021-12-12 21:43:56,335 INFO [utils.nio.Link] (main:null) (logid:) Conf
> file found: /etc/cloudstack/agent/agent.properties
> 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> SSL: Handshake done
> 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connected to 192.168.30.59:8250
> 2021-12-12 21:43:56,941 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> 2021-12-12 21:43:56,945 ERROR [kvm.resource.LibvirtConnection]
> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken:
> invalid
> connection pointer in virConnectGetVersion
> 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> 18796842-a137-475d-9799-9874240e3c0c, using it.
> 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> 18796842-a137-475d-9799-9874240e3c0c from libvirt
> 2021-12-12 21:43:56,998 INFO [cloud.serializer.GsonHelper]
> (Agent-Handler-1:null) (logid:) Default Builder inited.
> 2021-12-12 21:43:57,031 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Proccess agent startup answer, agent id = 0
> 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent]
> (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Set agent id 0
> 2021-12-12 21:43:57,040 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Startup Response Received: agent id = 0
>
> Any ideas? I’m working toward upgrading the entire cluster along with
> Cloudstack itself.
>
> -jeremy
>
>
>
>
>

Re: Issues with libvirt after upgrading to RockyLinux 8.5

Posted by Jeremy Hansen <je...@skidrow.la.INVALID>.

Any tips on how I would troubleshoot this? I’ve tried downgrading libvirt and qemu and ca-certificates to the same version as the other functional nodes. That didn’t seem to help. This is obviously an ssl issue but I don’t really know what to do about it.

2021-12-15 18:04:14,438 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2021-12-15 18:04:14,444 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.15.0.0
2021-12-15 18:04:14,447 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2021-12-15 18:04:14,466 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage
2021-12-15 18:04:14,467 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm
2021-12-15 18:04:14,471 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2021-12-15 18:04:14,485 INFO [cloud.agent.AgentShell] (main:null) (logid:) Using default Java settings for IPv6 preference for agent connection
2021-12-15 18:04:14,592 INFO [cloud.agent.Agent] (main:null) (logid:) id is 0
2021-12-15 18:04:14,606 ERROR [kvm.resource.LibvirtComputingResource] (main:null) (logid:) uefi properties file not found due to: Unable to find file uefi.properties.
2021-12-15 18:04:14,663 INFO [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one
2021-12-15 18:04:14,890 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) No libvirt.vif.driver specified. Defaults to BridgeVifDriver.
2021-12-15 18:04:15,086 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) iscsi session clean up is disabled
2021-12-15 18:04:15,129 INFO [cloud.agent.Agent] (main:null) (logid:) Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 : host = 192.168.30.59 : port = 8250
2021-12-15 18:04:15,139 INFO [utils.nio.NioClient] (main:null) (logid:) Connecting to 192.168.30.59:8250
2021-12-15 18:04:15,153 INFO [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2021-12-15 18:04:15,919 INFO [utils.nio.NioClient] (main:null) (logid:) SSL: Handshake done
2021-12-15 18:04:15,920 INFO [utils.nio.NioClient] (main:null) (logid:) Connected to 192.168.30.59:8250
2021-12-15 18:04:16,057 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
2021-12-15 18:04:16,062 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Found existing defined storage pool 18796842-a137-475d-9799-9874240e3c0c, using it.
2021-12-15 18:04:16,066 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Trying to fetch storage pool 18796842-a137-475d-9799-9874240e3c0c from libvirt
2021-12-15 18:04:16,151 INFO [cloud.serializer.GsonHelper] (Agent-Handler-1:null) (logid:) Default Builder inited.
2021-12-15 18:04:16,272 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Proccess agent startup answer, agent id = 0
2021-12-15 18:04:16,273 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Set agent id 0
2021-12-15 18:04:16,289 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Startup Response Received: agent id = 0
2021-12-15 18:04:16,289 INFO [cloud.agent.Agent] (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill

I also noticed this:

2021-12-15 18:05:34,542 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-19617:ctx-e0dfdb18) (logid:d3930423) Failed to handle host connection:
java.lang.IllegalArgumentException: Can't add host: 192.168.30.54 with hostOS: Rocky into a cluster,in which there are CentOS hosts added
at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.createHostVOForConnectedAgent(LibvirtServerDiscoverer.java:456)
at com.cloud.resource.ResourceManagerImpl.dispatchToStateAdapters(ResourceManagerImpl.java:1719)
at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:1946)
at com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:2278)
at jdk.internal.reflect.GeneratedMethodAccessor510.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy188.createHostVOForConnectedAgent(Unknown Source)
at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1097)
at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1194)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

The fact that this host has been converted to Rocky Linux is causing an issue? What’s the work around for this?

Thanks
-jeremy

> On Monday, Dec 13, 2021 at 12:19 AM, Jeremy Hansen <jeremy@skidrow.la (mailto:jeremy@skidrow.la)> wrote:
> It doesn’t error out but there is purposely no VMs running on these hosts to test the upgrade of the underlying distro before having to stop any active VMs.
>
> [jeremy@cm02 ~]$ sudo virsh list
> Id Name State
> --------------------
>
> [jeremy@cm02 ~]$
>
>
> -jeremy
>
>
>
>
> > On Monday, Dec 13, 2021 at 12:06 AM, Wei ZHOU <ustcweizhou@gmail.com (mailto:ustcweizhou@gmail.com)> wrote:
> > Hi,
> >
> > Do virsh commands e.g. "virsh list" work ?
> >
> > -Wei
> >
> > On Mon, 13 Dec 2021 at 06:46, Jeremy Hansen <je...@skidrow.la.invalid>
> > wrote:
> >
> > > Testing on an unused compute node, I tested upgrading to RockyLinux 8.5.
> > >
> > > I’m running Cloudstack 4.15.0.0.
> > >
> > > Trying to bring up cloudstack agent, I’m seeing some issues communicating
> > > with libvirt:
> > >
> > > 2021-12-12 21:25:03,992 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > > d1b1e853-1c30-473d-badc-6c30318aa5b0 (Filesystem) in libvirt
> > > 2021-12-12 21:25:03,997 ERROR [kvm.resource.LibvirtConnection]
> > > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid
> > > connection pointer in virConnectGetVersion
> > > 2021-12-12 21:25:04,000 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > > d1b1e853-1c30-473d-badc-6c30318aa5b0, using it.
> > >
> > > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> > > Input/output error
> > > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> > > Input/output error
> > > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> > > Input/output error
> > > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> > > Input/output error
> > > Dec 12 21:24:26 cm02 libvirtd[269244]: Cannot recv data: Input/output error
> > >
> > > Libvirt version is: libvirt-6.0.0-37.module+el8.5.0+670+c4aa478c.x86_64
> > >
> > > Functional hosts that have yet to be upgraded are using:
> > >
> > > libvirt-6.0.0-35.module_el8.4.0+783+f8734d30.x86_64
> > >
> > >
> > > My libvirtd.conf looks like this:
> > >
> > > listen_tcp=0
> > > listen_tls=1
> > > tcp_port="16509"
> > > auth_tcp="none"
> > > mdns_adv = 0
> > > key_file="/etc/pki/libvirt/private/serverkey.pem"
> > > cert_file="/etc/pki/libvirt/servercert.pem"
> > > ca_file="/etc/pki/CA/cacert.pem"
> > > tls_port="16514"
> > > auth_tls=“none"
> > >
> > > 2021-12-12 21:43:42,841 ERROR [kvm.resource.LibvirtComputingResource]
> > > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > > file uefi.properties.
> > > 2021-12-12 21:43:42,901 INFO [kvm.resource.LibvirtConnection] (main:null)
> > > (logid:) No existing libvirtd connection found. Opening a new one
> > > 2021-12-12 21:43:43,127 INFO [kvm.resource.LibvirtComputingResource]
> > > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > > BridgeVifDriver.
> > > 2021-12-12 21:43:43,296 INFO [kvm.resource.LibvirtComputingResource]
> > > (main:null) (logid:) iscsi session clean up is disabled
> > > 2021-12-12 21:43:43,312 INFO [cloud.agent.Agent] (main:null) (logid:)
> > > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > > workers = 5 : host = 192.168.30.59 : port = 8250
> > > 2021-12-12 21:43:43,321 INFO [utils.nio.NioClient] (main:null) (logid:)
> > > Connecting to 192.168.30.59:8250
> > > 2021-12-12 21:43:43,325 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > > file found: /etc/cloudstack/agent/agent.properties
> > > 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> > > SSL: Handshake done
> > > 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> > > Connected to 192.168.30.59:8250
> > > 2021-12-12 21:43:43,925 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > > 2021-12-12 21:43:43,929 ERROR [kvm.resource.LibvirtConnection]
> > > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid
> > > connection pointer in virConnectGetVersion
> > > 2021-12-12 21:43:43,932 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > > 2021-12-12 21:43:43,933 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > > 2021-12-12 21:43:43,985 INFO [cloud.serializer.GsonHelper]
> > > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > > 2021-12-12 21:43:44,020 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > > (logid:) Proccess agent startup answer, agent id = 0
> > > 2021-12-12 21:43:44,022 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > > (logid:) Set agent id 0
> > > 2021-12-12 21:43:44,028 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > > (logid:) Startup Response Received: agent id = 0
> > > 2021-12-12 21:43:44,031 INFO [cloud.agent.Agent]
> > > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> > > 2021-12-12 21:43:55,682 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) Agent started
> > > 2021-12-12 21:43:55,688 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) Implementation Version is 4.15.0.0
> > > 2021-12-12 21:43:55,690 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> > > 2021-12-12 21:43:55,709 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) Defaulting to using properties file for storage
> > > 2021-12-12 21:43:55,711 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) Defaulting to the constant time backoff algorithm
> > > 2021-12-12 21:43:55,714 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> > > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> > > 2021-12-12 21:43:55,728 INFO [cloud.agent.AgentShell] (main:null)
> > > (logid:) Using default Java settings for IPv6 preference for agent
> > > connection
> > > 2021-12-12 21:43:55,840 INFO [cloud.agent.Agent] (main:null) (logid:) id
> > > is 0
> > > 2021-12-12 21:43:55,853 ERROR [kvm.resource.LibvirtComputingResource]
> > > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > > file uefi.properties.
> > > 2021-12-12 21:43:55,909 INFO [kvm.resource.LibvirtConnection] (main:null)
> > > (logid:) No existing libvirtd connection found. Opening a new one
> > > 2021-12-12 21:43:56,145 INFO [kvm.resource.LibvirtComputingResource]
> > > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > > BridgeVifDriver.
> > > 2021-12-12 21:43:56,307 INFO [kvm.resource.LibvirtComputingResource]
> > > (main:null) (logid:) iscsi session clean up is disabled
> > > 2021-12-12 21:43:56,322 INFO [cloud.agent.Agent] (main:null) (logid:)
> > > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > > workers = 5 : host = 192.168.30.59 : port = 8250
> > > 2021-12-12 21:43:56,331 INFO [utils.nio.NioClient] (main:null) (logid:)
> > > Connecting to 192.168.30.59:8250
> > > 2021-12-12 21:43:56,335 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > > file found: /etc/cloudstack/agent/agent.properties
> > > 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> > > SSL: Handshake done
> > > 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> > > Connected to 192.168.30.59:8250
> > > 2021-12-12 21:43:56,941 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > > 2021-12-12 21:43:56,945 ERROR [kvm.resource.LibvirtConnection]
> > > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid
> > > connection pointer in virConnectGetVersion
> > > 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > > 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> > > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > > 2021-12-12 21:43:56,998 INFO [cloud.serializer.GsonHelper]
> > > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > > 2021-12-12 21:43:57,031 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > > (logid:) Proccess agent startup answer, agent id = 0
> > > 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent]
> > > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> > > 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > > (logid:) Set agent id 0
> > > 2021-12-12 21:43:57,040 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > > (logid:) Startup Response Received: agent id = 0
> > >
> > > Any ideas? I’m working toward upgrading the entire cluster along with
> > > Cloudstack itself.
> > >
> > > -jeremy
> > >
> > >
> > >
> > >

Re: Issues with libvirt after upgrading to RockyLinux 8.5

Posted by Jeremy Hansen <je...@skidrow.la.INVALID>.

It doesn’t error out but there is purposely no VMs running on these hosts to test the upgrade of the underlying distro before having to stop any active VMs.

[jeremy@cm02 ~]$ sudo virsh list
Id Name State
--------------------

[jeremy@cm02 ~]$

-jeremy

> On Monday, Dec 13, 2021 at 12:06 AM, Wei ZHOU <ustcweizhou@gmail.com (mailto:ustcweizhou@gmail.com)> wrote:
> Hi,
>
> Do virsh commands e.g. "virsh list" work ?
>
> -Wei
>
> On Mon, 13 Dec 2021 at 06:46, Jeremy Hansen <je...@skidrow.la.invalid>
> wrote:
>
> > Testing on an unused compute node, I tested upgrading to RockyLinux 8.5.
> >
> > I’m running Cloudstack 4.15.0.0.
> >
> > Trying to bring up cloudstack agent, I’m seeing some issues communicating
> > with libvirt:
> >
> > 2021-12-12 21:25:03,992 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > d1b1e853-1c30-473d-badc-6c30318aa5b0 (Filesystem) in libvirt
> > 2021-12-12 21:25:03,997 ERROR [kvm.resource.LibvirtConnection]
> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid
> > connection pointer in virConnectGetVersion
> > 2021-12-12 21:25:04,000 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > d1b1e853-1c30-473d-badc-6c30318aa5b0, using it.
> >
> > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> > Input/output error
> > Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> > Input/output error
> > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> > Input/output error
> > Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> > Input/output error
> > Dec 12 21:24:26 cm02 libvirtd[269244]: Cannot recv data: Input/output error
> >
> > Libvirt version is: libvirt-6.0.0-37.module+el8.5.0+670+c4aa478c.x86_64
> >
> > Functional hosts that have yet to be upgraded are using:
> >
> > libvirt-6.0.0-35.module_el8.4.0+783+f8734d30.x86_64
> >
> >
> > My libvirtd.conf looks like this:
> >
> > listen_tcp=0
> > listen_tls=1
> > tcp_port="16509"
> > auth_tcp="none"
> > mdns_adv = 0
> > key_file="/etc/pki/libvirt/private/serverkey.pem"
> > cert_file="/etc/pki/libvirt/servercert.pem"
> > ca_file="/etc/pki/CA/cacert.pem"
> > tls_port="16514"
> > auth_tls=“none"
> >
> > 2021-12-12 21:43:42,841 ERROR [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > file uefi.properties.
> > 2021-12-12 21:43:42,901 INFO [kvm.resource.LibvirtConnection] (main:null)
> > (logid:) No existing libvirtd connection found. Opening a new one
> > 2021-12-12 21:43:43,127 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > BridgeVifDriver.
> > 2021-12-12 21:43:43,296 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) iscsi session clean up is disabled
> > 2021-12-12 21:43:43,312 INFO [cloud.agent.Agent] (main:null) (logid:)
> > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > workers = 5 : host = 192.168.30.59 : port = 8250
> > 2021-12-12 21:43:43,321 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connecting to 192.168.30.59:8250
> > 2021-12-12 21:43:43,325 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > file found: /etc/cloudstack/agent/agent.properties
> > 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> > SSL: Handshake done
> > 2021-12-12 21:43:43,840 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connected to 192.168.30.59:8250
> > 2021-12-12 21:43:43,925 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > 2021-12-12 21:43:43,929 ERROR [kvm.resource.LibvirtConnection]
> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid
> > connection pointer in virConnectGetVersion
> > 2021-12-12 21:43:43,932 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > 2021-12-12 21:43:43,933 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > 2021-12-12 21:43:43,985 INFO [cloud.serializer.GsonHelper]
> > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > 2021-12-12 21:43:44,020 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Proccess agent startup answer, agent id = 0
> > 2021-12-12 21:43:44,022 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Set agent id 0
> > 2021-12-12 21:43:44,028 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Startup Response Received: agent id = 0
> > 2021-12-12 21:43:44,031 INFO [cloud.agent.Agent]
> > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> > 2021-12-12 21:43:55,682 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Agent started
> > 2021-12-12 21:43:55,688 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Implementation Version is 4.15.0.0
> > 2021-12-12 21:43:55,690 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> > 2021-12-12 21:43:55,709 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to using properties file for storage
> > 2021-12-12 21:43:55,711 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Defaulting to the constant time backoff algorithm
> > 2021-12-12 21:43:55,714 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> > 2021-12-12 21:43:55,728 INFO [cloud.agent.AgentShell] (main:null)
> > (logid:) Using default Java settings for IPv6 preference for agent
> > connection
> > 2021-12-12 21:43:55,840 INFO [cloud.agent.Agent] (main:null) (logid:) id
> > is 0
> > 2021-12-12 21:43:55,853 ERROR [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) uefi properties file not found due to: Unable to find
> > file uefi.properties.
> > 2021-12-12 21:43:55,909 INFO [kvm.resource.LibvirtConnection] (main:null)
> > (logid:) No existing libvirtd connection found. Opening a new one
> > 2021-12-12 21:43:56,145 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> > BridgeVifDriver.
> > 2021-12-12 21:43:56,307 INFO [kvm.resource.LibvirtComputingResource]
> > (main:null) (logid:) iscsi session clean up is disabled
> > 2021-12-12 21:43:56,322 INFO [cloud.agent.Agent] (main:null) (logid:)
> > Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> > workers = 5 : host = 192.168.30.59 : port = 8250
> > 2021-12-12 21:43:56,331 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connecting to 192.168.30.59:8250
> > 2021-12-12 21:43:56,335 INFO [utils.nio.Link] (main:null) (logid:) Conf
> > file found: /etc/cloudstack/agent/agent.properties
> > 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> > SSL: Handshake done
> > 2021-12-12 21:43:56,843 INFO [utils.nio.NioClient] (main:null) (logid:)
> > Connected to 192.168.30.59:8250
> > 2021-12-12 21:43:56,941 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> > 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> > 2021-12-12 21:43:56,945 ERROR [kvm.resource.LibvirtConnection]
> > (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid
> > connection pointer in virConnectGetVersion
> > 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> > 18796842-a137-475d-9799-9874240e3c0c, using it.
> > 2021-12-12 21:43:56,948 INFO [kvm.storage.LibvirtStorageAdaptor]
> > (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> > 18796842-a137-475d-9799-9874240e3c0c from libvirt
> > 2021-12-12 21:43:56,998 INFO [cloud.serializer.GsonHelper]
> > (Agent-Handler-1:null) (logid:) Default Builder inited.
> > 2021-12-12 21:43:57,031 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Proccess agent startup answer, agent id = 0
> > 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent]
> > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> > 2021-12-12 21:43:57,033 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Set agent id 0
> > 2021-12-12 21:43:57,040 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
> > (logid:) Startup Response Received: agent id = 0
> >
> > Any ideas? I’m working toward upgrading the entire cluster along with
> > Cloudstack itself.
> >
> > -jeremy
> >
> >
> >
> >

Re: Issues with libvirt after upgrading to RockyLinux 8.5

Posted by Wei ZHOU <us...@gmail.com>.

Hi,

Do virsh commands e.g. "virsh list" work ?

-Wei

On Mon, 13 Dec 2021 at 06:46, Jeremy Hansen <je...@skidrow.la.invalid>
wrote:

> Testing on an unused compute node, I tested upgrading to RockyLinux 8.5.
>
> I’m running Cloudstack 4.15.0.0.
>
> Trying to bring up cloudstack agent, I’m seeing some issues communicating
> with libvirt:
>
> 2021-12-12 21:25:03,992 INFO  [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> d1b1e853-1c30-473d-badc-6c30318aa5b0 (Filesystem) in libvirt
> 2021-12-12 21:25:03,997 ERROR [kvm.resource.LibvirtConnection]
> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid
> connection pointer in virConnectGetVersion
> 2021-12-12 21:25:04,000 INFO  [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> d1b1e853-1c30-473d-badc-6c30318aa5b0, using it.
>
> Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> Input/output error
> Dec 12 21:24:13 cm02 libvirtd[269244]: End of file while reading data:
> Input/output error
> Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> Input/output error
> Dec 12 21:24:26 cm02 libvirtd[269244]: End of file while reading data:
> Input/output error
> Dec 12 21:24:26 cm02 libvirtd[269244]: Cannot recv data: Input/output error
>
> Libvirt version is: libvirt-6.0.0-37.module+el8.5.0+670+c4aa478c.x86_64
>
> Functional hosts that have yet to be upgraded are using:
>
> libvirt-6.0.0-35.module_el8.4.0+783+f8734d30.x86_64
>
>
> My libvirtd.conf looks like this:
>
> listen_tcp=0
> listen_tls=1
> tcp_port="16509"
> auth_tcp="none"
> mdns_adv = 0
> key_file="/etc/pki/libvirt/private/serverkey.pem"
> cert_file="/etc/pki/libvirt/servercert.pem"
> ca_file="/etc/pki/CA/cacert.pem"
> tls_port="16514"
> auth_tls=“none"
>
> 2021-12-12 21:43:42,841 ERROR [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) uefi properties file not found due to: Unable to find
> file uefi.properties.
> 2021-12-12 21:43:42,901 INFO  [kvm.resource.LibvirtConnection] (main:null)
> (logid:) No existing libvirtd connection found. Opening a new one
> 2021-12-12 21:43:43,127 INFO  [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> BridgeVifDriver.
> 2021-12-12 21:43:43,296 INFO  [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) iscsi session clean up is disabled
> 2021-12-12 21:43:43,312 INFO  [cloud.agent.Agent] (main:null) (logid:)
> Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> workers = 5 : host = 192.168.30.59 : port = 8250
> 2021-12-12 21:43:43,321 INFO  [utils.nio.NioClient] (main:null) (logid:)
> Connecting to 192.168.30.59:8250
> 2021-12-12 21:43:43,325 INFO  [utils.nio.Link] (main:null) (logid:) Conf
> file found: /etc/cloudstack/agent/agent.properties
> 2021-12-12 21:43:43,840 INFO  [utils.nio.NioClient] (main:null) (logid:)
> SSL: Handshake done
> 2021-12-12 21:43:43,840 INFO  [utils.nio.NioClient] (main:null) (logid:)
> Connected to 192.168.30.59:8250
> 2021-12-12 21:43:43,925 INFO  [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> 2021-12-12 21:43:43,929 ERROR [kvm.resource.LibvirtConnection]
> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid
> connection pointer in virConnectGetVersion
> 2021-12-12 21:43:43,932 INFO  [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> 18796842-a137-475d-9799-9874240e3c0c, using it.
> 2021-12-12 21:43:43,933 INFO  [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> 18796842-a137-475d-9799-9874240e3c0c from libvirt
> 2021-12-12 21:43:43,985 INFO  [cloud.serializer.GsonHelper]
> (Agent-Handler-1:null) (logid:) Default Builder inited.
> 2021-12-12 21:43:44,020 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Proccess agent startup answer, agent id = 0
> 2021-12-12 21:43:44,022 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Set agent id 0
> 2021-12-12 21:43:44,028 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Startup Response Received: agent id = 0
> 2021-12-12 21:43:44,031 INFO  [cloud.agent.Agent]
> (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> 2021-12-12 21:43:55,682 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) Agent started
> 2021-12-12 21:43:55,688 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) Implementation Version is 4.15.0.0
> 2021-12-12 21:43:55,690 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
> 2021-12-12 21:43:55,709 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to using properties file for storage
> 2021-12-12 21:43:55,711 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) Defaulting to the constant time backoff algorithm
> 2021-12-12 21:43:55,714 INFO  [cloud.utils.LogUtils] (main:null) (logid:)
> log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> 2021-12-12 21:43:55,728 INFO  [cloud.agent.AgentShell] (main:null)
> (logid:) Using default Java settings for IPv6 preference for agent
> connection
> 2021-12-12 21:43:55,840 INFO  [cloud.agent.Agent] (main:null) (logid:) id
> is 0
> 2021-12-12 21:43:55,853 ERROR [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) uefi properties file not found due to: Unable to find
> file uefi.properties.
> 2021-12-12 21:43:55,909 INFO  [kvm.resource.LibvirtConnection] (main:null)
> (logid:) No existing libvirtd connection found. Opening a new one
> 2021-12-12 21:43:56,145 INFO  [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> BridgeVifDriver.
> 2021-12-12 21:43:56,307 INFO  [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) iscsi session clean up is disabled
> 2021-12-12 21:43:56,322 INFO  [cloud.agent.Agent] (main:null) (logid:)
> Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 :
> workers = 5 : host = 192.168.30.59 : port = 8250
> 2021-12-12 21:43:56,331 INFO  [utils.nio.NioClient] (main:null) (logid:)
> Connecting to 192.168.30.59:8250
> 2021-12-12 21:43:56,335 INFO  [utils.nio.Link] (main:null) (logid:) Conf
> file found: /etc/cloudstack/agent/agent.properties
> 2021-12-12 21:43:56,843 INFO  [utils.nio.NioClient] (main:null) (logid:)
> SSL: Handshake done
> 2021-12-12 21:43:56,843 INFO  [utils.nio.NioClient] (main:null) (logid:)
> Connected to 192.168.30.59:8250
> 2021-12-12 21:43:56,941 INFO  [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Attempting to create storage pool
> 18796842-a137-475d-9799-9874240e3c0c (Filesystem) in libvirt
> 2021-12-12 21:43:56,945 ERROR [kvm.resource.LibvirtConnection]
> (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid
> connection pointer in virConnectGetVersion
> 2021-12-12 21:43:56,948 INFO  [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Found existing defined storage pool
> 18796842-a137-475d-9799-9874240e3c0c, using it.
> 2021-12-12 21:43:56,948 INFO  [kvm.storage.LibvirtStorageAdaptor]
> (Agent-Handler-1:null) (logid:) Trying to fetch storage pool
> 18796842-a137-475d-9799-9874240e3c0c from libvirt
> 2021-12-12 21:43:56,998 INFO  [cloud.serializer.GsonHelper]
> (Agent-Handler-1:null) (logid:) Default Builder inited.
> 2021-12-12 21:43:57,031 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Proccess agent startup answer, agent id = 0
> 2021-12-12 21:43:57,033 INFO  [cloud.agent.Agent]
> (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
> 2021-12-12 21:43:57,033 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Set agent id 0
> 2021-12-12 21:43:57,040 INFO  [cloud.agent.Agent] (Agent-Handler-2:null)
> (logid:) Startup Response Received: agent id = 0
>
> Any ideas?  I’m working toward upgrading the entire cluster along with
> Cloudstack itself.
>
> -jeremy
>
>
>
>