You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "Janis Viklis | Files.fm" <ja...@files.fm> on 2024/08/03 13:02:45 UTC
Re: All 4 hosts disconnected in Alert state due to ComputeCapacityListener NULL: how to fix?
Solution after 2 moths of random experimenting
1) if you free up host from any vms or stop them, it gets back as UP,
without any action
2) if you restart cloudstack-management server, all hosts goes to Alert
state...
Total resolution: UPDATE cloud.vm_instance SET host_id = NULL where id >
0; Hosts comes UP and CS displays real count of vms running.
Janis
On 2024-07-01 21:44, Janis Viklis | Files.fm wrote:
> Hi,
>
> looking for help after 2 weeks: What could be the reason that
> suddenly after restarting the 4.13.1 Management server, all 4 XEN
> (xcp-ng 8.1) hosts of one Intel cluster disconnects and goes into
> "Alert state" with an error:
>
> Monitor ComputeCapacityListener says there is an error in the connect
> process for 248 due to null
>
> I can't find the reason for 2 weeks. The other AMD Xenserver 6.5
> cluster is working just fine.
>
> Everything seems ok: network is working, I restarted: toolstack, both
> system vms (SSVM, consolev), one of the hosts, then removed and added
> back.
>
> Previously there were 3 management servers via Haproxy and Galera
> Mariadb, I left only one. (tried upgrade to 3.14.1, didn't help). I
> can manage hosts via Xencenter. There ar 5 storage pools and 3 secondary.
>
> Thanks, hoping on some clues or directions, Janis.
>
> Below is LOG output:
>
> [root@cs1 ~]# cat
> /var/log/cloudstack/management/management-server.log | grep ctx-f411cc14
> 2024-07-01 16:31:25,641 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Simulating start for
> resource 5ed7f899-a267-4278-b06d-5e2bc8fea685 id 248
> 2024-07-01 16:31:25,641 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Creating agent for
> host 248
> 2024-07-01 16:31:25,802 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Management network
> is on pif=cdb44e56-6c0b-32ab-9113-ae9c8dc5a93f
> 2024-07-01 16:31:25,816 DEBUG [c.c.h.x.r.XsLocalNetwork]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Found a network
> called Management on host=10.10.10.131;
> Network=f94c6cef-2a03-f964-b272 -a214b6abdaa8;
> pif=cdb44e56-6c0b-32ab-9113-ae9c8dc5a93f
> 2024-07-01 16:31:25,816 INFO [c.c.h.x.r.CitrixResourceBase]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) XenServer Version is
> 8.1.0 for host 10.10.10.131
> 2024-07-01 16:31:25,817 INFO [c.c.h.x.r.CitrixResourceBase]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Private Network is
> Management for host 10.10.10.131
> 2024-07-01 16:31:25,817 INFO [c.c.h.x.r.CitrixResourceBase]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Guest Network is
> Management for host 10.10.10.131
> 2024-07-01 16:31:25,817 INFO [c.c.h.x.r.CitrixResourceBase]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Public Network is
> Management for host 10.10.10.131
> 2024-07-01 16:31:25,831 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Total Ram:
> 257724032384 dom0 Ram: 8589934592
> 2024-07-01 16:31:26,103 ERROR [c.c.u.s.SshHelper]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) SSH execution of
> command xe sm-list | grep "resigning of duplicates" has an error
> status code i n return. Result output:
> 2024-07-01 16:31:26,113 DEBUG [c.c.h.x.r.XenServer620Resource]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) can't get patches
> information for hotFix: 996dd2e7-ad95-49cc-a0be-2c9adc4dfb0b
> 2024-07-01 16:31:26,122 DEBUG [c.c.h.x.r.XenServer620Resource]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) can't get patches
> information for hotFix: 0850b186-4d47-11e3-a720-001b2151a503
> 2024-07-01 16:31:26,165 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Found 0 local
> storage of type [lvm] for host [5ed7f899-a267-4278-b06d-5e2bc8fea685]
> 2024-07-01 16:31:26,171 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Found 0 local
> storage of type [ext] for host [5ed7f899-a267-4278-b06d-5e2bc8fea685]
> 2024-07-01 16:31:26,184 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq -1--1: Startup
> request from directly connected host: { Cmd , MgmtId: -1, via: -1,
> Ver: v1, F lags: 11,
> [{"cpuSockets":2,"cpus":48,"speed":2500,"memory":257724032384,"dom0MinMemory":8589934592,"poolSync":false,"supportsClonedVolumes":false,"caps":"xen-3.0-x86_64
> , xen-3.0-x86_32p , hvm-3. 0-x86_32 , hvm-3.0-x86_32p ,
> hvm-3.0-x86_64","pool":"3e71abd3-f524-1cbd-da4d-47d0b77d85a3","hypervisorType":"XenServer","hostDetails":{"product_version":"8.1.0","com.cloud.network.Networks.Router
> PrivateIpStrategy":"DcGlobal","product_version_text_short":"8.1","private.network.device":"Management","product_brand":"XCP-ng"},"hostTags":["GPU"],"hypervisorVersion":"8.1.0","groupDetails":{"Gr
> oup of ASPEED Technology, Inc. ASPEED Graphics Family
> GPUs":{}},"type":"Routing","dataCenter":"9","pod":"12","cluster":"29","guid":"5ed7f899-a267-4278-b06d-5e2bc8fea685","name":"xs31.failiem.lv",
> "version":"4.13.1.0","iqn":"iqn.2020-06.lv.failiem:7264f035","publicIpAddress":"10.10.10.131","publicNetmask":"255.255.255.0","publicMacAddress":"0c:c4:7a:bb:cb:22","privateIpAddress":"10.10.10.1
> 31","privateMacAddress":"0c:c4:7a:bb:cb:22","privateNetmask":"255.255.255.0","storageIpAddress":"10.10.10.131","storageNetmask":"255.255.255.0","storageMacAddress":"0c:c4:7a:bb:cb:22","wait":0}]
> }
> 2024-07-01 16:31:26,198 DEBUG [c.c.r.ResourceManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Dispatching resource
> state event CREATE_HOST_VO_FOR_DIRECT_CONNECT to CiscoVnmcElement
> 2024-07-01 16:31:26,198 DEBUG [c.c.r.ResourceManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Dispatching resource
> state event CREATE_HOST_VO_FOR_DIRECT_CONNECT to
> BareMetalDiscover er
> 2024-07-01 16:31:26,198 DEBUG [c.c.r.ResourceManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Dispatching resource
> state event CREATE_HOST_VO_FOR_DIRECT_CONNECT to NetscalerElement
> 2024-07-01 16:31:26,198 DEBUG [c.c.r.ResourceManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Dispatching resource
> state event CREATE_HOST_VO_FOR_DIRECT_CONNECT to
> JuniperSRXExterna lFirewallElement
> 2024-07-01 16:31:26,198 DEBUG [c.c.r.ResourceManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Dispatching resource
> state event CREATE_HOST_VO_FOR_DIRECT_CONNECT to
> HypervServerDisco verer
> 2024-07-01 16:31:26,198 DEBUG [c.c.r.ResourceManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Dispatching resource
> state event CREATE_HOST_VO_FOR_DIRECT_CONNECT to
> BaremetalPxeManag erImpl
> 2024-07-01 16:31:26,198 DEBUG [c.c.r.ResourceManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Dispatching resource
> state event CREATE_HOST_VO_FOR_DIRECT_CONNECT to
> XcpServerDiscover er
> 2024-07-01 16:31:26,202 INFO [c.c.h.x.d.XcpServerDiscoverer]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Host:
> xs31.failiem.lv connected with hypervisor type: XenServer. Checking
> CIDR...
> 2024-07-01 16:31:26,263 DEBUG [c.c.r.ResourceState]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Resource state
> update: [id = 248; name = xs31.failiem.lv; old state = Enabled; event
> = Intern alCreated; new state = Enabled]
> 2024-07-01 16:31:26,263 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Transition:[Resource
> state = Enabled, Agent event = AgentConnected, Host id = 248,
> name = xs31.failiem.lv]
> 2024-07-01 16:31:26,270 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) create
> ClusteredDirectAgentAttache for 248
> 2024-07-01 16:31:26,270 INFO [c.c.a.m.DirectAgentAttache]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) StartupAnswer
> received 248 Interval = 60
> 2024-07-01 16:31:26,273 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Connect to
> listener: XcpServerDiscoverer
> 2024-07-01 16:31:26,279 WARN [c.c.h.x.d.XcpServerDiscoverer]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) defaulting to
> xenserver650 resource for product brand: XCP-ng with product
> version: 8.1.0
> 2024-07-01 16:31:26,279 DEBUG [c.c.h.x.d.XcpServerDiscoverer]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Setting up host 248
> 2024-07-01 16:31:26,289 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381185: Sending { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver : v1, Flags: 100111,
> [{"com.cloud.agent.api.SetupCommand":{"env":{},"multipath":false,"needSetup":false,"wait":0}}]
> }
> 2024-07-01 16:31:26,289 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381185: Executing: { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.SetupCommand":{"env":{},"multipath":false,"needSetup":false,"wait":0}}]
> }
> 2024-07-01 16:31:29,517 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381185: Received: { Ans: , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver: v1, Flags: 110, { SetupAnswer } }
> 2024-07-01 16:31:29,524 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Connect to
> listener: HypervServerDiscoverer
> 2024-07-01 16:31:29,524 DEBUG [c.c.h.h.d.HypervServerDiscoverer]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Not Hyper-V
> hypervisor, so moving on.
> 2024-07-01 16:31:29,524 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Connect to
> listener: VmwareManagerImpl
> 2024-07-01 16:31:29,524 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Connect to
> listener: ClusteredVirtualMachineManagerImpl
> 2024-07-01 16:31:29,524 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Received startup
> command from hypervisor host. host id: 248
> 2024-07-01 16:31:29,524 INFO [c.c.v.VirtualMachinePowerStateSyncImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Reset VM power state
> sync for host: 248
> 2024-07-01 16:31:29,539 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381186: Sending { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver : v1, Flags: 100011,
> [{"com.cloud.agent.api.ClusterVMMetaDataSyncCommand":{"_interval":180,"_clusterId":29,"wait":0}}]
> }
> 2024-07-01 16:31:29,539 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381186: Executing: { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.ClusterVMMetaDataSyncCommand":{"_interval":180,"_clusterId":29,"wait":0}}]
> }
> 2024-07-01 16:31:29,539 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Cluster VM metadata
> sync started with jobid 1798343626204381186
> 2024-07-01 16:31:29,539 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Connect to
> listener: NetworkOrchestrator
> 2024-07-01 16:31:29,546 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Host's
> hypervisorType is: XenServer
> 2024-07-01 16:31:29,581 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending
> CheckNetworkCommand to check the Network is setup correctly on Agent
> 2024-07-01 16:31:29,594 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381187: Sending { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver : v1, Flags: 100111,
> [{"com.cloud.agent.api.CheckNetworkCommand":{"networkInfoList":[{"physicalNetworkId":202,"privateNetworkName":"Management","storageNetworkName":"Management"},{"physicalNetwor
> kId":203,"publicNetworkName":"Public","guestNetworkName":"Public"}],"wait":0}}]
> }
> 2024-07-01 16:31:29,594 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381187: Executing: { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.CheckNetworkCommand":{"networkInfoList":[{"physicalNetworkId":202,"privateNetworkName":"Management","storageNetworkName":"Management"},{"physicalNet
> workId":203,"publicNetworkName":"Public","guestNetworkName":"Public"}],"wait":0}}]
> }
> 2024-07-01 16:31:29,605 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381187: Received: { Ans: , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver: v1, Flags: 110, {
> CheckNetworkAnswer } }
> 2024-07-01 16:31:29,605 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Details from
> executing class com.cloud.agent.api.CheckNetworkCommand: Network Setup
> chec k by names is done
> 2024-07-01 16:31:29,605 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Network setup is
> correct on Agent
> 2024-07-01 16:31:29,605 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Connect to
> listener: StoragePoolMonitor
> 2024-07-01 16:31:29,617 DEBUG [c.c.s.l.StoragePoolMonitor]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Host 248 connected,
> connecting host to shared pool id 152 and sending storage pool
> inf ormation ...
> 2024-07-01 16:31:29,620 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Adding pool null to
> host 248
> 2024-07-01 16:31:29,635 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381188: Sending { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver : v1, Flags: 100011,
> [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":true,"pool":{"id":152,"uuid":"47c142c0-770c-32b0-ba43-20e0fa2d7f9e","host":"10.10.10.41","path":"/xs3-SSD","port":2049
> ,"type":"NetworkFilesystem"},"localPath":"/mnt//47c142c0-770c-32b0-ba43-20e0fa2d7f9e","wait":0}}]
> }
> 2024-07-01 16:31:29,636 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381188: Executing: { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":true,"pool":{"id":152,"uuid":"47c142c0-770c-32b0-ba43-20e0fa2d7f9e","host":"10.10.10.41","path":"/xs3-SSD","port":2
> 049,"type":"NetworkFilesystem"},"localPath":"/mnt//47c142c0-770c-32b0-ba43-20e0fa2d7f9e","wait":0}}]
> }
> 2024-07-01 16:31:29,839 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381188: Received: { Ans: , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver: v1, Flags: 10, {
> ModifyStoragePoolAnswer } }
> 2024-07-01 16:31:29,863 INFO [o.a.c.s.d.p.DefaultHostListener]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Connection
> established between storage pool org.apache.cloudstack.storage.datasto
> re.PrimaryDataStoreImpl@7df02a59 and host 248
> 2024-07-01 16:31:29,872 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Found storage pool
> csstorage2-xs3-SSD of type NetworkFilesystem with overprovisioning
> fa ctor 2
> 2024-07-01 16:31:29,872 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Total over
> provisioned capacity calculated is 2 * 3829052473344
> 2024-07-01 16:31:29,872 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Total over
> provisioned capacity of the pool csstorage2-xs3-SSD id: 152 is
> 7658104946688
> 2024-07-01 16:31:29,877 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Successfully set
> Capacity - 7658104946688 for capacity type - 3 , DataCenterId - 9,
> Host OrPoolId - 152, PodId 12
> 2024-07-01 16:31:29,877 DEBUG [c.c.s.l.StoragePoolMonitor]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Host 248 connected,
> connecting host to shared pool id 158 and sending storage pool
> inf ormation ...
> 2024-07-01 16:31:29,879 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Adding pool null to
> host 248
> 2024-07-01 16:31:29,888 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381189: Sending { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver : v1, Flags: 100011,
> [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":true,"pool":{"id":158,"uuid":"56d6552e-b8a8-3f8b-b218-49b12f5443db","host":"csstorage3.failiem.lv","path":"/primary-NV
> M/xs3","port":2049,"type":"NetworkFilesystem"},"localPath":"/mnt//56d6552e-b8a8-3f8b-b218-49b12f5443db","wait":0}}]
> }
> 2024-07-01 16:31:29,888 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381189: Executing: { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":true,"pool":{"id":158,"uuid":"56d6552e-b8a8-3f8b-b218-49b12f5443db","host":"csstorage3.failiem.lv","path":"/primary
> -NVM/xs3","port":2049,"type":"NetworkFilesystem"},"localPath":"/mnt//56d6552e-b8a8-3f8b-b218-49b12f5443db","wait":0}}]
> }
> 2024-07-01 16:31:30,086 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381189: Received: { Ans: , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver: v1, Flags: 10, {
> ModifyStoragePoolAnswer } }
> 2024-07-01 16:31:30,094 INFO [o.a.c.s.d.p.DefaultHostListener]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Connection
> established between storage pool org.apache.cloudstack.storage.datasto
> re.PrimaryDataStoreImpl@42d3c9d9 and host 248
> 2024-07-01 16:31:30,104 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Found storage pool
> csstorage3-xs3-NVM of type NetworkFilesystem with overprovisioning
> fa ctor 2
> 2024-07-01 16:31:30,104 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Total over
> provisioned capacity calculated is 2 * 1362611929088
> 2024-07-01 16:31:30,104 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Total over
> provisioned capacity of the pool csstorage3-xs3-NVM id: 158 is
> 2725223858176
> 2024-07-01 16:31:30,109 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Successfully set
> Capacity - 2725223858176 for capacity type - 3 , DataCenterId - 9,
> Host OrPoolId - 158, PodId 12
> 2024-07-01 16:31:30,109 DEBUG [c.c.s.l.StoragePoolMonitor]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Host 248 connected,
> connecting host to shared pool id 160 and sending storage pool
> inf ormation ...
> 2024-07-01 16:31:30,111 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Adding pool null to
> host 248
> 2024-07-01 16:31:30,121 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381190: Sending { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver : v1, Flags: 100011,
> [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":true,"pool":{"id":160,"uuid":"f1af685f-f357-390e-9224-84feecd41343","host":"csstorage4.failiem.lv","path":"/primary-SA
> TA2-XS3","port":2049,"type":"NetworkFilesystem"},"localPath":"/mnt//f1af685f-f357-390e-9224-84feecd41343","wait":0}}]
> }
> 2024-07-01 16:31:30,121 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381190: Executing: { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":true,"pool":{"id":160,"uuid":"f1af685f-f357-390e-9224-84feecd41343","host":"csstorage4.failiem.lv","path":"/primary
> -SATA2-XS3","port":2049,"type":"NetworkFilesystem"},"localPath":"/mnt//f1af685f-f357-390e-9224-84feecd41343","wait":0}}]
> }
> 2024-07-01 16:31:30,322 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381190: Received: { Ans: , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver: v1, Flags: 10, {
> ModifyStoragePoolAnswer } }
> 2024-07-01 16:31:30,330 INFO [o.a.c.s.d.p.DefaultHostListener]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Connection
> established between storage pool org.apache.cloudstack.storage.datasto
> re.PrimaryDataStoreImpl@306585d2 and host 248
> 2024-07-01 16:31:30,339 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Found storage pool
> csstorage4-xs3-SATA of type NetworkFilesystem with overprovisioning
> f actor 2
> 2024-07-01 16:31:30,339 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Total over
> provisioned capacity calculated is 2 * 59766547152896
> 2024-07-01 16:31:30,339 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Total over
> provisioned capacity of the pool csstorage4-xs3-SATA id: 160 is
> 1195330943057 92
> 2024-07-01 16:31:30,343 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Successfully set
> Capacity - 119533094305792 for capacity type - 3 , DataCenterId - 9,
> Ho stOrPoolId - 160, PodId 12
> 2024-07-01 16:31:30,343 DEBUG [c.c.s.l.StoragePoolMonitor]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Host 248 connected,
> connecting host to shared pool id 166 and sending storage pool
> inf ormation ...
> 2024-07-01 16:31:30,346 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Adding pool null to
> host 248
> 2024-07-01 16:31:30,354 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381191: Sending { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver : v1, Flags: 100011,
> [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":true,"pool":{"id":166,"uuid":"e55b8062-4a38-3842-99ea-b1c18909e0e6","host":"10.10.10.103","path":"/primary-SSD/xs3","p
> ort":2049,"type":"NetworkFilesystem"},"localPath":"/mnt//e55b8062-4a38-3842-99ea-b1c18909e0e6","wait":0}}]
> }
> 2024-07-01 16:31:30,354 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381191: Executing: { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":true,"pool":{"id":166,"uuid":"e55b8062-4a38-3842-99ea-b1c18909e0e6","host":"10.10.10.103","path":"/primary-SSD/xs3"
> ,"port":2049,"type":"NetworkFilesystem"},"localPath":"/mnt//e55b8062-4a38-3842-99ea-b1c18909e0e6","wait":0}}]
> }
> 2024-07-01 16:31:30,550 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381191: Received: { Ans: , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver: v1, Flags: 10, {
> ModifyStoragePoolAnswer } }
> 2024-07-01 16:31:30,558 INFO [o.a.c.s.d.p.DefaultHostListener]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Connection
> established between storage pool org.apache.cloudstack.storage.datasto
> re.PrimaryDataStoreImpl@1c98de8a and host 248
> 2024-07-01 16:31:30,567 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Found storage pool
> csstorage3-xs3-SSD of type NetworkFilesystem with overprovisioning
> fa ctor 2
> 2024-07-01 16:31:30,567 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Total over
> provisioned capacity calculated is 2 * 5753096110080
> 2024-07-01 16:31:30,567 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Total over
> provisioned capacity of the pool csstorage3-xs3-SSD id: 166 is
> 11506192220160
> 2024-07-01 16:31:30,571 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Successfully set
> Capacity - 11506192220160 for capacity type - 3 , DataCenterId - 9,
> Hos tOrPoolId - 166, PodId 12
> 2024-07-01 16:31:30,571 DEBUG [c.c.s.l.StoragePoolMonitor]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Host 248 connected,
> connecting host to shared pool id 169 and sending storage pool
> inf ormation ...
> 2024-07-01 16:31:30,573 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Adding pool null to
> host 248
> 2024-07-01 16:31:30,581 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381192: Sending { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver : v1, Flags: 100011,
> [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":true,"pool":{"id":169,"uuid":"25418de5-a6fb-3026-a983-c6a2d8f61322","host":"csstorage5.failiem.lv","path":"/primary-NV
> Me","port":2049,"type":"NetworkFilesystem"},"localPath":"/mnt//25418de5-a6fb-3026-a983-c6a2d8f61322","wait":0}}]
> }
> 2024-07-01 16:31:30,581 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381192: Executing: { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":true,"pool":{"id":169,"uuid":"25418de5-a6fb-3026-a983-c6a2d8f61322","host":"csstorage5.failiem.lv","path":"/primary
> -NVMe","port":2049,"type":"NetworkFilesystem"},"localPath":"/mnt//25418de5-a6fb-3026-a983-c6a2d8f61322","wait":0}}]
> }
> 2024-07-01 16:31:30,790 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381192: Received: { Ans: , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver: v1, Flags: 10, {
> ModifyStoragePoolAnswer } }
> 2024-07-01 16:31:30,798 INFO [o.a.c.s.d.p.DefaultHostListener]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Connection
> established between storage pool org.apache.cloudstack.storage.datasto
> re.PrimaryDataStoreImpl@55325ecf and host 248
> 2024-07-01 16:31:30,807 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Found storage pool
> csstorage5-xs3-NVMe of type NetworkFilesystem with overprovisioning
> f actor 2
> 2024-07-01 16:31:30,807 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Total over
> provisioned capacity calculated is 2 * 4079092760576
> 2024-07-01 16:31:30,807 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Total over
> provisioned capacity of the pool csstorage5-xs3-NVMe id: 169 is
> 8158185521152
> 2024-07-01 16:31:30,815 DEBUG [c.c.s.StorageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Successfully set
> Capacity - 8158185521152 for capacity type - 3 , DataCenterId - 9,
> Host OrPoolId - 169, PodId 12
> 2024-07-01 16:31:30,815 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Connect to
> listener: SecondaryStorageListener
> 2024-07-01 16:31:30,815 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Connect to
> listener: DeploymentPlanningManagerImpl
> 2024-07-01 16:31:30,819 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Connect to
> listener: SecurityGroupListener
> 2024-07-01 16:31:30,819 INFO [c.c.n.s.SecurityGroupListener]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Received a host
> startup notification
> 2024-07-01 16:31:30,822 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381193: Sending { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver : v1, Flags: 100011,
> [{"com.cloud.agent.api.CleanupNetworkRulesCmd":{"interval":2414,"wait":0}}]
> }
> 2024-07-01 16:31:30,822 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381193: Executing: { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.CleanupNetworkRulesCmd":{"interval":2414,"wait":0}}]
> }
> 2024-07-01 16:31:30,822 INFO [c.c.n.s.SecurityGroupListener]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Scheduled network
> rules cleanup, interval=2414
> 2024-07-01 16:31:30,823 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Connect to
> listener: BehindOnPingListener
> 2024-07-01 16:31:30,823 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Connect to
> listener: SetHostParamsListener
> 2024-07-01 16:31:30,823 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Connect to
> listener: DownloadListener
> 2024-07-01 16:31:30,826 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Connect to
> listener: ConsoleProxyListener
> 2024-07-01 16:31:30,826 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Connect to
> listener: SshKeysDistriMonitor
> 2024-07-01 16:31:30,838 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381194: Sending { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver : v1, Flags: 100011,
> [{"com.cloud.agent.api.ModifySshKeysCommand":{"wait":0}}] }
> 2024-07-01 16:31:30,838 DEBUG [c.c.a.t.Request]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Seq
> 248-1798343626204381194: Executing: { Cmd , MgmtId: 95534596974, via:
> 248(xs31.failiem.lv), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.ModifySshKeysCommand":{"wait":0}}] }
> 2024-07-01 16:31:30,838 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Connect to
> listener: VpcVirtualNetworkApplianceManagerImpl
> 2024-07-01 16:31:30,854 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Connect to
> listener: StorageCapacityListener
> 2024-07-01 16:31:30,854 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Connect to
> listener: ComputeCapacityListener
> 2024-07-01 16:31:30,906 DEBUG [c.c.c.CapacityManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Found 32 VMs on host
> 248
> 2024-07-01 16:31:30,910 DEBUG [c.c.c.CapacityManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Found 0 VMs are
> Migrating from host 248
> 2024-07-01 16:31:30,993 ERROR [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Monitor
> ComputeCapacityListener says there is an error in the connect process
> for 248 du e to null
> 2024-07-01 16:31:30,993 INFO [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Host 248 is
> disconnecting with event AgentDisconnected
> 2024-07-01 16:31:30,996 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) The next status of
> agent 248is Alert, current status is Connecting
> 2024-07-01 16:31:30,996 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Deregistering link
> for 248 with state Alert
> 2024-07-01 16:31:30,996 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Remove Agent : 248
> 2024-07-01 16:31:30,996 DEBUG [c.c.a.m.DirectAgentAttache]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Processing
> disconnect 248(xs31.failiem.lv)
> 2024-07-01 16:31:30,996 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.hypervisor.xenserver.discoverer.XcpServerDisco
> verer
> 2024-07-01 16:31:30,996 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.hypervisor.hyperv.discoverer.HypervServerDisco
> verer
> 2024-07-01 16:31:30,996 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.hypervisor.vmware.manager.VmwareManagerImpl
> 2024-07-01 16:31:30,996 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl
> 2024-07-01 16:31:30,996 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: org.apache.cloudstack.engine.orchestration.NetworkOrches
> trator
> 2024-07-01 16:31:30,996 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.storage.listener.StoragePoolMonitor
> 2024-07-01 16:31:30,997 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.storage.secondary.SecondaryStorageListener
> 2024-07-01 16:31:30,997 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.deploy.DeploymentPlanningManagerImpl
> 2024-07-01 16:31:30,997 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.network.security.SecurityGroupListener
> 2024-07-01 16:31:30,997 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.agent.manager.AgentManagerImpl$BehindOnPingLis
> tener
> 2024-07-01 16:31:30,997 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.agent.manager.AgentManagerImpl$SetHostParamsLi
> stener
> 2024-07-01 16:31:30,997 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.storage.download.DownloadListener
> 2024-07-01 16:31:30,997 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.consoleproxy.ConsoleProxyListener
> 2024-07-01 16:31:31,000 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.network.SshKeysDistriMonitor
> 2024-07-01 16:31:31,000 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManag
> erImpl
> 2024-07-01 16:31:31,000 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.capacity.StorageCapacityListener
> 2024-07-01 16:31:31,000 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.capacity.ComputeCapacityListener
> 2024-07-01 16:31:31,000 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.storage.LocalStoragePoolListener
> 2024-07-01 16:31:31,000 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.network.SshKeysDistriMonitor
> 2024-07-01 16:31:31,000 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.network.router.VirtualNetworkApplianceManagerI mpl
> 2024-07-01 16:31:31,000 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkS
> tatsListener
> 2024-07-01 16:31:31,000 DEBUG [c.c.n.NetworkUsageManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Disconnected called
> on 248 with status Alert
> 2024-07-01 16:31:31,000 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Sending Disconnect
> to listener: com.cloud.storage.upload.UploadListener
> 2024-07-01 16:31:31,000 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Transition:[Resource
> state = Enabled, Agent event = AgentDisconnected, Host id = 248,
> na me = xs31.failiem.lv]
> 2024-07-01 16:31:31,007 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Notifying other
> nodes of to disconnect
> 2024-07-01 16:31:31,013 WARN [c.c.r.ResourceManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Unable to connect
> due to
> 2024-07-01 16:31:31,014 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Transition:[Resource
> state = Enabled, Agent event = AgentDisconnected, Host id = 248,
> na me = xs31.failiem.lv]
> 2024-07-01 16:31:31,022 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-380:ctx-f411cc14) (logid:284129f8) Completed creating
> agent for host 248
>
> ----------------
>
> LAST WARNINGS: cat
> /var/log/cloudstack/management/management-server.log | grep WARN
> ----------------
> 2024-07-01 21:37:48,038 WARN [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-2cd13b14) (logid:c460a47e) AlertType:: 24 |
> dataCenterId:: 9 | podId:: null | clusterId:: null | message:: System
> Alert: Number of unallocated shared network IPs is low in availability
> zone LTC-DC
> 2024-07-01 21:37:48,115 WARN [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-2cd13b14) (logid:c460a47e) AlertType:: 1 |
> dataCenterId:: 9 | podId:: 12 | clusterId:: 29 | message:: System
> Alert: Low Unallocated CPU in cluster xs3-intel-xcp-ng pod 29/1 of
> availability zone LTC-DC
> 2024-07-01 21:38:48,133 WARN [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-254eadd9) (logid:477960f4) AlertType:: 24 |
> dataCenterId:: 9 | podId:: null | clusterId:: null | message:: System
> Alert: Number of unallocated shared network IPs is low in availability
> zone LTC-DC
> 2024-07-01 21:38:48,240 WARN [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-254eadd9) (logid:477960f4) AlertType:: 1 |
> dataCenterId:: 9 | podId:: 12 | clusterId:: 29 | message:: System
> Alert: Low Unallocated CPU in cluster xs3-intel-xcp-ng pod 29/1 of
> availability zone LTC-DC
> 2024-07-01 21:39:17,689 WARN [c.c.v.UserVmManagerImpl]
> (StatsCollector-6:ctx-7cde3df8) (logid:bef4aeac) Unable to obtain VM
> network statistics.
> 2024-07-01 21:39:17,876 WARN
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:ctx-d811632c) (logid:7fc274df) Invalid last
> alert timestamp received while collecting alerts from router: r-1485-VM
> 2024-07-01 21:39:25,981 WARN [c.c.a.m.ClusteredAgentManagerImpl]
> (ClusteredAgentManager Timer:ctx-06f6388c) (logid:a7949301) can not
> load directly connected host 18(xs13.failiem.lv) due to
> 2024-07-01 21:39:26,663 WARN [c.c.h.x.d.XcpServerDiscoverer]
> (AgentTaskPool-496:ctx-5f6aa810) (logid:bc575c27) defaulting to
> xenserver650 resource for product brand: XCP-ng with product version:
> 8.1.0
> 2024-07-01 21:39:26,860 WARN [c.c.h.x.d.XcpServerDiscoverer]
> (AgentTaskPool-497:ctx-2718223e) (logid:ee799242) defaulting to
> xenserver650 resource for product brand: XCP-ng with product version:
> 8.1.0
> 2024-07-01 21:39:27,002 WARN [c.c.h.x.d.XcpServerDiscoverer]
> (AgentTaskPool-498:ctx-39723084) (logid:a026fd97) defaulting to
> xenserver650 resource for product brand: XCP-ng with product version:
> 8.1.0
> 2024-07-01 21:39:27,059 WARN [c.c.h.x.d.XcpServerDiscoverer]
> (AgentTaskPool-499:ctx-e7eaf7e0) (logid:887afb4f) defaulting to
> xenserver650 resource for product brand: XCP-ng with product version:
> 8.1.0
> 2024-07-01 21:39:31,324 WARN [c.c.r.ResourceManagerImpl]
> (AgentTaskPool-496:ctx-5f6aa810) (logid:bc575c27) Unable to connect
> due to
> 2024-07-01 21:39:32,678 WARN [c.c.r.ResourceManagerImpl]
> (AgentTaskPool-497:ctx-2718223e) (logid:ee799242) Unable to connect
> due to
> 2024-07-01 21:39:32,886 WARN [c.c.r.ResourceManagerImpl]
> (AgentTaskPool-499:ctx-e7eaf7e0) (logid:887afb4f) Unable to connect
> due to
> 2024-07-01 21:39:32,973 WARN [c.c.r.ResourceManagerImpl]
> (AgentTaskPool-498:ctx-39723084) (logid:a026fd97) Unable to connect
> due to
> 2024-07-01 21:39:48,791 WARN [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-fd2d7789) (logid:5d6d6dad) AlertType:: 24 |
> dataCenterId:: 9 | podId:: null | clusterId:: null | message:: System
> Alert: Number of unallocated shared network IPs is low in availability
> zone LTC-DC
> 2024-07-01 21:39:48,883 WARN [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-fd2d7789) (logid:5d6d6dad) AlertType:: 1 |
> dataCenterId:: 9 | podId:: 12 | clusterId:: 29 | message:: System
> Alert: Low Unallocated CPU in cluster xs3-intel-xcp-ng pod 29/1 of
> availability zone LTC-DC
>
>