You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@cloudstack.apache.org by "sbrueseke (via GitHub)" <gi...@apache.org> on 2025/01/23 08:53:27 UTC

[D] Health checks failed on VR [cloudstack]

GitHub user sbrueseke created a discussion: Health checks failed on VR

I this this kind of alerts from time to time in our UI Alerts:
Health checks failed: 1 failing checks on router r-555-VM / 9d2d3c6c-d1a7-4fd0-8143-7a94b7032eb6

In management.log I find:
2025-01-22 17:41:24,521 WARN  [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-ee7be78a) (logid:4a205737) Health checks failed: 1 failing checks on router r-555-VM / 9d2d3c6c-d1a7-4fd0-8143-7a94b7032eb6. Checking failed health checks to see if router needs recreate

In diagnostic cloud.log I am able to see this:
2025-01-22 17:50:39,837 ERROR    Not able to setup source-nat for a regular router yet

This router was started at 2025-01-22 17:39:11
2025-01-22 17:39:11,339 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-81:ctx-48ac6c20 job-20526/job-20527 ctx-8fd308b0) (logid:bde48037) VM instance {"id":555,"instanceName":"r-555-VM","type":"DomainRouter","uuid":"9d2d3c6c-d1a7-4fd0-8143-7a94b7032eb6"} state transited from [Starting] to [Running] with event [OperationSucceeded]. VM's original host: Host {"id":21,"name":"pc-kvm-8","type":"Routing","uuid":"2fbe41ca-73c9-43d4-a16a-1a44be2b1bd2"}, new host: Host {"id":21,"name":"pc-kvm-8","type":"Routing","uuid":"2fbe41ca-73c9-43d4-a16a-1a44be2b1bd2"}, host before state transition: Host {"id":21,"name":"pc-kvm-8","type":"Routing","uuid":"2fbe41ca-73c9-43d4-a16a-1a44be2b1bd2"}

And it looks like the router is working fine. Is this kind of alert expected short after a start or should we investigate this?

Regards,
Swen

GitHub link: https://github.com/apache/cloudstack/discussions/10249

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org

Re: [D] Health checks failed on VR [cloudstack]

Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.

GitHub user sbrueseke added a comment to the discussion: Health checks failed on VR

It looks like we do get this kind of alert everytime a VR is starting. We run 4.19.1.2 at the moment.

GitHub link: https://github.com/apache/cloudstack/discussions/10249#discussioncomment-11928068

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org

Re: [D] Health checks failed on VR [cloudstack]

Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.

GitHub user sbrueseke deleted a comment on the discussion: Health checks failed on VR

We did some deeper investigation and it looks like this has something to do with our backups (volume-snapshots). It looks like one of our nfs-servers (secondary storage) has some cpu stalls in the same timeframe when health checks are happening. We do not have any idea how this is related. What we also are able to see in the metrics of all instances, system vms and virtual routers is, that iops are going up.
Our primary storage is not on nfs and not related. But we do use sds which uses network (different nic), too. Is it possible that the amount of backups is too mich for the tcpstack on the kvm hosts?

This I found in the agent log:
2025-01-31 02:01:19,168 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:null) (logid:) Seq 19-7793760630141465558: Processing: { Ans: , MgmtId: 165989263229575, via: 19, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.88.32, file: monitor_service.json.772738ed-480e-456f-96d9-f2deb211485b","null - success: Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
","null - success: Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
"],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }

GitHub link: https://github.com/apache/cloudstack/discussions/10249#discussioncomment-12015892

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org

Re: [D] Health checks failed on VR [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.

GitHub user weizhouapache added a comment to the discussion: Health checks failed on VR

@sbrueseke 
is it the VR of shared, isolated network or vpc ?

GitHub link: https://github.com/apache/cloudstack/discussions/10249#discussioncomment-11926847

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org

Re: [D] Health checks failed on VR [cloudstack]

Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.

GitHub user sbrueseke closed a discussion: Health checks failed on VR

I this this kind of alerts from time to time in our UI Alerts:
Health checks failed: 1 failing checks on router r-555-VM / 9d2d3c6c-d1a7-4fd0-8143-7a94b7032eb6

In management.log I find:
2025-01-22 17:41:24,521 WARN  [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-ee7be78a) (logid:4a205737) Health checks failed: 1 failing checks on router r-555-VM / 9d2d3c6c-d1a7-4fd0-8143-7a94b7032eb6. Checking failed health checks to see if router needs recreate

In diagnostic cloud.log I am able to see this:
2025-01-22 17:50:39,837 ERROR    Not able to setup source-nat for a regular router yet

This router was started at 2025-01-22 17:39:11
2025-01-22 17:39:11,339 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-81:ctx-48ac6c20 job-20526/job-20527 ctx-8fd308b0) (logid:bde48037) VM instance {"id":555,"instanceName":"r-555-VM","type":"DomainRouter","uuid":"9d2d3c6c-d1a7-4fd0-8143-7a94b7032eb6"} state transited from [Starting] to [Running] with event [OperationSucceeded]. VM's original host: Host {"id":21,"name":"pc-kvm-8","type":"Routing","uuid":"2fbe41ca-73c9-43d4-a16a-1a44be2b1bd2"}, new host: Host {"id":21,"name":"pc-kvm-8","type":"Routing","uuid":"2fbe41ca-73c9-43d4-a16a-1a44be2b1bd2"}, host before state transition: Host {"id":21,"name":"pc-kvm-8","type":"Routing","uuid":"2fbe41ca-73c9-43d4-a16a-1a44be2b1bd2"}

And it looks like the router is working fine. Is this kind of alert expected short after a start or should we investigate this?

Regards,
Swen

GitHub link: https://github.com/apache/cloudstack/discussions/10249

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org

Re: [D] Health checks failed on VR [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.

GitHub user weizhouapache added a comment to the discussion: Health checks failed on VR

@sbrueseke 
I have no idea for now.

can you share more mgmt/agent logs 

GitHub link: https://github.com/apache/cloudstack/discussions/10249#discussioncomment-11927663

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org

Re: [D] Health checks failed on VR [cloudstack]

Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.

GitHub user sbrueseke added a comment to the discussion: Health checks failed on VR

Any idea what triggered this alert in the first place?

GitHub link: https://github.com/apache/cloudstack/discussions/10249#discussioncomment-11927335

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org

Re: [D] Health checks failed on VR [cloudstack]

Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.

GitHub user sbrueseke added a comment to the discussion: Health checks failed on VR

GitHub link: https://github.com/apache/cloudstack/discussions/10249#discussioncomment-12015892

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org

Re: [D] Health checks failed on VR [cloudstack]

Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.

GitHub user sbrueseke added a comment to the discussion: Health checks failed on VR

It is an isolated network in an advanced zone.

GitHub link: https://github.com/apache/cloudstack/discussions/10249#discussioncomment-11926951

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org

Re: [D] Health checks failed on VR [cloudstack]

Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.

GitHub user sbrueseke added a comment to the discussion: Health checks failed on VR

yes, this is a BACKUP VR, thx for the info.

Any input on relationship between health checks and slow secondary storage?

GitHub link: https://github.com/apache/cloudstack/discussions/10249#discussioncomment-12017736

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org

Re: [D] Health checks failed on VR [cloudstack]

Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.

GitHub user sbrueseke added a comment to the discussion: Health checks failed on VR

what exactly is the id of the router? Docu is not really clear about it. Is it the id in vm_instance table, the uuid, or something else?
I guess it is the id from the vm_instance table?

(proIO) 🐱 > get routerhealthcheckresults routerid=555
{
  "routerhealthchecks": {
    "healthchecks": [
      {
        "checkname": "connectivity.test",
        "checktype": "basic",
        "details": "Successfully connected to router",
        "lastupdated": "2025-01-23T10:01:37+0100",
        "success": true
      },
      {
        "checkname": "cpu_usage_check.py",
        "checktype": "basic",
        "details": "CPU Usage within limits with current at 3.7%",
        "lastupdated": "2025-01-23T10:00:01+0100",
        "success": true
      },
      {
        "checkname": "dhcp.service",
        "checktype": "basic",
        "details": "service is running",
        "lastupdated": "2025-01-23T10:00:01+0100",
        "success": true
      },
      {
        "checkname": "dhcp_check.py",
        "checktype": "advanced",
        "details": "All 1 VMs are present in dhcphosts.txt",
        "lastupdated": "2025-01-23T10:00:01+0100",
        "success": true
      },
      {
        "checkname": "disk_space_check.py",
        "checktype": "basic",
        "details": "Sufficient free space is 2976 MB",
        "lastupdated": "2025-01-23T10:00:04+0100",
        "success": true
      },
      {
        "checkname": "dns_check.py",
        "checktype": "advanced",
        "details": "All 1 VMs are present in /etc/hosts",
        "lastupdated": "2025-01-23T10:00:01+0100",
        "success": true
      },
      {
        "checkname": "filesystem.writable.test",
        "checktype": "basic",
        "details": "Successfully written to file system",
        "lastupdated": "2025-01-23T10:01:37+0100",
        "success": true
      },
      {
        "checkname": "gateways_check.py",
        "checktype": "basic",
        "details": "All 2 gateways are reachable via ping",
        "lastupdated": "2025-01-23T10:00:04+0100",
        "success": true
      },
      {
        "checkname": "haproxy_check.py",
        "checktype": "advanced",
        "details": "No data provided to check, skipping",
        "lastupdated": "2025-01-23T10:00:01+0100",
        "success": true
      },
      {
        "checkname": "iptables_check.py",
        "checktype": "advanced",
        "details": "No portforwarding rules provided to check, skipping",
        "lastupdated": "2025-01-23T10:00:01+0100",
        "success": true
      },
      {
        "checkname": "loadbalancing.service",
        "checktype": "basic",
        "details": "service is running",
        "lastupdated": "2025-01-23T10:00:01+0100",
        "success": true
      },
      {
        "checkname": "memory_usage_check.py",
        "checktype": "basic",
        "details": "Memory Usage within limits with current at 52.5651%",
        "lastupdated": "2025-01-23T10:00:12+0100",
        "success": true
      },
      {
        "checkname": "router_version_check.py",
        "checktype": "basic",
        "details": "Template and scripts version match successful",
        "lastupdated": "2025-01-23T10:00:12+0100",
        "success": true
      },
      {
        "checkname": "ssh.service",
        "checktype": "basic",
        "details": "service is running",
        "lastupdated": "2025-01-23T10:00:01+0100",
        "success": true
      },
      {
        "checkname": "webserver.service",
        "checktype": "basic",
        "details": "service is running",
        "lastupdated": "2025-01-23T10:00:01+0100",
        "success": true
      }
    ],
    "routerid": "9d2d3c6c-d1a7-4fd0-8143-7a94b7032eb6"
  }
}

GitHub link: https://github.com/apache/cloudstack/discussions/10249#discussioncomment-11927007

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org

Re: [D] Health checks failed on VR [cloudstack]

Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.

GitHub user sbrueseke added a comment to the discussion: Health checks failed on VR

GitHub link: https://github.com/apache/cloudstack/discussions/10249#discussioncomment-12015898

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org

Re: [D] Health checks failed on VR [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.

GitHub user weizhouapache added a comment to the discussion: Health checks failed on VR

you can get the health check result by cmk

```
get routerhealthcheckresults routerid=xxx
```

GitHub link: https://github.com/apache/cloudstack/discussions/10249#discussioncomment-11926887

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org

Re: [D] Health checks failed on VR [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.

GitHub user weizhouapache added a comment to the discussion: Health checks failed on VR

> what exactly is the id of the router? Docu is not really clear about it. Is it the id in vm_instance table, the uuid, or something else? I guess it is the id from the vm_instance table?
> 
normally we use uuid
id also works, but it might be unsupported in future versions.


health check results look good now

GitHub link: https://github.com/apache/cloudstack/discussions/10249#discussioncomment-11927213

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org

Re: [D] Health checks failed on VR [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.

GitHub user weizhouapache added a comment to the discussion: Health checks failed on VR

if `Error: Nexthop has invalid gateway.` appears in a BACKUP VR, I think we can ignore it

GitHub link: https://github.com/apache/cloudstack/discussions/10249#discussioncomment-12016023

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org