You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "fnavidan (via GitHub)" <gi...@apache.org> on 2026/10/10 15:02:33 UTC
[D] High Latency Between Two VMs In a L2 Private Guest Network [cloudstack]
GitHub user fnavidan created a discussion: High Latency Between Two VMs In a L2 Private Guest Network
Hi everyone,
I am running Apache CloudStack v4.22.1.0. I have created a private L2 guest network, then added a second NIC to each VM connected to private network. When I ping other VMs in the same private network, I get following results:
[root@da1 ~]# ping -c 100 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
...
[root@da1 ~]# traceroute 10.0.0.2
traceroute to 10.0.0.2 (10.0.0.2), 30 hops max, 60 byte packets
1 da-node2 (10.0.0.2) 1.334 ms !X 1.088 ms !X 1.013 ms !X
All VMs are running on the same KVM hosts which should normally give under 1ms ping rtt between guests but it doesn't. My test scenario implements GlusterFS having 2 replicas + 1 arbiter which is sensitive to latency. Currently when I test write performance on the gluster volume, I get very low throughput around 30MB/s while writing to the local filesystem of VMs gives me a 350-400MB/s write speed.
Additionally, all VMs are running on the same primary storage which is mounted as a local NFS on the KVM host, and all nodes connected to the same l2 private network.
As my gluster traffic is passing through L2 network, there must not be any extra networking overheads in a l2 network type.
GitHub link: https://github.com/apache/cloudstack/discussions/14380
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] High Latency Between Two VMs In a L2 Private Guest Network [cloudstack]
Posted by "fnavidan (via GitHub)" <gi...@apache.org>.
GitHub user fnavidan added a comment to the discussion: High Latency Between Two VMs In a L2 Private Guest Network
We found out that the KVM host is experiencing extremely high CPU utilization so that no cpu time was left for real-time network packet processing. Further investigation revealed that we has a security incident on the host where xmrig, a crypto miner was installed on the host.
Further investigation, showed no sign of successful SSH login from untrusted IPs, So that we suspect that the attacker might have been able to execute remote commands using libvirt remote management port 16514, 8250 or 9090.
We have blocked ports 16514, 5900–6100, 8250, 9090, 49152–49216 in public zone and added to trusted zone. Also, because we only have one NIC on the host, all traffic types are going through cloudbr0 bridge. We needed to manually edit Network Manager config and change zone to trusted.
At this time, all services are working fine and all critical ports are filtered from public Internet. Also, significant reduction of CPU utilization on KVM host, actually fixed the network latency between VMs.
GitHub link: https://github.com/apache/cloudstack/discussions/14380#discussioncomment-18925612
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] High Latency Between Two VMs In a L2 Private Guest Network [cloudstack]
Posted by "kiwiflyer (via GitHub)" <gi...@apache.org>.
GitHub user kiwiflyer added a comment to the discussion: High Latency Between Two VMs In a L2 Private Guest Network
Can you run virsh dumpxml on the host for this vm and post it?
GitHub link: https://github.com/apache/cloudstack/discussions/14380#discussioncomment-18855335
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] High Latency Between Two VMs In a L2 Private Guest Network [cloudstack]
Posted by "fnavidan (via GitHub)" <gi...@apache.org>.
GitHub user fnavidan added a comment to the discussion: High Latency Between Two VMs In a L2 Private Guest Network
Here are routing and network info:
[root@da1 ~]# ip route
default via 149.50.127.129 dev ens3 proto dhcp src 149.50.127.163 metric 100
10.0.0.0/24 dev ens9 proto kernel scope link src 10.0.0.1 metric 101
149.50.127.128/26 dev ens3 proto kernel scope link src 149.50.127.163 metric 100
[root@da1 ~]# traceroute -I ens9 10.0.0.2
ens9: Name or service not known
Cannot handle "host" cmdline arg `ens9' on position 1 (argc 2)
[root@da1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 1e:01:2c:00:02:d1 brd ff:ff:ff:ff:ff:ff
altname enp0s3
inet 149.50.127.163/26 brd 149.50.127.191 scope global noprefixroute ens3
valid_lft forever preferred_lft forever
inet6 fe80::1c01:2cff:fe00:2d1/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: ens9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 02:01:00:d4:00:01 brd ff:ff:ff:ff:ff:ff
altname enp0s9
inet 10.0.0.1/24 brd 10.0.0.255 scope global noprefixroute ens9
valid_lft forever preferred_lft forever
inet6 fe80::acc7:f278:d355:ff8f/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Here are 4 samples of traceroute that shows variance in icmp tests:
[root@da1 ~]# traceroute -I -i ens9 10.0.0.2
traceroute to 10.0.0.2 (10.0.0.2), 30 hops max, 60 byte packets
1 da-node2 (10.0.0.2) 1.053 ms * *
[root@da1 ~]# traceroute -I -i ens9 10.0.0.2
traceroute to 10.0.0.2 (10.0.0.2), 30 hops max, 60 byte packets
1 da-node2 (10.0.0.2) 7.838 ms 8.433 ms 8.229 ms
[root@da1 ~]# traceroute -I -i ens9 10.0.0.2
traceroute to 10.0.0.2 (10.0.0.2), 30 hops max, 60 byte packets
1 da-node2 (10.0.0.2) 0.961 ms * *
[root@da1 ~]# traceroute -I -i ens9 10.0.0.2
traceroute to 10.0.0.2 (10.0.0.2), 30 hops max, 60 byte packets
1 da-node2 (10.0.0.2) 16.925 ms 16.459 ms 13.767 ms
And this is exact route path:
[root@da1 ~]# ip route get 10.0.0.2 from 10.0.0.1
10.0.0.2 from 10.0.0.1 dev ens9 uid 0
cache
And this ping results from exactly same private interface:
[root@da1 ~]# ping -I ens9 -c 100 10.0.0.2
PING 10.0.0.2 (10.0.0.2) from 10.0.0.1 ens9: 56(84) bytes of data.
...
100 packets transmitted, 100 received, 0% packet loss, time 100048ms
rtt min/avg/max/mdev = 0.268/2.195/21.840/2.881 ms
Interestingly even when I ping a VM's public IP inside its own KVM host, I get same high ping result.
GitHub link: https://github.com/apache/cloudstack/discussions/14380#discussioncomment-18854293
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] High Latency Between Two VMs In a L2 Private Guest Network [cloudstack]
Posted by "fnavidan (via GitHub)" <gi...@apache.org>.
GitHub user fnavidan closed the discussion with a comment: High Latency Between Two VMs In a L2 Private Guest Network
Thanks for all your points. We can now move to next deployment steps. I'm going to close the discussion.
GitHub link: https://github.com/apache/cloudstack/discussions/14380#discussioncomment-19018493
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] High Latency Between Two VMs In a L2 Private Guest Network [cloudstack]
Posted by "fnavidan (via GitHub)" <gi...@apache.org>.
GitHub user fnavidan edited a discussion: High Latency Between Two VMs In a L2 Private Guest Network
Hi everyone,
I am running Apache CloudStack v4.22.1.0. I have created a private L2 guest network, then added a second NIC to each VM connected to private network. When I ping other VMs in the same private network, I get following results:
[root@da1 ~]# ping -c 100 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
...
100 packets transmitted, 100 received, 0% packet loss, time 100024ms
rtt min/avg/max/mdev = 0.266/2.122/11.104/2.249 ms
[root@da1 ~]# traceroute 10.0.0.2
traceroute to 10.0.0.2 (10.0.0.2), 30 hops max, 60 byte packets
1 da-node2 (10.0.0.2) 1.334 ms !X 1.088 ms !X 1.013 ms !X
All VMs are running on the same KVM hosts which should normally give under 1ms ping rtt between guests but it doesn't. My test scenario implements GlusterFS having 2 replicas + 1 arbiter which is sensitive to latency. Currently when I test write performance on the gluster volume, I get very low throughput around 30MB/s while writing to the local filesystem of VMs gives me a 350-400MB/s write speed.
Additionally, all VMs are running on the same primary storage which is mounted as a local NFS on the KVM host, and all nodes connected to the same l2 private network.
As my gluster traffic is passing through L2 network, there must not be any extra networking overheads in a l2 network type.
GitHub link: https://github.com/apache/cloudstack/discussions/14380
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] High Latency Between Two VMs In a L2 Private Guest Network [cloudstack]
Posted by "nickaccuweb (via GitHub)" <gi...@apache.org>.
GitHub user nickaccuweb added a comment to the discussion: High Latency Between Two VMs In a L2 Private Guest Network
@fnavidan
Good to hear that you found the cause of the issue and that GlusterFS performance has improved after stopping the crypto miner.
Regarding your question, I think 10 Gbps should be sufficient to start with for your GlusterFS setup with 2 replicas and 1 arbiter. However, it also depends on the amount of data being written and the number of VMs accessing the storage.
Since you are planning to move each GlusterFS VM to a separate KVM host, I would suggest using a dedicated 10 Gbps network for GlusterFS traffic, if possible. This will help avoid network congestion from other VMs.
Once you move the VMs, run an iperf3 test between the nodes to check the actual available bandwidth. You can also test GlusterFS write performance again to see how much difference the physical network makes.
One more thing worth checking is your NFS primary storage. If all three GlusterFS VMs are using the same underlying NFS server, that storage will still be a single point of failure, even if the VMs are running on separate KVM hosts.
For reference, you can check the official GlusterFS documentation:
https://docs.gluster.org/en/latest/Administrator-Guide/Network-Configurations-Techniques/
Here, I would start with 10 Gbps and monitor the performance before considering a network upgrade.
GitHub link: https://github.com/apache/cloudstack/discussions/14380#discussioncomment-19017939
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] High Latency Between Two VMs In a L2 Private Guest Network [cloudstack]
Posted by "kiwiflyer (via GitHub)" <gi...@apache.org>.
GitHub user kiwiflyer added a comment to the discussion: High Latency Between Two VMs In a L2 Private Guest Network
Your traceroute seems to be showing ICMP unreachable. Can you provide information on your two networks and as well as the routing table on your VMs?
What happens if you source your ping or traceroute from a specific interface?
GitHub link: https://github.com/apache/cloudstack/discussions/14380#discussioncomment-18853126
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] High Latency Between Two VMs In a L2 Private Guest Network [cloudstack]
Posted by "nickaccuweb (via GitHub)" <gi...@apache.org>.
GitHub user nickaccuweb added a comment to the discussion: High Latency Between Two VMs In a L2 Private Guest Network
@fnavidan
Are you still experiencing high latency between the 2 VMs in the L2 Network?
If yes, I would suggest checking a few things:
1. Run an iperf3 test between the VMs to check the actual network throughput.
2. Confirm that all VM NICs are using VirtIO.
3. Check CPU usage and network bridge configuration on the KVM host.
4. Since your VMs are using the same NFS primary storage, check for storage I/O latency or contention.
5. Check GlusterFS performance using gluster volume profile to identify any storage bottlenecks.
Also, are all three GlusterFS VMs running on the same physical KVM host, or are they distributed across multiple KVM hosts?
The difference between 350–400 MB/s local writes and 30 MB/s GlusterFS writes is quite high. However, GlusterFS replication adds additional network and storage operations, so we should first check whether the bottleneck is on the network or storage side.
You can also refer to the official GlusterFS documentation:
- https://docs.gluster.org/en/latest/Administrator-Guide/Performance-Testing
- https://docs.gluster.org/en/latest/Administrator-Guide/Performance-Tuning
Please share the iperf3 results and confirm the KVM host placement. This will help us narrow down the issue.
Regards,
Nick R
AccuWeb.Cloud
GitHub link: https://github.com/apache/cloudstack/discussions/14380#discussioncomment-18973213
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] High Latency Between Two VMs In a L2 Private Guest Network [cloudstack]
Posted by "fnavidan (via GitHub)" <gi...@apache.org>.
GitHub user fnavidan added a comment to the discussion: High Latency Between Two VMs In a L2 Private Guest Network
After stopping the malicious crypto miner service on the KVM host, network latency in all paths including shared network and L2 network has been reduced to under 1ms which significantly improved glusterfs throughput from 30MB/s to 350MB/s. The bottleneck was CPU usage on KVM host.
Also, all glusterfs VMs arerunning on the same KVM host. However, we would want to run each gluster VM on a separate host in the same zone, for real high-availability. In this case, how much private bandwidth will we need to have between hosts? Will 10Gbps be enough for glusterfs?
GitHub link: https://github.com/apache/cloudstack/discussions/14380#discussioncomment-18977827
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] High Latency Between Two VMs In a L2 Private Guest Network [cloudstack]
Posted by "fnavidan (via GitHub)" <gi...@apache.org>.
GitHub user fnavidan closed a discussion: High Latency Between Two VMs In a L2 Private Guest Network
Hi everyone,
I am running Apache CloudStack v4.22.1.0. I have created a private L2 guest network, then added a second NIC to each VM connected to private network. When I ping other VMs in the same private network, I get following results:
[root@da1 ~]# ping -c 100 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
...
100 packets transmitted, 100 received, 0% packet loss, time 100024ms
rtt min/avg/max/mdev = 0.266/2.122/11.104/2.249 ms
[root@da1 ~]# traceroute 10.0.0.2
traceroute to 10.0.0.2 (10.0.0.2), 30 hops max, 60 byte packets
1 da-node2 (10.0.0.2) 1.334 ms !X 1.088 ms !X 1.013 ms !X
All VMs are running on the same KVM hosts which should normally give under 1ms ping rtt between guests but it doesn't. My test scenario implements GlusterFS having 2 replicas + 1 arbiter which is sensitive to latency. Currently when I test write performance on the gluster volume, I get very low throughput around 30MB/s while writing to the local filesystem of VMs gives me a 350-400MB/s write speed.
Additionally, all VMs are running on the same primary storage which is mounted as a local NFS on the KVM host, and all nodes connected to the same l2 private network.
As my gluster traffic is passing through L2 network, there must not be any extra networking overheads in a l2 network type.
GitHub link: https://github.com/apache/cloudstack/discussions/14380
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org