You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "fermosan (via GitHub)" <gi...@apache.org> on 2026/06/01 15:13:14 UTC

[D] KVM IPv6 Route Manager — feature proposal [cloudstack]

GitHub user fermosan created a discussion: KVM IPv6 Route Manager — feature proposal

Hello all, i hope you are doing well. 

As it stands now,  IPv6 guest network traffic is routed through a Virtual Router whose link-local address acts as the next-hop gateway. Unlike IPv4, which relies on ARP and existing bridge mechanics, IPv6 requires explicit kernel routes (ip -6 route add <prefix> via <fe80::...> dev <bridge>) to be  present on the hypervisor itself. Today these routes are either installed manually or left to the operator, meaning any VR lifecycle event — start, stop, or live migration — silently leaves the hypervisor routing table in an inconsistent state. Guests on IPv6-only or dual-stack networks lose connectivity  until the routes are restored by hand.

For our infastructure we currently managing this externaly which works (most of the time). 

We were thinking if this could be a new feature that will be incorporated in cloudstack out of the box and we start implementing a new plugin (cloud-plugin-network-kvm-ipv6-vr-routes) that subscribes to three internal MessageBus topics published from VirtualNetworkApplianceManagerImpl — RouterStarted, RouterStopped, and RouterMigrated — and in response sends a new agent command
  (SetupIpv6RoutesCommand) to the KVM hypervisor. The KVM-side wrapper resolves the guest bridge and the VR's link-local address automatically via virsh domiflist and the kernel neighbour table, then applies or removes routes idempotently. Live migration is handled correctly for both the success path  (remove from source, install on destination) and the failure/revert path (undo partial state on the failed destination). 

Two global configuration knobs control the feature: router.ipv6.persist.routes optionally writes systemd-networkd drop-in files so routes survive hypervisor reboots, and router.ipv6.extra.prefixes allows injecting additional prefixes beyond those derived from network CIDRs.

  The design deliberately keeps the plugin non-invasive: the three _messageBus.publish() calls added to VirtualNetworkApplianceManagerImpl are each wrapped in a try/catch so any failure in the IPv6 layer cannot propagate back and disrupt VR operations. The plugin is a complete no-op on deployments without IPv6 guest networks — buildRouteEntries() returns an empty list and no agent command is ever sent.

The natural next step is to expose route provisioning as an on-demand management API. A syncRouterIpv6Routes command, scoped to the Admin role, would allow operators to re-provision routes for a specific Virtual Router without restarting it — the primary recovery scenario being a KVM hypervisor reboot, after which all in-memory kernel routes are lost. Since SetupIpv6RoutesCommand is already idempotent, the API would simply trigger the same setupRoutesForRouter path used by the lifecycle events, with no additional risk. We have deliberately left this out of the current scope to keep the initial contribution focused, but consider it a natural follow-up once the core plugin lands.

 We are sharing this as a starting point for discussion; in particular we would welcome feedback on whether the MessageBus approach is preferred over a dedicated VirtualRouterElement hook, and whether an  on-demand syncRouterIpv6Routes API should be included in scope.

Kind regards,


GitHub link: https://github.com/apache/cloudstack/discussions/13316

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] KVM IPv6 Route Manager — feature proposal [cloudstack]

Posted by "fermosan (via GitHub)" <gi...@apache.org>.
GitHub user fermosan edited a discussion: KVM IPv6 Route Manager — feature proposal

Hello all, i hope you are doing well. 

As it stands now, on KVM hypervisors, IPv6 guest network traffic is routed through a Virtual Router whose upstream global unicast address acts as the next-hop gateway. Unlike IPv4, which relies on ARP and existing bridge mechanics, IPv6 requires explicit kernel routes (ip -6 route add <guest-prefix>/64 via <vr-upstream-ipv6>) to be present on the hypervisor itself. Today these routes are either installed manually or left to the operator, meaning any VR lifecycle event — start, stop, or live migration — silently leaves the hypervisor routing table in an inconsistent state. Guests on IPv6-only or dual-stack networks lose connectivity until the routes are restored by hand.

For our infastructure we currently managing this externaly which works (most of the time). 

We were thinking if this could be a new feature that will be incorporated in cloudstack out of the box and we start implementing a new plugin (cloud-plugin-network-kvm-ipv6-vr-routes) that subscribes to three internal MessageBus topics published from VirtualNetworkApplianceManagerImpl — RouterStarted, RouterStopped, and RouterMigrated — and in response sends a new agent command  (SetupIpv6RoutesCommand) to the KVM hypervisor. The management server side resolves the guest network prefixes and the VR's upstream IPv6 address from the database and includes both in the command, so the KVM agent performs a straightforward ip -6 route add|del with no kernel probing required. Live migration is handled correctly for both the success path (remove from source, install on destination) and the failure/revert path (undo partial state on the failed destination). Two global configuration knobs control the feature: router.ipv6.persist.routes optionally writes systemd-networkd drop-in files so rou
 tes survive hypervisor reboots, and router.ipv6.extra.prefixes allows injecting additional prefixes beyond those derived from network CIDRs.

Two global configuration knobs control the feature: router.ipv6.persist.routes optionally writes systemd-networkd drop-in files so routes survive hypervisor reboots, and router.ipv6.extra.prefixes allows injecting additional prefixes beyond those derived from network CIDRs.

  The design deliberately keeps the plugin non-invasive: the three _messageBus.publish() calls added to VirtualNetworkApplianceManagerImpl are each wrapped in a try/catch so any failure in the IPv6 layer cannot propagate back and disrupt VR operations. The plugin is a complete no-op on deployments without IPv6 guest networks — buildRouteEntries() returns an empty list and no agent command is ever sent.

The natural next step is to expose route provisioning as an on-demand management API. A syncRouterIpv6Routes command, scoped to the Admin role, would allow operators to re-provision routes for a specific Virtual Router without restarting it — the primary recovery scenario being a KVM hypervisor reboot, after which all in-memory kernel routes are lost. Since SetupIpv6RoutesCommand is already idempotent, the API would simply trigger the same setupRoutesForRouter path used by the lifecycle events, with no additional risk. We have deliberately left this out of the current scope to keep the initial contribution focused, but consider it a natural follow-up once the core plugin lands.

 We are sharing this as a starting point for discussion; in particular we would welcome feedback on whether the MessageBus approach is preferred over a dedicated VirtualRouterElement hook, and whether an  on-demand syncRouterIpv6Routes API should be included in scope.

Kind regards,


GitHub link: https://github.com/apache/cloudstack/discussions/13316

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] KVM IPv6 Route Manager — feature proposal [cloudstack]

Posted by "fermosan (via GitHub)" <gi...@apache.org>.
GitHub user fermosan added a comment to the discussion: KVM IPv6 Route Manager — feature proposal

Yes, the anycast gateway routing is taking place on the hypervisor.  From the perspective of the cloudstack VR the upstream is the hypervisor. 

GitHub link: https://github.com/apache/cloudstack/discussions/13316#discussioncomment-17184769

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] KVM IPv6 Route Manager — feature proposal [cloudstack]

Posted by "fermosan (via GitHub)" <gi...@apache.org>.
GitHub user fermosan added a comment to the discussion: KVM IPv6 Route Manager — feature proposal

Yes. 

GitHub link: https://github.com/apache/cloudstack/discussions/13316#discussioncomment-17153720

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] KVM IPv6 Route Manager — feature proposal [cloudstack]

Posted by "fermosan (via GitHub)" <gi...@apache.org>.
GitHub user fermosan added a comment to the discussion: KVM IPv6 Route Manager — feature proposal

In our setup we run frr (bgp and isis)  on all the KVM hypervisors so they are the "upstream" routers. 

GitHub link: https://github.com/apache/cloudstack/discussions/13316#discussioncomment-17172195

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] KVM IPv6 Route Manager — feature proposal [cloudstack]

Posted by "wido (via GitHub)" <gi...@apache.org>.
GitHub user wido added a comment to the discussion: KVM IPv6 Route Manager — feature proposal

Do you do your anycast gateway routing on the hypervisor? Is that the case here?

You posted a topology, but that doesn't say much about the routes. Normally the hypervisors just shuffle around VXLAN packets and not route any VM traffic.

I don't understand the topology yet.

GitHub link: https://github.com/apache/cloudstack/discussions/13316#discussioncomment-17183696

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] KVM IPv6 Route Manager — feature proposal [cloudstack]

Posted by "wido (via GitHub)" <gi...@apache.org>.
GitHub user wido added a comment to the discussion: KVM IPv6 Route Manager — feature proposal

> In our setup we run frr (bgp and isis) on all the KVM hypervisors so they are the "upstream" routers.

But do you run VXLAN, EVPN with BGP? Can you explain a bit on how you build your env?

GitHub link: https://github.com/apache/cloudstack/discussions/13316#discussioncomment-17176417

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] KVM IPv6 Route Manager — feature proposal [cloudstack]

Posted by "fermosan (via GitHub)" <gi...@apache.org>.
GitHub user fermosan added a comment to the discussion: KVM IPv6 Route Manager — feature proposal

From the cloudstack perspective there has been a small fix  (https://github.com/apache/cloudstack/pull/7836) that was merged at somepoint and some minor modifications to the modifyvxlan script.  For our topology see the diagram below: 
<img width="740" height="406" alt="image" src="https://github.com/user-attachments/assets/243014a5-ea2a-4ae9-be69-e69e79e95b2b" />. If you need more information let me know. 


GitHub link: https://github.com/apache/cloudstack/discussions/13316#discussioncomment-17178776

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] KVM IPv6 Route Manager — feature proposal [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: KVM IPv6 Route Manager — feature proposal

@fermosan 
did you mean managing static routes on the kvm hosts ?

GitHub link: https://github.com/apache/cloudstack/discussions/13316#discussioncomment-17153070

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] KVM IPv6 Route Manager — feature proposal [cloudstack]

Posted by "fermosan (via GitHub)" <gi...@apache.org>.
GitHub user fermosan added a comment to the discussion: KVM IPv6 Route Manager — feature proposal

The proposal has to do with creating a plugin to handle the routes consistently. A route must be added on the hypervisor side (frr) to point to the customers virtual router. If the router migrates to another KVM host the route must be removed from the previous host and added to the new one.  The modifyvxlan.sh does not handle route creation and in my opinion shouldn't.  The idea is to have a plugin that will perform the changes on the hypervisor to keep everything aligned.  For a broader scope the plugin could also be modular and support cisco/juniper or any other IaC tools or orchestrators that will need to do similar tasks.  It could also be activated on demand for those who want to utilize it (like backup providers currently do). 

GitHub link: https://github.com/apache/cloudstack/discussions/13316#discussioncomment-17181871

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] KVM IPv6 Route Manager — feature proposal [cloudstack]

Posted by "wido (via GitHub)" <gi...@apache.org>.
GitHub user wido added a comment to the discussion: KVM IPv6 Route Manager — feature proposal

> @fermosan can you explain why the static routes are needed on the kvm hosts ?

I would like to understand the same. Why do you need routes on the KVM hosts?

GitHub link: https://github.com/apache/cloudstack/discussions/13316#discussioncomment-17161517

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] KVM IPv6 Route Manager — feature proposal [cloudstack]

Posted by "wido (via GitHub)" <gi...@apache.org>.
GitHub user wido added a comment to the discussion: KVM IPv6 Route Manager — feature proposal

You can override the modifyvxlan.sh script to fit your environment. As the network components are difficult to standardize, would that work for you for example?

I'm still trying to understand what your proposal actually means.

GitHub link: https://github.com/apache/cloudstack/discussions/13316#discussioncomment-17180422

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] KVM IPv6 Route Manager — feature proposal [cloudstack]

Posted by "wido (via GitHub)" <gi...@apache.org>.
GitHub user wido added a comment to the discussion: KVM IPv6 Route Manager — feature proposal

> Yes, the anycast gateway routing is taking place on the hypervisor. From the perspective of the cloudstack VR the upstream is the hypervisor.

Got it. Cool setup! However, I think this would outside CloudStack's scope right? There are so many ways people would want to do this.

Can't you also use Libvirt/Qemu hooks for this to execute certain commands during VM startup and shutdown.

GitHub link: https://github.com/apache/cloudstack/discussions/13316#discussioncomment-17249574

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] KVM IPv6 Route Manager — feature proposal [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: KVM IPv6 Route Manager — feature proposal

@fermosan 
can you explain why the static routes are needed on the kvm hosts ?


GitHub link: https://github.com/apache/cloudstack/discussions/13316#discussioncomment-17161467

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org