You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Abhishek Kumar <Ab...@shapeblue.com> on 2022/05/10 08:19:15 UTC
Re: IPV6 in Isolated/VPC networks
Hi all,
IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
Documentation for it is available at http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier (will be available in the official docs once 4.17.0 version of docs is built).
[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
Regards,
Abhishek
________________________________
From: Rohit Yadav <ro...@shapeblue.com>
Sent: 13 September 2021 14:30
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks
Thanks Alex, Wei. I've updated the docs here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
I'll leave the thread open for futher discussion/ideas/feedback. I think we've completed the phase1 design doc including all feedback comments for adding IPv6 support in CloudStack and some initial poc/work can be started. My colleagues and I will keep everyone posted on this thread and/or on a Github PR as and when we're able to start our work on the same (after 4.16, potentially towards 4.17).
Regards.
________________________________
From: Wei ZHOU <us...@gmail.com>
Sent: Friday, September 10, 2021 20:22
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks
Agree with Alex.
We only need to know how many /64 are allocated. We do not care how many
ipv6 addresses are used by VMs.
-Wei
On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <Al...@shapeblue.com>
wrote:
> Hi Rohit,
>
> I'd go for option 2, don't see a point tracking anything smaller than a
> /64 tbh.
>
> Cheers
> Alex
>
>
>
>
> -----Original Message-----
> From: Rohit Yadav <ro...@shapeblue.com>
> Sent: 09 September 2021 12:44
> To: dev@cloudstack.apache.org
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Thanks Alex, Kristaps. I've updated the design doc to reflect two
> agreements:
>
> * Allocate /64 for both isolated network and VPC tiers, no large
> allocation of prefixes to VPC (cons: more static routing rules for upstream
> router/admins)
> * All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a
> dedicated /64 prefix/block for systemvms
>
> The only outstanding question now is:
>
> * How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage
> for shared network (design, implementation and use-cases?)
> Option1: We don't do it, all user VMs nics have ipv4 address which whose
> usage we don't track. For public VR/nics/networks, we can simply add the
> IPv6 details for a related IPv4 address.
> Option2: Implement a separate, first-class IPv6 address or /64 prefix
> tracking/management and usage for all VMs and systemvms nic (this means
> account/domain level limits and new billing/records)
> Option3: other thoughts?
>
>
> Regards.
>
> ________________________________
> From: Alex Mattioli <Al...@shapeblue.com>
> Sent: Wednesday, September 8, 2021 23:24
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: RE: IPV6 in Isolated/VPC networks
>
> Hi Rohit, Kristaps,
>
> I'd say option 1 as well, it does create a bit more overhead with static
> routes but if that's automated for a VPC it can also be easily automated
> for several tiers of a VPC. We also don't constrain the amount of tiers in
> a VPC.
> It has the added advantage to be closer to the desired behaviour with
> dynamic routing in the future, where a VPC VR can announce several subnets
> upstream.
>
> Cheers
> Alex
>
>
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: Rohit Yadav <ro...@shapeblue.com>
> Sent: 08 September 2021 19:04
> To: dev@cloudstack.apache.org
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Hi Kristaps,
>
> Thanks for sharing, I suppose that means individual tiers should be
> allocated /64 instead of larger ipv6 blocks to the whole VPC which could
> cause wastage.
>
> Any objection from anybody?
>
> Regards.
> ________________________________
> From: Kristaps Cudars <kr...@gmail.com>
> Sent: Wednesday, September 8, 2021 9:24:01 PM
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Hello,
>
> I asked networking team to comment on "How should the IPv6
> block/allocation work in VPCs?"
> Option1: They haven't seen lately devices with limits on how many static
> routes can be created.
> Option2: With /60 and /62 assignments and big quantity of routers IPv6
> assignment from RIPE NNC can be drained fast.
>
> /48 contains 64000 /64
> /60 contains 16 /64
> 64000 / 16 = 4000 routers
>
>
> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
> > All,
> >
> > After another iteration with Alex, I've updated the design doc. Kindly
> review:
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> > +Isolated+Network+and+VPC+with+Static+Routing
> >
> >
> > ... and advise on some outstanding questions:
> >
> > * How should the IPv6 block/allocation work in VPCs?
> > Option1: Should this be simply /64 allocation on any new tier, the
> > cons of this option is one static route/rule per VPC tier. (many
> > upstream routers may have limit on no. of static routes?)
> > Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
> (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks
> global setting (3 is default). The pros of this option are less no. of
> static route/rule and easy programming, but potential wastage of multiple
> /64 prefix blocks for unused/uncreated tiers.
> > * How do we manage IPv6 usage? Can anyone advise how we do IPv6
> usage for shared network (design, implementation and use-cases?)
> > Option1: We don't do it, all user VMs nics have ipv4 address which whose
> usage we don't track. For public VR/nics/networks, we can simply add the
> IPv6 details for a related IPv4 address.
> > Option2: Implement a separate, first-class IPv6 address or /64 prefix
> tracking/management and usage for all VMs and systemvms nic (this means
> account/domain level limits and new billing/records)
> > * Enable IPv6 on systemvms (specifically SSVM and CPVM) by default
> if zone has a IPv6 address block allocated/assigned for use for systemvms
> (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons
> of this?)
> > *
> >
> > Regards.
> >
> > ________________________________
> > From: Rohit Yadav <ro...@shapeblue.com>
> > Sent: Thursday, August 19, 2021 15:45
> > To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> > Subject: Re: IPV6 in Isolated/VPC networks
> >
> > Hi all,
> >
> > I've taken feedback from this thread and wrote this design doc:
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> > +Isolated+Network+and+VPC+with+Static+Routing
> >
> > Kindly review and advise if I missed anything or anything that needs to
> be changed/updated. You may comment on the wiki directly too.
> >
> > Kindly suggest your views on the following (also in the design doc
> above):
> >
> > Outstanding Questions:
> >
> > * Should admin or user be able to specify how VPC super CIDRs are
> created/needed; for example a user can ask for VPC with /60 super CIDR? Or
> should CloudStack automatically find/allocate a /64 for a new VPC tier from
> the root-admin configured /64-/48 block?
> > * Should we explore FRR and iBGP or other strategies to do dynamic
> routing which may not require advance/complex configuration in the VR or
> for the users/admin?
> > * With SLAAC and no dhcpv6, is there a way to support secondary IPv6
> addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's
> nics?
> > * Any thoughts on UI/UX for firewall/routing management?
> > * Any other feature/support for isolated network or VPC feature that
> must be explored or supported such as PF, VPN, LB, vpc static routes, vpc
> gateway etc.
> > * For usage - should we have any consideration, or should we assume
> that IPv4 and IPv6 address will go together for every nic; so IPv6 usage
> for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new
> biling/usage needed?
> > * For smoketests, local dev-test should we explore ULA? Unique Local
> Address - in the range fc00::/7. Typically only within the 'local' half
> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
> This prefix can be randomly generated at first install by CloudStack in a
> zone using zoneid etc?
> > * Should we expand support for VR diagnostic feature to include
> support for ipv6, traceroute6?
> > * Discuss - expand support/ability to allocate a /60, or /56 etc
> prefix to an individual VM in shared network (Wido's suggestion)
> >
> >
> > Regards.
> >
> > ________________________________
> > From: Wei ZHOU <us...@gmail.com>
> > Sent: Tuesday, August 17, 2021 21:16
> > To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> > Subject: Re: IPV6 in Isolated/VPC networks
> >
> > Thanks Kristaps, Wido, Rohit and Alex for your replies.
> >
> > I am fine with not having stateful dhcpv6 and privacy
> > extension/temporary address in phase 1. If community decides not to do
> > eventually , it is also ok to me.
> >
> > We could explore how to better use secondary ipv6 addresses as Wido
> > advised. It would be great if anyone share some user experience.
> >
> > -Wei
> >
> >
> > On Tuesday, 17 August 2021, Kristaps Cudars
> > <kr...@gmail.com>
> > wrote:
> >
> > > Hi Wei,
> > >
> > > Published this month's RFC 9099 and explains in different
> > > words/perspective what has been written by Alex, Rohit and Wido.
> > > https://www.rfc-editor.org/rfc/rfc9099.html
> > >
> > >
> > > On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
> > > > Hi Wido,
> > > >
> > > > (cc to Rohit and Alex)
> > > >
> > > > It is a good suggestion to use FRR for ipv6. The configuration is
> > > > quite simple and the VMs can get SLAAC, routes, etc.
> > > >
> > > > Privacy extension looks not the same as what you mentioned. see
> > > > https://datatracker.ietf.org/doc/html/rfc4941
> > > >
> > > > You are right. To use static routing, the admins need to configure
> > > > the routes in the upstream router, and add some ipv6 ranges (eg
> > > > /56 for VPCs and /64 for isolated networks) and their next-hop
> > > > (which will be configured in VRs) in CloudStack. CloudStack will
> > > > pick up an IPv6 range
> > > and
> > > > assign it to an isolated network or vpc. @Rohit, correct me if I'm
> wrong.
> > > >
> > > > I have a question, it looks stateless dhcpv6 (SLAAC from
> > > > router/VR, router/dns etc via RA messages) will be the only option
> > > > for now (related
> > > to
> > > > your pr https://github.com/apache/cloudstack/pull/3077) . Would it
> > > > be
> > > good
> > > > to provide stateful dhcpv6 (which can be implemented by dnsmasq)
> > > > as an option in cloudstack ? The advantages are
> > > > (1) support other ipv6 cidr sizes than /64.
> > > > (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
> > > > addresses can be changed
> > > > (4) an Ipv6 addresses can be re-used by multiple vms.
> > > > The problem is, stateful dhcpv6 does not support
> > > > routers,nameservers,
> > > etc.
> > > > we need to figure it out (probably use radvd/frr and dnsmasq both).
> > > >
> > > > -Wei
> > > >
> > > >
> > >
> >
> >
> >
> >
> >
> >
>
>
>
> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > See my inline responses:
> > > > >
> > > > > Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> > > > > > Hi all,
> > > > > >
> > > > > > Thanks for your feedback and ideas, I've gone ahead with
> > > > > > discussing
> > > them
> > > > > with Alex and came up with a PoC/design which can be implemented
> > > > > in the following phases:
> > > > > >
> > > > > > * Phase1: implement ipv6 support in isolated networks and
> VPC
> > > with
> > > > > static routing
> > > > > > * Phase2: discuss and implement support for dynamic routing
> > > (TBD)
> > > > > >
> > > > > > For Phase1 here's the high-level proposal:
> > > > > >
> > > > > > * IPv6 address management:
> > > > > > * At the zone level root-admin specifies a /64 public
> range
> > > that
> > > > > will be used for VRs, then they can add a /48, or /56 IPv6 range
> > > > > for
> > > guest
> > > > > networks (to be used by isolated networks and VPC tiers)
> > > > > > * On creation of any IPv6 enabled isolated network or VPC
> > > tier,
> > > > > from the /48 or /56 block a /64 network is allocated/used
> > > > > > * We assume SLAAC and autoconfiguration, no DHCPv6 in the
> > > zone
> > > > > (discuss: is privacy a concern, can privacy extensions rfc4941
> > > > > of
> > > slaac be
> > > > > explored?)
> > > > >
> > > > > Privacy Extensions are only a concern for client devices which
> > > > > roam between different IPv6 networks.
> > > > >
> > > > > If you IPv6 address of a client keeps the same suffix (MAC
> > > > > based) and switches network then only the prefix (/64) will change.
> > > > >
> > > > > This way a network like Google, Facebook, etc could track your
> > > > > device moving from network to network if they only look at the
> > > > > last 64-bits of the IPv6 address.
> > > > >
> > > > > For servers this is not a problem as you already know in which
> > > > > network they are.
> > > > >
> > > > > > * Network offerings: root-admin can create new network
> offerings
> > > > > (with VPC too) that specifies a network stack option:
> > > > > > * ipv4 only (default, for backward compatibility all
> > > > > networks/offerings post-upgrade migrate to this option)
> > > > > > * ipv4-and-ipv6
> > > > > > * ipv6-only (this can be phase 1.b)
> > > > > > * A new routing option: static (phase1), dynamic (phase2,
> > > with
> > > > > multiple sub-options such as ospf/bgp etc...)
> > > > >
> > > > > This means that the network admin will need to statically route
> > > > > the
> > > IPv6
> > > > > subnet to the VR's outside IPv6 address, for example, on a JunOS
> > > router:
> > > > >
> > > > > set routing-options rib inet6.0 static route 2001:db8:500::/48
> > > > > next-hop
> > > > > 2001:db8:100::50
> > > > >
> > > > > I'm assuming that 2001:db8:100::50 is the address of the VR on
> > > > > the outside (/64) network. In reality this will probably be a
> > > > > longer address, but this is for just the example.
> > > > >
> > > > > > * VR changes:
> > > > > > * VR gets its guest and public nics set to inet6 auto
> > > > > > * For each /64 allocated to guest network and VPC tiers,
> > > radvd
> > > > > is configured to do RA
> > > > >
> > > > > radvd is fine, but looking at phase 2 with dynamic routing you
> > > > > might already want to look into FRRouting. FRR can also
> > > > > advertise RAs while not doing any routing.
> > > > >
> > > > > interface ens4
> > > > > no ipv6 nd suppress-ra
> > > > > ipv6 nd prefix 2001:db8:500::/64
> > > > > ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> > > > >
> > > > > See: http://docs.frrouting.org/en/latest/ipv6.html
> > > > >
> > > > > > * Firewall: a new ipv6 zone/chain is created for ipv6
> where
> > > ipv6
> > > > > firewall rules (ACLs, ingress, egress) are implemented; ACLs
> > > > > between
> > > VPC
> > > > > tiers are managed/implemented by ipv6 firewall on VR
> > > > >
> > > > > Please take a look at the existing security_group.py script
> > > > > which implements RFC4890
> > > > >
> > > > > https://datatracker.ietf.org/doc/html/rfc4890
> > > > >
> > > > > ICMPv6 is a vital part of IPv6 and certain packets should always
> > > > > be allowed.
> > > > >
> > > > > > * It is assumed that static routes are created on the
> > > core/main
> > > > > router by the admin or automated using some scripts/tools; for
> > > > > this CloudStack will announce events with details of /64
> > > > > networks and VR's public IPv6 address that can be consumed by a
> > > > > rabbitmq/message bus
> > > client
> > > > > (for example), or a custom cron job or script as part of
> orchestration.
> > > > > (this wouldn't be necessary for dynamic routing bgp with
> > > > > phase2)\\
> > > > >
> > > > > You would only need to announce the /48 or /56 allocated to the
> > > > > VR, that's all. You don't need to inform the upstream router
> > > > > about the /64 subnets created within that larger subnet.
> > > > >
> > > > > > * Guest Networking: With SLAAC, it's easy for CloudStack to
> > > > > calculate allocate and use a /64 and determine the IPv6 address
> > > > > of VR
> > > nics
> > > > > and guest VM nics
> > > > > > * A user create an isolated network/VPC with an offering
> > > that is
> > > > > ipv6 enabled
> > > > > > * A user can manage firewall for the IPv6 address/guest
> nics;
> > > > > there'll be no port forward and LB feature though for IPv6
> > > > > > * A users can run workloads in the guest VMs that listen
> on
> > > > > publically routable ipv6 addresses
> > > > > > * Usage/billing etc continue to work, no change needed
> > > > > >
> > > > > > Network layout:
> > > > > >
> > > > > > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
> > > > > > VLAN] ->
> > > > > [guest VMs/nics]
> > > > > > *core/ISP router needs static routes to be added (manually or
> > > > > automated), assumes a /48 or /56 configured for the zone
> > > > > >
> > > > > > Thoughts, feedback?
> > > > >
> > > > > Looks doable!
> > > > >
> > > > > Side-note: It would be very cool if you could use parts of this
> > > > > implementation to also route /48, /56, or /60 subnets to
> > > > > individual VMs in Shared networks.
> > > > >
> > > > > Why? This allows for running Docker containers with native IPv6
> > > > > inside the VM or running a (Open)VPN server inside a VM which
> > > > > then assigns public IPv6 addresses to clients connected.
> > > > >
> > > > > Instead of routing the subnet to a VR we route the subnet to a
> > > > > single instance in a shared network.
> > > > >
> > > > > If we could then also move these subnets between Instances
> > > > > easily one can quickly migrate to a different instance while
> > > > > keeping the same IPv6 subnet.
> > > > >
> > > > > Wido
> > > > >
> > > > > >
> > > > > > Proof-of-concept commentary: here's what I did to test the idea:
> > > > > >
> > > > > > * Created an isolated network and deployed a VM in my home
> lab
> > > > > > The VR running on KVM has following nics
> > > > > > eth0 - guest network
> > > > > > eth1 - link local
> > > > > > eth2 - public network
> > > > > >
> > > > > > * I setup a custom openwrt router on a RPi4 to serve as a
> > > toy-core
> > > > > router where I create a wan6 IPv6 tunnel using tunnel broker and
> > > > > I got
> > > a
> > > > > /48 allocated. My configuration looks like:
> > > > > > /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> > > > > > /64 - 2001:470:36:3e2::/64 (default allocated by)
> > > > > >
> > > > > > I create a LAN ipv6 (public network for CloudStack VR): at
> > > subnet/prefix
> > > > > 0:
> > > > > > LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
> > > > > > SLAAC+stateless DHCP (no dhcpv6)
> > > > > > *
> > > > > > *
> > > > > > In the isolated VR, I enabled ipv6 as:
> > > > > > net.ipv6.conf.all.disable_ipv6 = 0
> > > > > > net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
> > > > > > 1 net.ipv6.conf.all.accept_redirects = 1
> > > > > > net.ipv6.conf.all.autoconf = 1
> > > > > >
> > > > > > Set up a IPv6 nameserver/dns in /etc/resolve.conf And
> > > > > > configured the nics:
> > > > > > echo iface eth0 inet6 auto >> /etc/network/interfaces echo
> > > > > > iface eth2 inet6 auto >> /etc/network/interfaces
> > > > > > /etc/init.d/networking restart Next, restart ACS isolated
> > > > > > network without cleanup to have it
> > > > > reconfigure IPv4 nics, firewall, NAT etc
> > > > > >
> > > > > > *
> > > > > > Next, I created a /64 network for the isolated guest network
> > > > > > on eth0
> > > of
> > > > > VR using radvd:
> > > > > >
> > > > > > # cat /etc/radvd.conf
> > > > > > interface eth0
> > > > > > {
> > > > > > AdvSendAdvert on;
> > > > > > MinRtrAdvInterval 5;
> > > > > > MaxRtrAdvInterval 15;
> > > > > > prefix 2001:470:ed36:1::/64
> > > > > > {
> > > > > > AdvOnLink on;
> > > > > > AdvAutonomous on;
> > > > > > };
> > > > > > };
> > > > > > systemctl restart radvd
> > > > > > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
> > > > > > this
> > > > > ...:1::/64 network
> > > > > > * Finally I added a static route in toy core-router for the
> new
> > > /64
> > > > > IPv6 range in the isolated network
> > > > > > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
> > > > > > eth2> dev
> > > > > <local lan nic>
> > > > > > *
> > > > > > ... and I enabled firewall rules to allow any traffic to pass
> > > > > > for the
> > > > > new /64 network
> > > > > >
> > > > > > And voila all done! I create a domain AAAA record that points
> > > > > > to my
> > > > > guest VM IPv6 address a test webserver on
> > > > > > http://ipv6-isolated-ntwk-demo.yadav.cloud/
> > > > > >
> > > > > > (Note: I'll get rid of the tunnel and request a new /48 block
> > > > > > after a
> > > > > few days, sharing this solely for testing purposes)
> > > > > >
> > > > > >
> > > > > > Regards.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
Re: IPV6 in Isolated/VPC networks
Posted by Abhishek Kumar <Ab...@shapeblue.com>.
Hi Wido, Wei,
I've created a PR to have static route in the event description, https://github.com/apache/cloudstack/pull/6389
Can you please check.
Regards,
Abhishek
________________________________
From: Wido den Hollander <wi...@widodh.nl>
Sent: 13 May 2022 15:30
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Wei ZHOU <us...@gmail.com>
Subject: Re: IPV6 in Isolated/VPC networks
Op 13-05-2022 om 11:39 schreef Wei ZHOU:
> Hi Wido,
>
> We do not allocate an ipv6 subnet to a VPC. Instead, we allocate an ipv6
> subnet for each vpc tier.
> These subnets have the same gateway (ipv6 addr in VR).
>
> for example
>
> ipv6 route fd23:313a:2f53:3111::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
> ipv6 route fd23:313a:2f53:3112::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
> ipv6 route fd23:313a:2f53:3113::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
>
Thanks Wei and Alex, that explains! So you can have multiple /64 subnets
being routed to the same VR.
It's up to the organization to only allocate /64s coming from a specific
/56 or /48 to make things logical, but underneath it's all /64s which we
need to route.
Wido
> -Wei
>
> On Fri, 13 May 2022 at 11:30, Wido den Hollander <wi...@widodh.nl> wrote:
>
>>
>>
>> Op 12-05-2022 om 16:10 schreef Alex Mattioli:
>>>> ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
>>>
>>> That's correct
>>
>> Ok. So in that case the subnet would be very welcome to be present in
>> the message on the bus.
>>
>>>
>>>> Or a larger subnet:
>>>> ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4
>>>
>>> Not really, the subnets for isolated/VPC networks are always /64. Which
>> means also no real need to include subnets as well.
>>>
>>
>> But there can be multiple networks behind the VPC router or not? If
>> there are multiple networks you need >/64 as you can then allocate /64s
>> from that larger subnet.
>>
>> Wido
>>
>>> Cheers
>>> Alex
>>>
>>>
>>>
>>>
>>
>
Re: IPV6 in Isolated/VPC networks
Posted by Wido den Hollander <wi...@widodh.nl>.
Op 13-05-2022 om 11:39 schreef Wei ZHOU:
> Hi Wido,
>
> We do not allocate an ipv6 subnet to a VPC. Instead, we allocate an ipv6
> subnet for each vpc tier.
> These subnets have the same gateway (ipv6 addr in VR).
>
> for example
>
> ipv6 route fd23:313a:2f53:3111::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
> ipv6 route fd23:313a:2f53:3112::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
> ipv6 route fd23:313a:2f53:3113::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
>
Thanks Wei and Alex, that explains! So you can have multiple /64 subnets
being routed to the same VR.
It's up to the organization to only allocate /64s coming from a specific
/56 or /48 to make things logical, but underneath it's all /64s which we
need to route.
Wido
> -Wei
>
> On Fri, 13 May 2022 at 11:30, Wido den Hollander <wi...@widodh.nl> wrote:
>
>>
>>
>> Op 12-05-2022 om 16:10 schreef Alex Mattioli:
>>>> ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
>>>
>>> That's correct
>>
>> Ok. So in that case the subnet would be very welcome to be present in
>> the message on the bus.
>>
>>>
>>>> Or a larger subnet:
>>>> ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4
>>>
>>> Not really, the subnets for isolated/VPC networks are always /64. Which
>> means also no real need to include subnets as well.
>>>
>>
>> But there can be multiple networks behind the VPC router or not? If
>> there are multiple networks you need >/64 as you can then allocate /64s
>> from that larger subnet.
>>
>> Wido
>>
>>> Cheers
>>> Alex
>>>
>>>
>>>
>>>
>>
>
RE: IPV6 in Isolated/VPC networks
Posted by Alex Mattioli <Al...@shapeblue.com>.
> But there can be multiple networks behind the VPC router or not? If there are multiple networks you need
>/64 as you can then allocate /64s from that larger subnet.
Yes, there will be multiple /64s in a VPC, but they are very likely to be non-contiguous, so in general we should treat as completely separate networks.
One could in theory dedicate a range to a domain/account and statically route to that, but I'm not sure if that will happen often.
Cheers
Alex
-----Original Message-----
From: Wido den Hollander <wi...@widodh.nl>
Sent: 13 May 2022 11:30
To: dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks
Op 12-05-2022 om 16:10 schreef Alex Mattioli:
>> ipv6 route fd23:313a:2f53:3cbf::/64
>> fd23:313a:2f53:3000:1c00:baff:fe00:4
>
> That's correct
Ok. So in that case the subnet would be very welcome to be present in the message on the bus.
>
>> Or a larger subnet:
>> ipv6 route fd23:313a:2f53:3c00::/56
>> fd23:313a:2f53:3000:1c00:baff:fe00:4
>
> Not really, the subnets for isolated/VPC networks are always /64. Which means also no real need to include subnets as well.
>
But there can be multiple networks behind the VPC router or not? If there are multiple networks you need >/64 as you can then allocate /64s from that larger subnet.
Wido
> Cheers
> Alex
>
>
>
>
>
> -----Original Message-----
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: 12 May 2022 16:04
> To: Abhishek Kumar <Ab...@shapeblue.com>;
> dev@cloudstack.apache.org
> Subject: Re: IPV6 in Isolated/VPC networks
>
>
>
> On 5/12/22 09:55, Abhishek Kumar wrote:
>> Hi Wido,
>>
>> I do not understand what you mean by WAB address but
>
> WAB was a type. I meant WAN.
>
>> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the public IP of the network
>> (IPv6 of the public NIC of the network VR) in the sample.
>> Yes, route for fd23:313a:2f53:3cbf::/64 need to be added to this IP.
>> fd23:313a:2f53:3cbf::/64 is guest IPv6 CIDR of the network here.
>>
>
> So that means that I would need to run this command on my upstream router:
>
> ipv6 route fd23:313a:2f53:3cbf::/64
> fd23:313a:2f53:3000:1c00:baff:fe00:4
>
> Or a larger subnet:
>
> ipv6 route fd23:313a:2f53:3c00::/56
> fd23:313a:2f53:3000:1c00:baff:fe00:4
>
>> Currently, the message on event bus does not include subnet. Should
>> that be included?
>
> Yes, because then you can pickup those messages and inject the route via ExaBGP into a routing table right away.
>
>> In case of VPCs, there could be multiple tiers which will need
>> multiple routes to be added. Will that be an issue if we include
>> current network/tier subnet in the event message?
>
> No, as long as it points to the same VR you simply have multiple subnets being routed to the same VR.
>
> I do wonder what happens if you destroy the VR and create a new one. The WAN address then changes (due to SLAAC) and thus the routes need to be re-programmed.
>
> Wido
>
>>
>> Regards,
>> Abhishek
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> -
>> --
>> *From:* Wido den Hollander <wi...@widodh.nl>
>> *Sent:* 10 May 2022 19:01
>> *To:* dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek
>> Kumar <Ab...@shapeblue.com>
>> *Subject:* Re: IPV6 in Isolated/VPC networks
>>
>> Hi,
>>
>> Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
>>> Yes. When a public IPv6 is assigned or released, CloudStack will publish event with type NET.IP6ASSIGN, NET.IP6RELEASE.
>>> These event notifications can be tracked. And with improvements in events framework, these event messages will have network uuid as entityuuid and Network as entitytype. Using this network can be queried using to list IPv6 routes that need to be added.
>>>
>>> Sample event message,
>>>
>>> {"eventDateTime":"2022-05-10 09:32:12
>>> +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":
>>> "14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned
>>> public
>>> IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID:
>>> 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":
>>> "bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec
>>> - af19-1e00320001f3","entity":"Network","status":"Completed"}
>>>
>>>
>>> ?Sample API call,
>>>> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98
>>>> filter=id,name,ip6routes
>>> {
>>> "count": 1,
>>> "network": [
>>> {
>>> "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
>>> "ip6routes": [
>>> {
>>> "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
>>> "subnet": "fd23:313a:2f53:3cbf::/64"
>>> }
>>
>> Looking at this JSON, does this mean that
>> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
>>
>> And that I would need to (statically) route fd23:313a:2f53:3cbf::/64
>> to that IP?
>>
>> The event message does not include the subnet, that makes it a bit
>> more difficult as you would then also need to do a API-call to gather
>> that information.
>>
>> Wido
>>
>> P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an
>> AAAA-record for IPv6!
>>
>>> ],
>>> "name": "routing_test"
>>> }
>>> ]
>>> }
>>>
>>>
>>>
>>>
>>> ________________________________
>>> From: Wido den Hollander <wi...@widodh.nl>
>>> Sent: 10 May 2022 13:59
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek
>>> Kumar <Ab...@shapeblue.com>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
>>>> Hi all,
>>>>
>>>> IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
>>>> Documentation for it is available at
>>>> http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.ht
>>>> m
>>>> l#isolated-network-and-vpc-tier
>> <http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.htm
>> l #isolated-network-and-vpc-tier> (will be available in the official
>> docs once 4.17.0 version of docs is built).
>>>>
>>>
>>> Great work!
>>>
>>> I see only static routing is supported. But do we publish something
>>> on the message bus once a new VR/VPC is created?
>>>
>>> This way you could pick up these messages and have the network
>>> create a
>>> (static) route based on those.
>>>
>>> ExaBGP for example could be used to inject such routes.
>>>
>>> Wido
>>>
>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support
>>>> +
>>>> in+Isolated+Network+and+VPC+with+Static+Routing
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>> i
>> n+Isolated+Network+and+VPC+with+Static+Routing>
>>>>
>>>> Regards,
>>>> Abhishek
>>>>
>>>> ________________________________
>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>> Sent: 13 September 2021 14:30
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Thanks Alex, Wei. I've updated the docs here:
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support
>>>> +
>>>> in+Isolated+Network+and+VPC+with+Static+Routing
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>> i
>> n+Isolated+Network+and+VPC+with+Static+Routing>
>>>>
>>>> I'll leave the thread open for futher discussion/ideas/feedback. I
>>>> think we've completed the phase1 design doc including all feedback
>>>> comments for adding IPv6 support in CloudStack and some initial
>>>> poc/work can be started. My colleagues and I will keep everyone
>>>> posted on this thread and/or on a Github PR as and when we're able
>>>> to
>> start our work on the same (after 4.16, potentially towards 4.17).
>>>>
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Wei ZHOU <us...@gmail.com>
>>>> Sent: Friday, September 10, 2021 20:22
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Agree with Alex.
>>>> We only need to know how many /64 are allocated. We do not care how
>>>> many
>>>> ipv6 addresses are used by VMs.
>>>>
>>>> -Wei
>>>>
>>>> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli
>>>> <Al...@shapeblue.com>
>>>> wrote:
>>>>
>>>>> Hi Rohit,
>>>>>
>>>>> I'd go for option 2, don't see a point tracking anything smaller
>>>>> than a
>>>>> /64 tbh.
>>>>>
>>>>> Cheers
>>>>> Alex
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>>> Sent: 09 September 2021 12:44
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>>>>> agreements:
>>>>>
>>>>> * Allocate /64 for both isolated network and VPC tiers, no
>>>>> large allocation of prefixes to VPC (cons: more static routing
>>>>> rules for upstream
>>>>> router/admins)
>>>>> * All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address
>>>>> if zone has a dedicated /64 prefix/block for systemvms
>>>>>
>>>>> The only outstanding question now is:
>>>>>
>>>>> * How do we manage IPv6 usage? Can anyone advise how we do
>>>>> IPv6 usage for shared network (design, implementation and
>>>>> use-cases?)
>>>>> Option1: We don't do it, all user VMs nics have ipv4 address which
>>>>> whose usage we don't track. For public VR/nics/networks, we can
>>>>> simply add the
>>>>> IPv6 details for a related IPv4 address.
>>>>> Option2: Implement a separate, first-class IPv6 address or /64
>>>>> prefix tracking/management and usage for all VMs and systemvms
>>>>> nic (this means account/domain level limits and new
>>>>> billing/records)
>>>>> Option3: other thoughts?
>>>>>
>>>>>
>>>>> Regards.
>>>>>
>>>>> ________________________________
>>>>> From: Alex Mattioli <Al...@shapeblue.com>
>>>>> Sent: Wednesday, September 8, 2021 23:24
>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>> Subject: RE: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hi Rohit, Kristaps,
>>>>>
>>>>> I'd say option 1 as well, it does create a bit more overhead with
>>>>> static routes but if that's automated for a VPC it can also be
>>>>> easily automated for several tiers of a VPC. We also don't
>>>>> constrain the amount of tiers in a VPC.
>>>>> It has the added advantage to be closer to the desired behaviour
>>>>> with dynamic routing in the future, where a VPC VR can announce
>>>>> several subnets upstream.
>>>>>
>>>>> Cheers
>>>>> Alex
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>>> Sent: 08 September 2021 19:04
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hi Kristaps,
>>>>>
>>>>> Thanks for sharing, I suppose that means individual tiers should
>>>>> be allocated /64 instead of larger ipv6 blocks to the whole VPC
>>>>> which could cause wastage.
>>>>>
>>>>> Any objection from anybody?
>>>>>
>>>>> Regards.
>>>>> ________________________________
>>>>> From: Kristaps Cudars <kr...@gmail.com>
>>>>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hello,
>>>>>
>>>>> I asked networking team to comment on "How should the IPv6
>>>>> block/allocation work in VPCs?"
>>>>> Option1: They haven't seen lately devices with limits on how many
>>>>> static routes can be created.
>>>>> Option2: With /60 and /62 assignments and big quantity of routers
>>>>> IPv6 assignment from RIPE NNC can be drained fast.
>>>>>
>>>>> /48 contains 64000 /64
>>>>> /60 contains 16 /64
>>>>> 64000 / 16 = 4000 routers
>>>>>
>>>>>
>>>>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
>>>>>> All,
>>>>>>
>>>>>> After another iteration with Alex, I've updated the design doc.
>>>>>> Kindly
>>>>> review:
>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppo
>>>>>> r
>>>>>> t+in
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>> i
>> n>
>>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>>
>>>>>>
>>>>>> ... and advise on some outstanding questions:
>>>>>>
>>>>>> * How should the IPv6 block/allocation work in VPCs?
>>>>>> Option1: Should this be simply /64 allocation on any new tier,
>>>>>> the cons of this option is one static route/rule per VPC tier.
>>>>>> (many upstream routers may have limit on no. of static routes?)
>>>>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers)
>>>>>> or
>>>>>> /62
>>>>> (4 tiers) for the VPC, this can be filtered based on the
>>>>> vpc.max.networks global setting (3 is default). The pros of this
>>>>> option are less no. of static route/rule and easy programming, but
>>>>> potential wastage of multiple
>>>>> /64 prefix blocks for unused/uncreated tiers.
>>>>>> * How do we manage IPv6 usage? Can anyone advise how we
>>>>>> do
>>>>>> IPv6
>>>>> usage for shared network (design, implementation and use-cases?)
>>>>>> Option1: We don't do it, all user VMs nics have ipv4 address
>>>>>> which whose
>>>>> usage we don't track. For public VR/nics/networks, we can simply
>>>>> add the
>>>>> IPv6 details for a related IPv4 address.
>>>>>> Option2: Implement a separate, first-class IPv6 address or /64
>>>>>> prefix
>>>>> tracking/management and usage for all VMs and systemvms nic (this
>>>>> means account/domain level limits and new billing/records)
>>>>>> * Enable IPv6 on systemvms (specifically SSVM and CPVM)
>>>>>> by default
>>>>> if zone has a IPv6 address block allocated/assigned for use for
>>>>> systemvms (this was mainly thought for VRs, but why no ssvm and
>>>>> cpvms too - any cons of this?)
>>>>>> *
>>>>>>
>>>>>> Regards.
>>>>>>
>>>>>> ________________________________
>>>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>>>> Sent: Thursday, August 19, 2021 15:45
>>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I've taken feedback from this thread and wrote this design doc:
>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppo
>>>>>> r
>>>>>> t+in
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>> i
>> n>
>>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>>
>>>>>> Kindly review and advise if I missed anything or anything that
>>>>>> needs to
>>>>> be changed/updated. You may comment on the wiki directly too.
>>>>>>
>>>>>> Kindly suggest your views on the following (also in the design
>>>>>> doc
>>>>> above):
>>>>>>
>>>>>> Outstanding Questions:
>>>>>>
>>>>>> * Should admin or user be able to specify how VPC super
>>>>>> CIDRs are
>>>>> created/needed; for example a user can ask for VPC with /60 super
>>>>> CIDR? Or should CloudStack automatically find/allocate a /64 for a
>>>>> new VPC tier from the root-admin configured /64-/48 block?
>>>>>> * Should we explore FRR and iBGP or other strategies to
>>>>>> do dynamic
>>>>> routing which may not require advance/complex configuration in the
>>>>> VR or for the users/admin?
>>>>>> * With SLAAC and no dhcpv6, is there a way to support
>>>>>> secondary IPv6
>>>>> addresses (or floating IPv6 addresses for VR/public traffic) for
>>>>> guest VM's nics?
>>>>>> * Any thoughts on UI/UX for firewall/routing management?
>>>>>> * Any other feature/support for isolated network or VPC
>>>>>> feature that
>>>>> must be explored or supported such as PF, VPN, LB, vpc static
>>>>> routes, vpc gateway etc.
>>>>>> * For usage - should we have any consideration, or should
>>>>>> we assume
>>>>> that IPv4 and IPv6 address will go together for every nic; so IPv6
>>>>> usage for a nic is in tandem with Ipv4 address for a nic, i.e. no
>>>>> explicit/new biling/usage needed?
>>>>>> * For smoketests, local dev-test should we explore ULA?
>>>>>> Unique Local
>>>>> Address - in the range fc00::/7. Typically only within the 'local'
>>>>> half fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
>>>>> This prefix can be randomly generated at first install by
>>>>> CloudStack in a zone using zoneid etc?
>>>>>> * Should we expand support for VR diagnostic feature to
>>>>>> include
>>>>> support for ipv6, traceroute6?
>>>>>> * Discuss - expand support/ability to allocate a /60, or
>>>>>> /56 etc
>>>>> prefix to an individual VM in shared network (Wido's suggestion)
>>>>>>
>>>>>>
>>>>>> Regards.
>>>>>>
>>>>>> ________________________________
>>>>>> From: Wei ZHOU <us...@gmail.com>
>>>>>> Sent: Tuesday, August 17, 2021 21:16
>>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>>
>>>>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>>>>
>>>>>> I am fine with not having stateful dhcpv6 and privacy
>>>>>> extension/temporary address in phase 1. If community decides not
>>>>>> to do eventually , it is also ok to me.
>>>>>>
>>>>>> We could explore how to better use secondary ipv6 addresses as
>>>>>> Wido advised. It would be great if anyone share some user experience.
>>>>>>
>>>>>> -Wei
>>>>>>
>>>>>>
>>>>>> On Tuesday, 17 August 2021, Kristaps Cudars
>>>>>> <kr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Wei,
>>>>>>>
>>>>>>> Published this month's RFC 9099 and explains in different
>>>>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>>>>> https://www.rfc-editor.org/rfc/rfc9099.html
>> <https://www.rfc-editor.org/rfc/rfc9099.html>
>>>>>>>
>>>>>>>
>>>>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
>>>>>>>> Hi Wido,
>>>>>>>>
>>>>>>>> (cc to Rohit and Alex)
>>>>>>>>
>>>>>>>> It is a good suggestion to use FRR for ipv6. The configuration
>>>>>>>> is quite simple and the VMs can get SLAAC, routes, etc.
>>>>>>>>
>>>>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>>>>> https://datatracker.ietf.org/doc/html/rfc4941
>> <https://datatracker.ietf.org/doc/html/rfc4941>
>>>>>>>>
>>>>>>>> You are right. To use static routing, the admins need to
>>>>>>>> configure the routes in the upstream router, and add some ipv6
>>>>>>>> ranges (eg
>>>>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
>>>>>>>> (which will be configured in VRs) in CloudStack. CloudStack
>>>>>>>> will pick up an IPv6 range
>>>>>>> and
>>>>>>>> assign it to an isolated network or vpc. @Rohit, correct me if
>>>>>>>> I'm
>>>>> wrong.
>>>>>>>>
>>>>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
>>>>>>>> router/VR, router/dns etc via RA messages) will be the only
>>>>>>>> option for now (related
>>>>>>> to
>>>>>>>> your pr https://github.com/apache/cloudstack/pull/3077)
>> <https://github.com/apache/cloudstack/pull/3077)> . Would it
>>>>>>>> be
>>>>>>> good
>>>>>>>> to provide stateful dhcpv6 (which can be implemented by
>>>>>>>> dnsmasq) as an option in cloudstack ? The advantages are
>>>>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
>>>>>>>> addresses can be changed
>>>>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>>>>> The problem is, stateful dhcpv6 does not support
>>>>>>>> routers,nameservers,
>>>>>>> etc.
>>>>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>>>>
>>>>>>>> -Wei
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> See my inline responses:
>>>>>>>>>
>>>>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
>>>>>>>>>> discussing
>>>>>>> them
>>>>>>>>> with Alex and came up with a PoC/design which can be
>>>>>>>>> implemented in the following phases:
>>>>>>>>>>
>>>>>>>>>> * Phase1: implement ipv6 support in isolated
>>>>>>>>>> networks and
>>>>> VPC
>>>>>>> with
>>>>>>>>> static routing
>>>>>>>>>> * Phase2: discuss and implement support for dynamic
>>>>>>>>>> routing
>>>>>>> (TBD)
>>>>>>>>>>
>>>>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>>>>
>>>>>>>>>> * IPv6 address management:
>>>>>>>>>> * At the zone level root-admin specifies a /64
>>>>>>>>>> public
>>>>> range
>>>>>>> that
>>>>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6
>>>>>>>>> range for
>>>>>>> guest
>>>>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>>>>> * On creation of any IPv6 enabled isolated
>>>>>>>>>> network or VPC
>>>>>>> tier,
>>>>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>>>>> * We assume SLAAC and autoconfiguration, no
>>>>>>>>>> DHCPv6 in the
>>>>>>> zone
>>>>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
>>>>>>>>> of
>>>>>>> slaac be
>>>>>>>>> explored?)
>>>>>>>>>
>>>>>>>>> Privacy Extensions are only a concern for client devices which
>>>>>>>>> roam between different IPv6 networks.
>>>>>>>>>
>>>>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>>>>
>>>>>>>>> This way a network like Google, Facebook, etc could track your
>>>>>>>>> device moving from network to network if they only look at the
>>>>>>>>> last 64-bits of the IPv6 address.
>>>>>>>>>
>>>>>>>>> For servers this is not a problem as you already know in which
>>>>>>>>> network they are.
>>>>>>>>>
>>>>>>>>>> * Network offerings: root-admin can create new
>>>>>>>>>> network
>>>>> offerings
>>>>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>>>>> * ipv4 only (default, for backward compatibility
>>>>>>>>>> all
>>>>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>>>>> * ipv4-and-ipv6
>>>>>>>>>> * ipv6-only (this can be phase 1.b)
>>>>>>>>>> * A new routing option: static (phase1), dynamic
>>>>>>>>>> (phase2,
>>>>>>> with
>>>>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>>>>
>>>>>>>>> This means that the network admin will need to statically
>>>>>>>>> route the
>>>>>>> IPv6
>>>>>>>>> subnet to the VR's outside IPv6 address, for example, on a
>>>>>>>>> JunOS
>>>>>>> router:
>>>>>>>>>
>>>>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
>>>>>>>>> next-hop
>>>>>>>>> 2001:db8:100::50
>>>>>>>>>
>>>>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
>>>>>>>>> the outside (/64) network. In reality this will probably be a
>>>>>>>>> longer address, but this is for just the example.
>>>>>>>>>
>>>>>>>>>> * VR changes:
>>>>>>>>>> * VR gets its guest and public nics set to inet6
>>>>>>>>>> auto
>>>>>>>>>> * For each /64 allocated to guest network and VPC
>>>>>>>>>> tiers,
>>>>>>> radvd
>>>>>>>>> is configured to do RA
>>>>>>>>>
>>>>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
>>>>>>>>> might already want to look into FRRouting. FRR can also
>>>>>>>>> advertise RAs while not doing any routing.
>>>>>>>>>
>>>>>>>>> interface ens4
>>>>>>>>> no ipv6 nd suppress-ra
>>>>>>>>> ipv6 nd prefix 2001:db8:500::/64
>>>>>>>>> ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>>>>
>>>>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
>> <http://docs.frrouting.org/en/latest/ipv6.html>
>>>>>>>>>
>>>>>>>>>> * Firewall: a new ipv6 zone/chain is created for
>>>>>>>>>> ipv6
>>>>> where
>>>>>>> ipv6
>>>>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
>>>>>>>>> between
>>>>>>> VPC
>>>>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>>>>
>>>>>>>>> Please take a look at the existing security_group.py script
>>>>>>>>> which implements RFC4890
>>>>>>>>>
>>>>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
>> <https://datatracker.ietf.org/doc/html/rfc4890>
>>>>>>>>>
>>>>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should
>>>>>>>>> always be allowed.
>>>>>>>>>
>>>>>>>>>> * It is assumed that static routes are created on
>>>>>>>>>> the
>>>>>>> core/main
>>>>>>>>> router by the admin or automated using some scripts/tools; for
>>>>>>>>> this CloudStack will announce events with details of /64
>>>>>>>>> networks and VR's public IPv6 address that can be consumed by
>>>>>>>>> a rabbitmq/message bus
>>>>>>> client
>>>>>>>>> (for example), or a custom cron job or script as part of
>>>>> orchestration.
>>>>>>>>> (this wouldn't be necessary for dynamic routing bgp with
>>>>>>>>> phase2)\\
>>>>>>>>>
>>>>>>>>> You would only need to announce the /48 or /56 allocated to
>>>>>>>>> the VR, that's all. You don't need to inform the upstream
>>>>>>>>> router about the /64 subnets created within that larger subnet.
>>>>>>>>>
>>>>>>>>>> * Guest Networking: With SLAAC, it's easy for
>>>>>>>>>> CloudStack to
>>>>>>>>> calculate allocate and use a /64 and determine the IPv6
>>>>>>>>> address of VR
>>>>>>> nics
>>>>>>>>> and guest VM nics
>>>>>>>>>> * A user create an isolated network/VPC with an
>>>>>>>>>> offering
>>>>>>> that is
>>>>>>>>> ipv6 enabled
>>>>>>>>>> * A user can manage firewall for the IPv6
>>>>>>>>>> address/guest
>>>>> nics;
>>>>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>>>>> * A users can run workloads in the guest VMs that
>>>>>>>>>> listen
>>>>> on
>>>>>>>>> publically routable ipv6 addresses
>>>>>>>>>> * Usage/billing etc continue to work, no change
>>>>>>>>>> needed
>>>>>>>>>>
>>>>>>>>>> Network layout:
>>>>>>>>>>
>>>>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
>>>>>>>>>> VLAN] ->
>>>>>>>>> [guest VMs/nics]
>>>>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>>>>
>>>>>>>>>> Thoughts, feedback?
>>>>>>>>>
>>>>>>>>> Looks doable!
>>>>>>>>>
>>>>>>>>> Side-note: It would be very cool if you could use parts of
>>>>>>>>> this implementation to also route /48, /56, or /60 subnets to
>>>>>>>>> individual VMs in Shared networks.
>>>>>>>>>
>>>>>>>>> Why? This allows for running Docker containers with native
>>>>>>>>> IPv6 inside the VM or running a (Open)VPN server inside a VM
>>>>>>>>> which then assigns public IPv6 addresses to clients connected.
>>>>>>>>>
>>>>>>>>> Instead of routing the subnet to a VR we route the subnet to a
>>>>>>>>> single instance in a shared network.
>>>>>>>>>
>>>>>>>>> If we could then also move these subnets between Instances
>>>>>>>>> easily one can quickly migrate to a different instance while
>>>>>>>>> keeping the same IPv6 subnet.
>>>>>>>>>
>>>>>>>>> Wido
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>>>>
>>>>>>>>>> * Created an isolated network and deployed a VM in
>>>>>>>>>> my home
>>>>> lab
>>>>>>>>>> The VR running on KVM has following nics
>>>>>>>>>> eth0 - guest network
>>>>>>>>>> eth1 - link local
>>>>>>>>>> eth2 - public network
>>>>>>>>>>
>>>>>>>>>> * I setup a custom openwrt router on a RPi4 to serve
>>>>>>>>>> as a
>>>>>>> toy-core
>>>>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker
>>>>>>>>> and I got
>>>>>>> a
>>>>>>>>> /48 allocated. My configuration looks like:
>>>>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>>>>
>>>>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>>>>> subnet/prefix
>>>>>>>>> 0:
>>>>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>>>>> *
>>>>>>>>>> *
>>>>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
>>>>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra
>>>>>>>>>> =
>>>>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
>>>>>>>>>> net.ipv6.conf.all.autoconf = 1
>>>>>>>>>>
>>>>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
>>>>>>>>>> configured the nics:
>>>>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
>>>>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
>>>>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
>>>>>>>>>> network without cleanup to have it
>>>>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>>>>
>>>>>>>>>> *
>>>>>>>>>> Next, I created a /64 network for the isolated guest network
>>>>>>>>>> on eth0
>>>>>>> of
>>>>>>>>> VR using radvd:
>>>>>>>>>>
>>>>>>>>>> # cat /etc/radvd.conf
>>>>>>>>>> interface eth0
>>>>>>>>>> {
>>>>>>>>>> AdvSendAdvert on;
>>>>>>>>>> MinRtrAdvInterval 5;
>>>>>>>>>> MaxRtrAdvInterval 15;
>>>>>>>>>> prefix 2001:470:ed36:1::/64
>>>>>>>>>> {
>>>>>>>>>> AdvOnLink on;
>>>>>>>>>> AdvAutonomous on;
>>>>>>>>>> };
>>>>>>>>>> };
>>>>>>>>>> systemctl restart radvd
>>>>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
>>>>>>>>>> this
>>>>>>>>> ...:1::/64 network
>>>>>>>>>> * Finally I added a static route in toy core-router
>>>>>>>>>> for the
>>>>> new
>>>>>>> /64
>>>>>>>>> IPv6 range in the isolated network
>>>>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>>>>> eth2> dev
>>>>>>>>> <local lan nic>
>>>>>>>>>> *
>>>>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
>>>>>>>>>> for the
>>>>>>>>> new /64 network
>>>>>>>>>>
>>>>>>>>>> And voila all done! I create a domain AAAA record that points
>>>>>>>>>> to my
>>>>>>>>> guest VM IPv6 address a test webserver on
>>>>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>> <http://ipv6-isolated-ntwk-demo.yadav.cloud/>
>>>>>>>>>>
>>>>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
>>>>>>>>>> after a
>>>>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
Re: IPV6 in Isolated/VPC networks
Posted by Wei ZHOU <us...@gmail.com>.
Hi Wido,
We do not allocate an ipv6 subnet to a VPC. Instead, we allocate an ipv6
subnet for each vpc tier.
These subnets have the same gateway (ipv6 addr in VR).
for example
ipv6 route fd23:313a:2f53:3111::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
ipv6 route fd23:313a:2f53:3112::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
ipv6 route fd23:313a:2f53:3113::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
-Wei
On Fri, 13 May 2022 at 11:30, Wido den Hollander <wi...@widodh.nl> wrote:
>
>
> Op 12-05-2022 om 16:10 schreef Alex Mattioli:
> >> ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
> >
> > That's correct
>
> Ok. So in that case the subnet would be very welcome to be present in
> the message on the bus.
>
> >
> >> Or a larger subnet:
> >> ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4
> >
> > Not really, the subnets for isolated/VPC networks are always /64. Which
> means also no real need to include subnets as well.
> >
>
> But there can be multiple networks behind the VPC router or not? If
> there are multiple networks you need >/64 as you can then allocate /64s
> from that larger subnet.
>
> Wido
>
> > Cheers
> > Alex
> >
> >
> >
> >
>
Re: IPV6 in Isolated/VPC networks
Posted by Wido den Hollander <wi...@widodh.nl>.
Op 12-05-2022 om 16:10 schreef Alex Mattioli:
>> ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
>
> That's correct
Ok. So in that case the subnet would be very welcome to be present in
the message on the bus.
>
>> Or a larger subnet:
>> ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4
>
> Not really, the subnets for isolated/VPC networks are always /64. Which means also no real need to include subnets as well.
>
But there can be multiple networks behind the VPC router or not? If
there are multiple networks you need >/64 as you can then allocate /64s
from that larger subnet.
Wido
> Cheers
> Alex
>
>
>
>
>
> -----Original Message-----
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: 12 May 2022 16:04
> To: Abhishek Kumar <Ab...@shapeblue.com>; dev@cloudstack.apache.org
> Subject: Re: IPV6 in Isolated/VPC networks
>
>
>
> On 5/12/22 09:55, Abhishek Kumar wrote:
>> Hi Wido,
>>
>> I do not understand what you mean by WAB address but
>
> WAB was a type. I meant WAN.
>
>> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the public IP of the network
>> (IPv6 of the public NIC of the network VR) in the sample.
>> Yes, route for fd23:313a:2f53:3cbf::/64 need to be added to this IP.
>> fd23:313a:2f53:3cbf::/64 is guest IPv6 CIDR of the network here.
>>
>
> So that means that I would need to run this command on my upstream router:
>
> ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
>
> Or a larger subnet:
>
> ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4
>
>> Currently, the message on event bus does not include subnet. Should
>> that be included?
>
> Yes, because then you can pickup those messages and inject the route via ExaBGP into a routing table right away.
>
>> In case of VPCs, there could be multiple tiers which will need
>> multiple routes to be added. Will that be an issue if we include
>> current network/tier subnet in the event message?
>
> No, as long as it points to the same VR you simply have multiple subnets being routed to the same VR.
>
> I do wonder what happens if you destroy the VR and create a new one. The WAN address then changes (due to SLAAC) and thus the routes need to be re-programmed.
>
> Wido
>
>>
>> Regards,
>> Abhishek
>>
>>
>>
>>
>>
>>
>> ----------------------------------------------------------------------
>> --
>> *From:* Wido den Hollander <wi...@widodh.nl>
>> *Sent:* 10 May 2022 19:01
>> *To:* dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek
>> Kumar <Ab...@shapeblue.com>
>> *Subject:* Re: IPV6 in Isolated/VPC networks
>>
>> Hi,
>>
>> Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
>>> Yes. When a public IPv6 is assigned or released, CloudStack will publish event with type NET.IP6ASSIGN, NET.IP6RELEASE.
>>> These event notifications can be tracked. And with improvements in events framework, these event messages will have network uuid as entityuuid and Network as entitytype. Using this network can be queried using to list IPv6 routes that need to be added.
>>>
>>> Sample event message,
>>>
>>> {"eventDateTime":"2022-05-10 09:32:12
>>> +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":
>>> "14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned public
>>> IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID:
>>> 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":
>>> "bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-
>>> af19-1e00320001f3","entity":"Network","status":"Completed"}
>>>
>>>
>>> ?Sample API call,
>>>> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98
>>>> filter=id,name,ip6routes
>>> {
>>> "count": 1,
>>> "network": [
>>> {
>>> "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
>>> "ip6routes": [
>>> {
>>> "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
>>> "subnet": "fd23:313a:2f53:3cbf::/64"
>>> }
>>
>> Looking at this JSON, does this mean that
>> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
>>
>> And that I would need to (statically) route fd23:313a:2f53:3cbf::/64
>> to that IP?
>>
>> The event message does not include the subnet, that makes it a bit
>> more difficult as you would then also need to do a API-call to gather
>> that information.
>>
>> Wido
>>
>> P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an
>> AAAA-record for IPv6!
>>
>>> ],
>>> "name": "routing_test"
>>> }
>>> ]
>>> }
>>>
>>>
>>>
>>>
>>> ________________________________
>>> From: Wido den Hollander <wi...@widodh.nl>
>>> Sent: 10 May 2022 13:59
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek
>>> Kumar <Ab...@shapeblue.com>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
>>>> Hi all,
>>>>
>>>> IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
>>>> Documentation for it is available at
>>>> http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.htm
>>>> l#isolated-network-and-vpc-tier
>> <http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html
>> #isolated-network-and-vpc-tier> (will be available in the official
>> docs once 4.17.0 version of docs is built).
>>>>
>>>
>>> Great work!
>>>
>>> I see only static routing is supported. But do we publish something
>>> on the message bus once a new VR/VPC is created?
>>>
>>> This way you could pick up these messages and have the network create
>>> a
>>> (static) route based on those.
>>>
>>> ExaBGP for example could be used to inject such routes.
>>>
>>> Wido
>>>
>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>>>> in+Isolated+Network+and+VPC+with+Static+Routing
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
>> n+Isolated+Network+and+VPC+with+Static+Routing>
>>>>
>>>> Regards,
>>>> Abhishek
>>>>
>>>> ________________________________
>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>> Sent: 13 September 2021 14:30
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Thanks Alex, Wei. I've updated the docs here:
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>>>> in+Isolated+Network+and+VPC+with+Static+Routing
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
>> n+Isolated+Network+and+VPC+with+Static+Routing>
>>>>
>>>> I'll leave the thread open for futher discussion/ideas/feedback. I
>>>> think we've completed the phase1 design doc including all feedback
>>>> comments for adding IPv6 support in CloudStack and some initial
>>>> poc/work can be started. My colleagues and I will keep everyone
>>>> posted on this thread and/or on a Github PR as and when we're able
>>>> to
>> start our work on the same (after 4.16, potentially towards 4.17).
>>>>
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Wei ZHOU <us...@gmail.com>
>>>> Sent: Friday, September 10, 2021 20:22
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Agree with Alex.
>>>> We only need to know how many /64 are allocated. We do not care how
>>>> many
>>>> ipv6 addresses are used by VMs.
>>>>
>>>> -Wei
>>>>
>>>> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli
>>>> <Al...@shapeblue.com>
>>>> wrote:
>>>>
>>>>> Hi Rohit,
>>>>>
>>>>> I'd go for option 2, don't see a point tracking anything smaller
>>>>> than a
>>>>> /64 tbh.
>>>>>
>>>>> Cheers
>>>>> Alex
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>>> Sent: 09 September 2021 12:44
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>>>>> agreements:
>>>>>
>>>>> * Allocate /64 for both isolated network and VPC tiers, no
>>>>> large allocation of prefixes to VPC (cons: more static routing
>>>>> rules for upstream
>>>>> router/admins)
>>>>> * All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if
>>>>> zone has a dedicated /64 prefix/block for systemvms
>>>>>
>>>>> The only outstanding question now is:
>>>>>
>>>>> * How do we manage IPv6 usage? Can anyone advise how we do
>>>>> IPv6 usage for shared network (design, implementation and
>>>>> use-cases?)
>>>>> Option1: We don't do it, all user VMs nics have ipv4 address which
>>>>> whose usage we don't track. For public VR/nics/networks, we can
>>>>> simply add the
>>>>> IPv6 details for a related IPv4 address.
>>>>> Option2: Implement a separate, first-class IPv6 address or /64
>>>>> prefix tracking/management and usage for all VMs and systemvms nic
>>>>> (this means account/domain level limits and new billing/records)
>>>>> Option3: other thoughts?
>>>>>
>>>>>
>>>>> Regards.
>>>>>
>>>>> ________________________________
>>>>> From: Alex Mattioli <Al...@shapeblue.com>
>>>>> Sent: Wednesday, September 8, 2021 23:24
>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>> Subject: RE: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hi Rohit, Kristaps,
>>>>>
>>>>> I'd say option 1 as well, it does create a bit more overhead with
>>>>> static routes but if that's automated for a VPC it can also be
>>>>> easily automated for several tiers of a VPC. We also don't
>>>>> constrain the amount of tiers in a VPC.
>>>>> It has the added advantage to be closer to the desired behaviour
>>>>> with dynamic routing in the future, where a VPC VR can announce
>>>>> several subnets upstream.
>>>>>
>>>>> Cheers
>>>>> Alex
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>>> Sent: 08 September 2021 19:04
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hi Kristaps,
>>>>>
>>>>> Thanks for sharing, I suppose that means individual tiers should be
>>>>> allocated /64 instead of larger ipv6 blocks to the whole VPC which
>>>>> could cause wastage.
>>>>>
>>>>> Any objection from anybody?
>>>>>
>>>>> Regards.
>>>>> ________________________________
>>>>> From: Kristaps Cudars <kr...@gmail.com>
>>>>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hello,
>>>>>
>>>>> I asked networking team to comment on "How should the IPv6
>>>>> block/allocation work in VPCs?"
>>>>> Option1: They haven't seen lately devices with limits on how many
>>>>> static routes can be created.
>>>>> Option2: With /60 and /62 assignments and big quantity of routers
>>>>> IPv6 assignment from RIPE NNC can be drained fast.
>>>>>
>>>>> /48 contains 64000 /64
>>>>> /60 contains 16 /64
>>>>> 64000 / 16 = 4000 routers
>>>>>
>>>>>
>>>>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
>>>>>> All,
>>>>>>
>>>>>> After another iteration with Alex, I've updated the design doc.
>>>>>> Kindly
>>>>> review:
>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppor
>>>>>> t+in
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
>> n>
>>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>>
>>>>>>
>>>>>> ... and advise on some outstanding questions:
>>>>>>
>>>>>> * How should the IPv6 block/allocation work in VPCs?
>>>>>> Option1: Should this be simply /64 allocation on any new tier, the
>>>>>> cons of this option is one static route/rule per VPC tier. (many
>>>>>> upstream routers may have limit on no. of static routes?)
>>>>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or
>>>>>> /62
>>>>> (4 tiers) for the VPC, this can be filtered based on the
>>>>> vpc.max.networks global setting (3 is default). The pros of this
>>>>> option are less no. of static route/rule and easy programming, but
>>>>> potential wastage of multiple
>>>>> /64 prefix blocks for unused/uncreated tiers.
>>>>>> * How do we manage IPv6 usage? Can anyone advise how we do
>>>>>> IPv6
>>>>> usage for shared network (design, implementation and use-cases?)
>>>>>> Option1: We don't do it, all user VMs nics have ipv4 address which
>>>>>> whose
>>>>> usage we don't track. For public VR/nics/networks, we can simply
>>>>> add the
>>>>> IPv6 details for a related IPv4 address.
>>>>>> Option2: Implement a separate, first-class IPv6 address or /64
>>>>>> prefix
>>>>> tracking/management and usage for all VMs and systemvms nic (this
>>>>> means account/domain level limits and new billing/records)
>>>>>> * Enable IPv6 on systemvms (specifically SSVM and CPVM) by
>>>>>> default
>>>>> if zone has a IPv6 address block allocated/assigned for use for
>>>>> systemvms (this was mainly thought for VRs, but why no ssvm and
>>>>> cpvms too - any cons of this?)
>>>>>> *
>>>>>>
>>>>>> Regards.
>>>>>>
>>>>>> ________________________________
>>>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>>>> Sent: Thursday, August 19, 2021 15:45
>>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I've taken feedback from this thread and wrote this design doc:
>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppor
>>>>>> t+in
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
>> n>
>>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>>
>>>>>> Kindly review and advise if I missed anything or anything that
>>>>>> needs to
>>>>> be changed/updated. You may comment on the wiki directly too.
>>>>>>
>>>>>> Kindly suggest your views on the following (also in the design doc
>>>>> above):
>>>>>>
>>>>>> Outstanding Questions:
>>>>>>
>>>>>> * Should admin or user be able to specify how VPC super
>>>>>> CIDRs are
>>>>> created/needed; for example a user can ask for VPC with /60 super
>>>>> CIDR? Or should CloudStack automatically find/allocate a /64 for a
>>>>> new VPC tier from the root-admin configured /64-/48 block?
>>>>>> * Should we explore FRR and iBGP or other strategies to do
>>>>>> dynamic
>>>>> routing which may not require advance/complex configuration in the
>>>>> VR or for the users/admin?
>>>>>> * With SLAAC and no dhcpv6, is there a way to support
>>>>>> secondary IPv6
>>>>> addresses (or floating IPv6 addresses for VR/public traffic) for
>>>>> guest VM's nics?
>>>>>> * Any thoughts on UI/UX for firewall/routing management?
>>>>>> * Any other feature/support for isolated network or VPC
>>>>>> feature that
>>>>> must be explored or supported such as PF, VPN, LB, vpc static
>>>>> routes, vpc gateway etc.
>>>>>> * For usage - should we have any consideration, or should we
>>>>>> assume
>>>>> that IPv4 and IPv6 address will go together for every nic; so IPv6
>>>>> usage for a nic is in tandem with Ipv4 address for a nic, i.e. no
>>>>> explicit/new biling/usage needed?
>>>>>> * For smoketests, local dev-test should we explore ULA?
>>>>>> Unique Local
>>>>> Address - in the range fc00::/7. Typically only within the 'local'
>>>>> half fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
>>>>> This prefix can be randomly generated at first install by
>>>>> CloudStack in a zone using zoneid etc?
>>>>>> * Should we expand support for VR diagnostic feature to
>>>>>> include
>>>>> support for ipv6, traceroute6?
>>>>>> * Discuss - expand support/ability to allocate a /60, or /56
>>>>>> etc
>>>>> prefix to an individual VM in shared network (Wido's suggestion)
>>>>>>
>>>>>>
>>>>>> Regards.
>>>>>>
>>>>>> ________________________________
>>>>>> From: Wei ZHOU <us...@gmail.com>
>>>>>> Sent: Tuesday, August 17, 2021 21:16
>>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>>
>>>>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>>>>
>>>>>> I am fine with not having stateful dhcpv6 and privacy
>>>>>> extension/temporary address in phase 1. If community decides not
>>>>>> to do eventually , it is also ok to me.
>>>>>>
>>>>>> We could explore how to better use secondary ipv6 addresses as
>>>>>> Wido advised. It would be great if anyone share some user experience.
>>>>>>
>>>>>> -Wei
>>>>>>
>>>>>>
>>>>>> On Tuesday, 17 August 2021, Kristaps Cudars
>>>>>> <kr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Wei,
>>>>>>>
>>>>>>> Published this month's RFC 9099 and explains in different
>>>>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>>>>> https://www.rfc-editor.org/rfc/rfc9099.html
>> <https://www.rfc-editor.org/rfc/rfc9099.html>
>>>>>>>
>>>>>>>
>>>>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
>>>>>>>> Hi Wido,
>>>>>>>>
>>>>>>>> (cc to Rohit and Alex)
>>>>>>>>
>>>>>>>> It is a good suggestion to use FRR for ipv6. The configuration
>>>>>>>> is quite simple and the VMs can get SLAAC, routes, etc.
>>>>>>>>
>>>>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>>>>> https://datatracker.ietf.org/doc/html/rfc4941
>> <https://datatracker.ietf.org/doc/html/rfc4941>
>>>>>>>>
>>>>>>>> You are right. To use static routing, the admins need to
>>>>>>>> configure the routes in the upstream router, and add some ipv6
>>>>>>>> ranges (eg
>>>>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
>>>>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
>>>>>>>> pick up an IPv6 range
>>>>>>> and
>>>>>>>> assign it to an isolated network or vpc. @Rohit, correct me if
>>>>>>>> I'm
>>>>> wrong.
>>>>>>>>
>>>>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
>>>>>>>> router/VR, router/dns etc via RA messages) will be the only
>>>>>>>> option for now (related
>>>>>>> to
>>>>>>>> your pr https://github.com/apache/cloudstack/pull/3077)
>> <https://github.com/apache/cloudstack/pull/3077)> . Would it
>>>>>>>> be
>>>>>>> good
>>>>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
>>>>>>>> as an option in cloudstack ? The advantages are
>>>>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
>>>>>>>> addresses can be changed
>>>>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>>>>> The problem is, stateful dhcpv6 does not support
>>>>>>>> routers,nameservers,
>>>>>>> etc.
>>>>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>>>>
>>>>>>>> -Wei
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> See my inline responses:
>>>>>>>>>
>>>>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
>>>>>>>>>> discussing
>>>>>>> them
>>>>>>>>> with Alex and came up with a PoC/design which can be
>>>>>>>>> implemented in the following phases:
>>>>>>>>>>
>>>>>>>>>> * Phase1: implement ipv6 support in isolated networks
>>>>>>>>>> and
>>>>> VPC
>>>>>>> with
>>>>>>>>> static routing
>>>>>>>>>> * Phase2: discuss and implement support for dynamic
>>>>>>>>>> routing
>>>>>>> (TBD)
>>>>>>>>>>
>>>>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>>>>
>>>>>>>>>> * IPv6 address management:
>>>>>>>>>> * At the zone level root-admin specifies a /64
>>>>>>>>>> public
>>>>> range
>>>>>>> that
>>>>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6
>>>>>>>>> range for
>>>>>>> guest
>>>>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>>>>> * On creation of any IPv6 enabled isolated network
>>>>>>>>>> or VPC
>>>>>>> tier,
>>>>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>>>>> * We assume SLAAC and autoconfiguration, no DHCPv6
>>>>>>>>>> in the
>>>>>>> zone
>>>>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
>>>>>>>>> of
>>>>>>> slaac be
>>>>>>>>> explored?)
>>>>>>>>>
>>>>>>>>> Privacy Extensions are only a concern for client devices which
>>>>>>>>> roam between different IPv6 networks.
>>>>>>>>>
>>>>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>>>>
>>>>>>>>> This way a network like Google, Facebook, etc could track your
>>>>>>>>> device moving from network to network if they only look at the
>>>>>>>>> last 64-bits of the IPv6 address.
>>>>>>>>>
>>>>>>>>> For servers this is not a problem as you already know in which
>>>>>>>>> network they are.
>>>>>>>>>
>>>>>>>>>> * Network offerings: root-admin can create new network
>>>>> offerings
>>>>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>>>>> * ipv4 only (default, for backward compatibility all
>>>>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>>>>> * ipv4-and-ipv6
>>>>>>>>>> * ipv6-only (this can be phase 1.b)
>>>>>>>>>> * A new routing option: static (phase1), dynamic
>>>>>>>>>> (phase2,
>>>>>>> with
>>>>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>>>>
>>>>>>>>> This means that the network admin will need to statically route
>>>>>>>>> the
>>>>>>> IPv6
>>>>>>>>> subnet to the VR's outside IPv6 address, for example, on a
>>>>>>>>> JunOS
>>>>>>> router:
>>>>>>>>>
>>>>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
>>>>>>>>> next-hop
>>>>>>>>> 2001:db8:100::50
>>>>>>>>>
>>>>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
>>>>>>>>> the outside (/64) network. In reality this will probably be a
>>>>>>>>> longer address, but this is for just the example.
>>>>>>>>>
>>>>>>>>>> * VR changes:
>>>>>>>>>> * VR gets its guest and public nics set to inet6
>>>>>>>>>> auto
>>>>>>>>>> * For each /64 allocated to guest network and VPC
>>>>>>>>>> tiers,
>>>>>>> radvd
>>>>>>>>> is configured to do RA
>>>>>>>>>
>>>>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
>>>>>>>>> might already want to look into FRRouting. FRR can also
>>>>>>>>> advertise RAs while not doing any routing.
>>>>>>>>>
>>>>>>>>> interface ens4
>>>>>>>>> no ipv6 nd suppress-ra
>>>>>>>>> ipv6 nd prefix 2001:db8:500::/64
>>>>>>>>> ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>>>>
>>>>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
>> <http://docs.frrouting.org/en/latest/ipv6.html>
>>>>>>>>>
>>>>>>>>>> * Firewall: a new ipv6 zone/chain is created for
>>>>>>>>>> ipv6
>>>>> where
>>>>>>> ipv6
>>>>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
>>>>>>>>> between
>>>>>>> VPC
>>>>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>>>>
>>>>>>>>> Please take a look at the existing security_group.py script
>>>>>>>>> which implements RFC4890
>>>>>>>>>
>>>>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
>> <https://datatracker.ietf.org/doc/html/rfc4890>
>>>>>>>>>
>>>>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should
>>>>>>>>> always be allowed.
>>>>>>>>>
>>>>>>>>>> * It is assumed that static routes are created on
>>>>>>>>>> the
>>>>>>> core/main
>>>>>>>>> router by the admin or automated using some scripts/tools; for
>>>>>>>>> this CloudStack will announce events with details of /64
>>>>>>>>> networks and VR's public IPv6 address that can be consumed by a
>>>>>>>>> rabbitmq/message bus
>>>>>>> client
>>>>>>>>> (for example), or a custom cron job or script as part of
>>>>> orchestration.
>>>>>>>>> (this wouldn't be necessary for dynamic routing bgp with
>>>>>>>>> phase2)\\
>>>>>>>>>
>>>>>>>>> You would only need to announce the /48 or /56 allocated to the
>>>>>>>>> VR, that's all. You don't need to inform the upstream router
>>>>>>>>> about the /64 subnets created within that larger subnet.
>>>>>>>>>
>>>>>>>>>> * Guest Networking: With SLAAC, it's easy for
>>>>>>>>>> CloudStack to
>>>>>>>>> calculate allocate and use a /64 and determine the IPv6 address
>>>>>>>>> of VR
>>>>>>> nics
>>>>>>>>> and guest VM nics
>>>>>>>>>> * A user create an isolated network/VPC with an
>>>>>>>>>> offering
>>>>>>> that is
>>>>>>>>> ipv6 enabled
>>>>>>>>>> * A user can manage firewall for the IPv6
>>>>>>>>>> address/guest
>>>>> nics;
>>>>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>>>>> * A users can run workloads in the guest VMs that
>>>>>>>>>> listen
>>>>> on
>>>>>>>>> publically routable ipv6 addresses
>>>>>>>>>> * Usage/billing etc continue to work, no change
>>>>>>>>>> needed
>>>>>>>>>>
>>>>>>>>>> Network layout:
>>>>>>>>>>
>>>>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
>>>>>>>>>> VLAN] ->
>>>>>>>>> [guest VMs/nics]
>>>>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>>>>
>>>>>>>>>> Thoughts, feedback?
>>>>>>>>>
>>>>>>>>> Looks doable!
>>>>>>>>>
>>>>>>>>> Side-note: It would be very cool if you could use parts of this
>>>>>>>>> implementation to also route /48, /56, or /60 subnets to
>>>>>>>>> individual VMs in Shared networks.
>>>>>>>>>
>>>>>>>>> Why? This allows for running Docker containers with native IPv6
>>>>>>>>> inside the VM or running a (Open)VPN server inside a VM which
>>>>>>>>> then assigns public IPv6 addresses to clients connected.
>>>>>>>>>
>>>>>>>>> Instead of routing the subnet to a VR we route the subnet to a
>>>>>>>>> single instance in a shared network.
>>>>>>>>>
>>>>>>>>> If we could then also move these subnets between Instances
>>>>>>>>> easily one can quickly migrate to a different instance while
>>>>>>>>> keeping the same IPv6 subnet.
>>>>>>>>>
>>>>>>>>> Wido
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>>>>
>>>>>>>>>> * Created an isolated network and deployed a VM in my
>>>>>>>>>> home
>>>>> lab
>>>>>>>>>> The VR running on KVM has following nics
>>>>>>>>>> eth0 - guest network
>>>>>>>>>> eth1 - link local
>>>>>>>>>> eth2 - public network
>>>>>>>>>>
>>>>>>>>>> * I setup a custom openwrt router on a RPi4 to serve as
>>>>>>>>>> a
>>>>>>> toy-core
>>>>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker
>>>>>>>>> and I got
>>>>>>> a
>>>>>>>>> /48 allocated. My configuration looks like:
>>>>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>>>>
>>>>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>>>>> subnet/prefix
>>>>>>>>> 0:
>>>>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>>>>> *
>>>>>>>>>> *
>>>>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
>>>>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
>>>>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
>>>>>>>>>> net.ipv6.conf.all.autoconf = 1
>>>>>>>>>>
>>>>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
>>>>>>>>>> configured the nics:
>>>>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
>>>>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
>>>>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
>>>>>>>>>> network without cleanup to have it
>>>>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>>>>
>>>>>>>>>> *
>>>>>>>>>> Next, I created a /64 network for the isolated guest network
>>>>>>>>>> on eth0
>>>>>>> of
>>>>>>>>> VR using radvd:
>>>>>>>>>>
>>>>>>>>>> # cat /etc/radvd.conf
>>>>>>>>>> interface eth0
>>>>>>>>>> {
>>>>>>>>>> AdvSendAdvert on;
>>>>>>>>>> MinRtrAdvInterval 5;
>>>>>>>>>> MaxRtrAdvInterval 15;
>>>>>>>>>> prefix 2001:470:ed36:1::/64
>>>>>>>>>> {
>>>>>>>>>> AdvOnLink on;
>>>>>>>>>> AdvAutonomous on;
>>>>>>>>>> };
>>>>>>>>>> };
>>>>>>>>>> systemctl restart radvd
>>>>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
>>>>>>>>>> this
>>>>>>>>> ...:1::/64 network
>>>>>>>>>> * Finally I added a static route in toy core-router for
>>>>>>>>>> the
>>>>> new
>>>>>>> /64
>>>>>>>>> IPv6 range in the isolated network
>>>>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>>>>> eth2> dev
>>>>>>>>> <local lan nic>
>>>>>>>>>> *
>>>>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
>>>>>>>>>> for the
>>>>>>>>> new /64 network
>>>>>>>>>>
>>>>>>>>>> And voila all done! I create a domain AAAA record that points
>>>>>>>>>> to my
>>>>>>>>> guest VM IPv6 address a test webserver on
>>>>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>> <http://ipv6-isolated-ntwk-demo.yadav.cloud/>
>>>>>>>>>>
>>>>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
>>>>>>>>>> after a
>>>>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
RE: IPV6 in Isolated/VPC networks
Posted by Alex Mattioli <Al...@shapeblue.com>.
> ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
That's correct
> Or a larger subnet:
> ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4
Not really, the subnets for isolated/VPC networks are always /64. Which means also no real need to include subnets as well.
Cheers
Alex
-----Original Message-----
From: Wido den Hollander <wi...@widodh.nl>
Sent: 12 May 2022 16:04
To: Abhishek Kumar <Ab...@shapeblue.com>; dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks
On 5/12/22 09:55, Abhishek Kumar wrote:
> Hi Wido,
>
> I do not understand what you mean by WAB address but
WAB was a type. I meant WAN.
> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the public IP of the network
> (IPv6 of the public NIC of the network VR) in the sample.
> Yes, route for fd23:313a:2f53:3cbf::/64 need to be added to this IP.
> fd23:313a:2f53:3cbf::/64 is guest IPv6 CIDR of the network here.
>
So that means that I would need to run this command on my upstream router:
ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
Or a larger subnet:
ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4
> Currently, the message on event bus does not include subnet. Should
> that be included?
Yes, because then you can pickup those messages and inject the route via ExaBGP into a routing table right away.
> In case of VPCs, there could be multiple tiers which will need
> multiple routes to be added. Will that be an issue if we include
> current network/tier subnet in the event message?
No, as long as it points to the same VR you simply have multiple subnets being routed to the same VR.
I do wonder what happens if you destroy the VR and create a new one. The WAN address then changes (due to SLAAC) and thus the routes need to be re-programmed.
Wido
>
> Regards,
> Abhishek
>
>
>
>
>
>
> ----------------------------------------------------------------------
> --
> *From:* Wido den Hollander <wi...@widodh.nl>
> *Sent:* 10 May 2022 19:01
> *To:* dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek
> Kumar <Ab...@shapeblue.com>
> *Subject:* Re: IPV6 in Isolated/VPC networks
>
> Hi,
>
> Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
>> Yes. When a public IPv6 is assigned or released, CloudStack will publish event with type NET.IP6ASSIGN, NET.IP6RELEASE.
>> These event notifications can be tracked. And with improvements in events framework, these event messages will have network uuid as entityuuid and Network as entitytype. Using this network can be queried using to list IPv6 routes that need to be added.
>>
>> Sample event message,
>>
>> {"eventDateTime":"2022-05-10 09:32:12
>> +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":
>> "14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned public
>> IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID:
>> 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":
>> "bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-
>> af19-1e00320001f3","entity":"Network","status":"Completed"}
>>
>>
>> ?Sample API call,
>>> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98
>>> filter=id,name,ip6routes
>> {
>> "count": 1,
>> "network": [
>> {
>> "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
>> "ip6routes": [
>> {
>> "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
>> "subnet": "fd23:313a:2f53:3cbf::/64"
>> }
>
> Looking at this JSON, does this mean that
> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
>
> And that I would need to (statically) route fd23:313a:2f53:3cbf::/64
> to that IP?
>
> The event message does not include the subnet, that makes it a bit
> more difficult as you would then also need to do a API-call to gather
> that information.
>
> Wido
>
> P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an
> AAAA-record for IPv6!
>
>> ],
>> "name": "routing_test"
>> }
>> ]
>> }
>>
>>
>>
>>
>> ________________________________
>> From: Wido den Hollander <wi...@widodh.nl>
>> Sent: 10 May 2022 13:59
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek
>> Kumar <Ab...@shapeblue.com>
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
>>> Hi all,
>>>
>>> IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
>>> Documentation for it is available at
>>> http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.htm
>>> l#isolated-network-and-vpc-tier
> <http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html
> #isolated-network-and-vpc-tier> (will be available in the official
> docs once 4.17.0 version of docs is built).
>>>
>>
>> Great work!
>>
>> I see only static routing is supported. But do we publish something
>> on the message bus once a new VR/VPC is created?
>>
>> This way you could pick up these messages and have the network create
>> a
>> (static) route based on those.
>>
>> ExaBGP for example could be used to inject such routes.
>>
>> Wido
>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>>> in+Isolated+Network+and+VPC+with+Static+Routing
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
> n+Isolated+Network+and+VPC+with+Static+Routing>
>>>
>>> Regards,
>>> Abhishek
>>>
>>> ________________________________
>>> From: Rohit Yadav <ro...@shapeblue.com>
>>> Sent: 13 September 2021 14:30
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Thanks Alex, Wei. I've updated the docs here:
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>>> in+Isolated+Network+and+VPC+with+Static+Routing
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
> n+Isolated+Network+and+VPC+with+Static+Routing>
>>>
>>> I'll leave the thread open for futher discussion/ideas/feedback. I
>>> think we've completed the phase1 design doc including all feedback
>>> comments for adding IPv6 support in CloudStack and some initial
>>> poc/work can be started. My colleagues and I will keep everyone
>>> posted on this thread and/or on a Github PR as and when we're able
>>> to
> start our work on the same (after 4.16, potentially towards 4.17).
>>>
>>>
>>> Regards.
>>>
>>> ________________________________
>>> From: Wei ZHOU <us...@gmail.com>
>>> Sent: Friday, September 10, 2021 20:22
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Agree with Alex.
>>> We only need to know how many /64 are allocated. We do not care how
>>> many
>>> ipv6 addresses are used by VMs.
>>>
>>> -Wei
>>>
>>> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli
>>> <Al...@shapeblue.com>
>>> wrote:
>>>
>>>> Hi Rohit,
>>>>
>>>> I'd go for option 2, don't see a point tracking anything smaller
>>>> than a
>>>> /64 tbh.
>>>>
>>>> Cheers
>>>> Alex
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>> Sent: 09 September 2021 12:44
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>>>> agreements:
>>>>
>>>> * Allocate /64 for both isolated network and VPC tiers, no
>>>>large allocation of prefixes to VPC (cons: more static routing
>>>>rules for upstream
>>>> router/admins)
>>>> * All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if
>>>>zone has a dedicated /64 prefix/block for systemvms
>>>>
>>>> The only outstanding question now is:
>>>>
>>>> * How do we manage IPv6 usage? Can anyone advise how we do
>>>>IPv6 usage for shared network (design, implementation and
>>>>use-cases?)
>>>> Option1: We don't do it, all user VMs nics have ipv4 address which
>>>>whose usage we don't track. For public VR/nics/networks, we can
>>>>simply add the
>>>> IPv6 details for a related IPv4 address.
>>>> Option2: Implement a separate, first-class IPv6 address or /64
>>>>prefix tracking/management and usage for all VMs and systemvms nic
>>>>(this means account/domain level limits and new billing/records)
>>>> Option3: other thoughts?
>>>>
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Alex Mattioli <Al...@shapeblue.com>
>>>> Sent: Wednesday, September 8, 2021 23:24
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: RE: IPV6 in Isolated/VPC networks
>>>>
>>>> Hi Rohit, Kristaps,
>>>>
>>>> I'd say option 1 as well, it does create a bit more overhead with
>>>> static routes but if that's automated for a VPC it can also be
>>>> easily automated for several tiers of a VPC. We also don't
>>>> constrain the amount of tiers in a VPC.
>>>> It has the added advantage to be closer to the desired behaviour
>>>> with dynamic routing in the future, where a VPC VR can announce
>>>> several subnets upstream.
>>>>
>>>> Cheers
>>>> Alex
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>> Sent: 08 September 2021 19:04
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Hi Kristaps,
>>>>
>>>> Thanks for sharing, I suppose that means individual tiers should be
>>>> allocated /64 instead of larger ipv6 blocks to the whole VPC which
>>>> could cause wastage.
>>>>
>>>> Any objection from anybody?
>>>>
>>>> Regards.
>>>> ________________________________
>>>> From: Kristaps Cudars <kr...@gmail.com>
>>>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Hello,
>>>>
>>>> I asked networking team to comment on "How should the IPv6
>>>> block/allocation work in VPCs?"
>>>> Option1: They haven't seen lately devices with limits on how many
>>>> static routes can be created.
>>>> Option2: With /60 and /62 assignments and big quantity of routers
>>>> IPv6 assignment from RIPE NNC can be drained fast.
>>>>
>>>> /48 contains 64000 /64
>>>> /60 contains 16 /64
>>>> 64000 / 16 = 4000 routers
>>>>
>>>>
>>>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
>>>>> All,
>>>>>
>>>>> After another iteration with Alex, I've updated the design doc.
>>>>> Kindly
>>>> review:
>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppor
>>>>> t+in
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
> n>
>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>
>>>>>
>>>>> ... and advise on some outstanding questions:
>>>>>
>>>>> * How should the IPv6 block/allocation work in VPCs?
>>>>> Option1: Should this be simply /64 allocation on any new tier, the
>>>>>cons of this option is one static route/rule per VPC tier. (many
>>>>>upstream routers may have limit on no. of static routes?)
>>>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or
>>>>>/62
>>>> (4 tiers) for the VPC, this can be filtered based on the
>>>> vpc.max.networks global setting (3 is default). The pros of this
>>>> option are less no. of static route/rule and easy programming, but
>>>> potential wastage of multiple
>>>> /64 prefix blocks for unused/uncreated tiers.
>>>>> * How do we manage IPv6 usage? Can anyone advise how we do
>>>>>IPv6
>>>> usage for shared network (design, implementation and use-cases?)
>>>>> Option1: We don't do it, all user VMs nics have ipv4 address which
>>>>> whose
>>>> usage we don't track. For public VR/nics/networks, we can simply
>>>> add the
>>>> IPv6 details for a related IPv4 address.
>>>>> Option2: Implement a separate, first-class IPv6 address or /64
>>>>> prefix
>>>> tracking/management and usage for all VMs and systemvms nic (this
>>>> means account/domain level limits and new billing/records)
>>>>> * Enable IPv6 on systemvms (specifically SSVM and CPVM) by
>>>>>default
>>>> if zone has a IPv6 address block allocated/assigned for use for
>>>> systemvms (this was mainly thought for VRs, but why no ssvm and
>>>> cpvms too - any cons of this?)
>>>>> *
>>>>>
>>>>> Regards.
>>>>>
>>>>> ________________________________
>>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>>> Sent: Thursday, August 19, 2021 15:45
>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I've taken feedback from this thread and wrote this design doc:
>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppor
>>>>> t+in
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
> n>
>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>
>>>>> Kindly review and advise if I missed anything or anything that
>>>>> needs to
>>>> be changed/updated. You may comment on the wiki directly too.
>>>>>
>>>>> Kindly suggest your views on the following (also in the design doc
>>>> above):
>>>>>
>>>>> Outstanding Questions:
>>>>>
>>>>> * Should admin or user be able to specify how VPC super
>>>>>CIDRs are
>>>> created/needed; for example a user can ask for VPC with /60 super
>>>> CIDR? Or should CloudStack automatically find/allocate a /64 for a
>>>> new VPC tier from the root-admin configured /64-/48 block?
>>>>> * Should we explore FRR and iBGP or other strategies to do
>>>>>dynamic
>>>> routing which may not require advance/complex configuration in the
>>>> VR or for the users/admin?
>>>>> * With SLAAC and no dhcpv6, is there a way to support
>>>>>secondary IPv6
>>>> addresses (or floating IPv6 addresses for VR/public traffic) for
>>>> guest VM's nics?
>>>>> * Any thoughts on UI/UX for firewall/routing management?
>>>>> * Any other feature/support for isolated network or VPC
>>>>>feature that
>>>> must be explored or supported such as PF, VPN, LB, vpc static
>>>> routes, vpc gateway etc.
>>>>> * For usage - should we have any consideration, or should we
>>>>>assume
>>>> that IPv4 and IPv6 address will go together for every nic; so IPv6
>>>> usage for a nic is in tandem with Ipv4 address for a nic, i.e. no
>>>> explicit/new biling/usage needed?
>>>>> * For smoketests, local dev-test should we explore ULA?
>>>>>Unique Local
>>>> Address - in the range fc00::/7. Typically only within the 'local'
>>>> half fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
>>>> This prefix can be randomly generated at first install by
>>>> CloudStack in a zone using zoneid etc?
>>>>> * Should we expand support for VR diagnostic feature to
>>>>>include
>>>> support for ipv6, traceroute6?
>>>>> * Discuss - expand support/ability to allocate a /60, or /56
>>>>>etc
>>>> prefix to an individual VM in shared network (Wido's suggestion)
>>>>>
>>>>>
>>>>> Regards.
>>>>>
>>>>> ________________________________
>>>>> From: Wei ZHOU <us...@gmail.com>
>>>>> Sent: Tuesday, August 17, 2021 21:16
>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>>>
>>>>> I am fine with not having stateful dhcpv6 and privacy
>>>>> extension/temporary address in phase 1. If community decides not
>>>>> to do eventually , it is also ok to me.
>>>>>
>>>>> We could explore how to better use secondary ipv6 addresses as
>>>>> Wido advised. It would be great if anyone share some user experience.
>>>>>
>>>>> -Wei
>>>>>
>>>>>
>>>>> On Tuesday, 17 August 2021, Kristaps Cudars
>>>>> <kr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Wei,
>>>>>>
>>>>>> Published this month's RFC 9099 and explains in different
>>>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>>>> https://www.rfc-editor.org/rfc/rfc9099.html
> <https://www.rfc-editor.org/rfc/rfc9099.html>
>>>>>>
>>>>>>
>>>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
>>>>>>> Hi Wido,
>>>>>>>
>>>>>>> (cc to Rohit and Alex)
>>>>>>>
>>>>>>> It is a good suggestion to use FRR for ipv6. The configuration
>>>>>>> is quite simple and the VMs can get SLAAC, routes, etc.
>>>>>>>
>>>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>>>> https://datatracker.ietf.org/doc/html/rfc4941
> <https://datatracker.ietf.org/doc/html/rfc4941>
>>>>>>>
>>>>>>> You are right. To use static routing, the admins need to
>>>>>>> configure the routes in the upstream router, and add some ipv6
>>>>>>> ranges (eg
>>>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
>>>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
>>>>>>> pick up an IPv6 range
>>>>>> and
>>>>>>> assign it to an isolated network or vpc. @Rohit, correct me if
>>>>>>> I'm
>>>> wrong.
>>>>>>>
>>>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
>>>>>>> router/VR, router/dns etc via RA messages) will be the only
>>>>>>> option for now (related
>>>>>> to
>>>>>>> your pr https://github.com/apache/cloudstack/pull/3077)
> <https://github.com/apache/cloudstack/pull/3077)> . Would it
>>>>>>> be
>>>>>> good
>>>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
>>>>>>> as an option in cloudstack ? The advantages are
>>>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
>>>>>>> addresses can be changed
>>>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>>>> The problem is, stateful dhcpv6 does not support
>>>>>>> routers,nameservers,
>>>>>> etc.
>>>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>>>
>>>>>>> -Wei
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> See my inline responses:
>>>>>>>>
>>>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
>>>>>>>>> discussing
>>>>>> them
>>>>>>>> with Alex and came up with a PoC/design which can be
>>>>>>>> implemented in the following phases:
>>>>>>>>>
>>>>>>>>> * Phase1: implement ipv6 support in isolated networks
>>>>>>>>>and
>>>> VPC
>>>>>> with
>>>>>>>> static routing
>>>>>>>>> * Phase2: discuss and implement support for dynamic
>>>>>>>>>routing
>>>>>> (TBD)
>>>>>>>>>
>>>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>>>
>>>>>>>>> * IPv6 address management:
>>>>>>>>> * At the zone level root-admin specifies a /64
>>>>>>>>>public
>>>> range
>>>>>> that
>>>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6
>>>>>>>> range for
>>>>>> guest
>>>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>>>> * On creation of any IPv6 enabled isolated network
>>>>>>>>>or VPC
>>>>>> tier,
>>>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>>>> * We assume SLAAC and autoconfiguration, no DHCPv6
>>>>>>>>>in the
>>>>>> zone
>>>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
>>>>>>>> of
>>>>>> slaac be
>>>>>>>> explored?)
>>>>>>>>
>>>>>>>> Privacy Extensions are only a concern for client devices which
>>>>>>>> roam between different IPv6 networks.
>>>>>>>>
>>>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>>>
>>>>>>>> This way a network like Google, Facebook, etc could track your
>>>>>>>> device moving from network to network if they only look at the
>>>>>>>> last 64-bits of the IPv6 address.
>>>>>>>>
>>>>>>>> For servers this is not a problem as you already know in which
>>>>>>>> network they are.
>>>>>>>>
>>>>>>>>> * Network offerings: root-admin can create new network
>>>> offerings
>>>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>>>> * ipv4 only (default, for backward compatibility all
>>>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>>>> * ipv4-and-ipv6
>>>>>>>>> * ipv6-only (this can be phase 1.b)
>>>>>>>>> * A new routing option: static (phase1), dynamic
>>>>>>>>>(phase2,
>>>>>> with
>>>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>>>
>>>>>>>> This means that the network admin will need to statically route
>>>>>>>> the
>>>>>> IPv6
>>>>>>>> subnet to the VR's outside IPv6 address, for example, on a
>>>>>>>> JunOS
>>>>>> router:
>>>>>>>>
>>>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
>>>>>>>> next-hop
>>>>>>>> 2001:db8:100::50
>>>>>>>>
>>>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
>>>>>>>> the outside (/64) network. In reality this will probably be a
>>>>>>>> longer address, but this is for just the example.
>>>>>>>>
>>>>>>>>> * VR changes:
>>>>>>>>> * VR gets its guest and public nics set to inet6
>>>>>>>>>auto
>>>>>>>>> * For each /64 allocated to guest network and VPC
>>>>>>>>>tiers,
>>>>>> radvd
>>>>>>>> is configured to do RA
>>>>>>>>
>>>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
>>>>>>>> might already want to look into FRRouting. FRR can also
>>>>>>>> advertise RAs while not doing any routing.
>>>>>>>>
>>>>>>>> interface ens4
>>>>>>>> no ipv6 nd suppress-ra
>>>>>>>> ipv6 nd prefix 2001:db8:500::/64
>>>>>>>> ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>>>
>>>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
> <http://docs.frrouting.org/en/latest/ipv6.html>
>>>>>>>>
>>>>>>>>> * Firewall: a new ipv6 zone/chain is created for
>>>>>>>>>ipv6
>>>> where
>>>>>> ipv6
>>>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
>>>>>>>> between
>>>>>> VPC
>>>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>>>
>>>>>>>> Please take a look at the existing security_group.py script
>>>>>>>> which implements RFC4890
>>>>>>>>
>>>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
> <https://datatracker.ietf.org/doc/html/rfc4890>
>>>>>>>>
>>>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should
>>>>>>>> always be allowed.
>>>>>>>>
>>>>>>>>> * It is assumed that static routes are created on
>>>>>>>>>the
>>>>>> core/main
>>>>>>>> router by the admin or automated using some scripts/tools; for
>>>>>>>> this CloudStack will announce events with details of /64
>>>>>>>> networks and VR's public IPv6 address that can be consumed by a
>>>>>>>> rabbitmq/message bus
>>>>>> client
>>>>>>>> (for example), or a custom cron job or script as part of
>>>> orchestration.
>>>>>>>> (this wouldn't be necessary for dynamic routing bgp with
>>>>>>>> phase2)\\
>>>>>>>>
>>>>>>>> You would only need to announce the /48 or /56 allocated to the
>>>>>>>> VR, that's all. You don't need to inform the upstream router
>>>>>>>> about the /64 subnets created within that larger subnet.
>>>>>>>>
>>>>>>>>> * Guest Networking: With SLAAC, it's easy for
>>>>>>>>>CloudStack to
>>>>>>>> calculate allocate and use a /64 and determine the IPv6 address
>>>>>>>> of VR
>>>>>> nics
>>>>>>>> and guest VM nics
>>>>>>>>> * A user create an isolated network/VPC with an
>>>>>>>>>offering
>>>>>> that is
>>>>>>>> ipv6 enabled
>>>>>>>>> * A user can manage firewall for the IPv6
>>>>>>>>>address/guest
>>>> nics;
>>>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>>>> * A users can run workloads in the guest VMs that
>>>>>>>>>listen
>>>> on
>>>>>>>> publically routable ipv6 addresses
>>>>>>>>> * Usage/billing etc continue to work, no change
>>>>>>>>>needed
>>>>>>>>>
>>>>>>>>> Network layout:
>>>>>>>>>
>>>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
>>>>>>>>> VLAN] ->
>>>>>>>> [guest VMs/nics]
>>>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>>>
>>>>>>>>> Thoughts, feedback?
>>>>>>>>
>>>>>>>> Looks doable!
>>>>>>>>
>>>>>>>> Side-note: It would be very cool if you could use parts of this
>>>>>>>> implementation to also route /48, /56, or /60 subnets to
>>>>>>>> individual VMs in Shared networks.
>>>>>>>>
>>>>>>>> Why? This allows for running Docker containers with native IPv6
>>>>>>>> inside the VM or running a (Open)VPN server inside a VM which
>>>>>>>> then assigns public IPv6 addresses to clients connected.
>>>>>>>>
>>>>>>>> Instead of routing the subnet to a VR we route the subnet to a
>>>>>>>> single instance in a shared network.
>>>>>>>>
>>>>>>>> If we could then also move these subnets between Instances
>>>>>>>> easily one can quickly migrate to a different instance while
>>>>>>>> keeping the same IPv6 subnet.
>>>>>>>>
>>>>>>>> Wido
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>>>
>>>>>>>>> * Created an isolated network and deployed a VM in my
>>>>>>>>>home
>>>> lab
>>>>>>>>> The VR running on KVM has following nics
>>>>>>>>> eth0 - guest network
>>>>>>>>> eth1 - link local
>>>>>>>>> eth2 - public network
>>>>>>>>>
>>>>>>>>> * I setup a custom openwrt router on a RPi4 to serve as
>>>>>>>>>a
>>>>>> toy-core
>>>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker
>>>>>>>> and I got
>>>>>> a
>>>>>>>> /48 allocated. My configuration looks like:
>>>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>>>
>>>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>>>> subnet/prefix
>>>>>>>> 0:
>>>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
>>>>>>>>>net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
>>>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
>>>>>>>>>net.ipv6.conf.all.autoconf = 1
>>>>>>>>>
>>>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
>>>>>>>>> configured the nics:
>>>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
>>>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
>>>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
>>>>>>>>> network without cleanup to have it
>>>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>>>
>>>>>>>>> *
>>>>>>>>> Next, I created a /64 network for the isolated guest network
>>>>>>>>>on eth0
>>>>>> of
>>>>>>>> VR using radvd:
>>>>>>>>>
>>>>>>>>> # cat /etc/radvd.conf
>>>>>>>>> interface eth0
>>>>>>>>> {
>>>>>>>>> AdvSendAdvert on;
>>>>>>>>> MinRtrAdvInterval 5;
>>>>>>>>> MaxRtrAdvInterval 15;
>>>>>>>>> prefix 2001:470:ed36:1::/64
>>>>>>>>> {
>>>>>>>>> AdvOnLink on;
>>>>>>>>> AdvAutonomous on;
>>>>>>>>> };
>>>>>>>>> };
>>>>>>>>> systemctl restart radvd
>>>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
>>>>>>>>>this
>>>>>>>> ...:1::/64 network
>>>>>>>>> * Finally I added a static route in toy core-router for
>>>>>>>>>the
>>>> new
>>>>>> /64
>>>>>>>> IPv6 range in the isolated network
>>>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>>>> eth2> dev
>>>>>>>> <local lan nic>
>>>>>>>>> *
>>>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
>>>>>>>>>for the
>>>>>>>> new /64 network
>>>>>>>>>
>>>>>>>>> And voila all done! I create a domain AAAA record that points
>>>>>>>>> to my
>>>>>>>> guest VM IPv6 address a test webserver on
>>>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
> <http://ipv6-isolated-ntwk-demo.yadav.cloud/>
>>>>>>>>>
>>>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
>>>>>>>>> after a
>>>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
Re: IPV6 in Isolated/VPC networks
Posted by Wido den Hollander <wi...@widodh.nl>.
On 5/12/22 09:55, Abhishek Kumar wrote:
> Hi Wido,
>
> I do not understand what you mean by WAB address but
WAB was a type. I meant WAN.
> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the public IP of the network
> (IPv6 of the public NIC of the network VR) in the sample.
> Yes, route for fd23:313a:2f53:3cbf::/64 need to be added to this IP.
> fd23:313a:2f53:3cbf::/64 is guest IPv6 CIDR of the network here.
>
So that means that I would need to run this command on my upstream router:
ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
Or a larger subnet:
ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4
> Currently, the message on event bus does not include subnet. Should that
> be included?
Yes, because then you can pickup those messages and inject the route via
ExaBGP into a routing table right away.
> In case of VPCs, there could be multiple tiers which will need multiple
> routes to be added. Will that be an issue if we include current
> network/tier subnet in the event message?
No, as long as it points to the same VR you simply have multiple subnets
being routed to the same VR.
I do wonder what happens if you destroy the VR and create a new one. The
WAN address then changes (due to SLAAC) and thus the routes need to be
re-programmed.
Wido
>
> Regards,
> Abhishek
>
>
>
>
>
>
> ------------------------------------------------------------------------
> *From:* Wido den Hollander <wi...@widodh.nl>
> *Sent:* 10 May 2022 19:01
> *To:* dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek
> Kumar <Ab...@shapeblue.com>
> *Subject:* Re: IPV6 in Isolated/VPC networks
>
> Hi,
>
> Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
>> Yes. When a public IPv6 is assigned or released, CloudStack will publish event with type NET.IP6ASSIGN, NET.IP6RELEASE.
>> These event notifications can be tracked. And with improvements in events framework, these event messages will have network uuid as entityuuid and Network as entitytype. Using this network can be queried using to list IPv6 routes that need to be added.
>>
>> Sample event message,
>>
>> {"eventDateTime":"2022-05-10 09:32:12 +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":"14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned public IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID: 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":"bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-af19-1e00320001f3","entity":"Network","status":"Completed"}
>>
>>
>> ?Sample API call,
>>> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98 filter=id,name,ip6routes
>> {
>> "count": 1,
>> "network": [
>> {
>> "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
>> "ip6routes": [
>> {
>> "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
>> "subnet": "fd23:313a:2f53:3cbf::/64"
>> }
>
> Looking at this JSON, does this mean that
> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
>
> And that I would need to (statically) route fd23:313a:2f53:3cbf::/64 to
> that IP?
>
> The event message does not include the subnet, that makes it a bit more
> difficult as you would then also need to do a API-call to gather that
> information.
>
> Wido
>
> P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an
> AAAA-record for IPv6!
>
>> ],
>> "name": "routing_test"
>> }
>> ]
>> }
>>
>>
>>
>>
>> ________________________________
>> From: Wido den Hollander <wi...@widodh.nl>
>> Sent: 10 May 2022 13:59
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek Kumar <Ab...@shapeblue.com>
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
>>> Hi all,
>>>
>>> IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
>>> Documentation for it is available at http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier
> <http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier>
> (will be available in the official docs once 4.17.0 version of docs is
> built).
>>>
>>
>> Great work!
>>
>> I see only static routing is supported. But do we publish something on
>> the message bus once a new VR/VPC is created?
>>
>> This way you could pick up these messages and have the network create a
>> (static) route based on those.
>>
>> ExaBGP for example could be used to inject such routes.
>>
>> Wido
>>
>>> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing>
>>>
>>> Regards,
>>> Abhishek
>>>
>>> ________________________________
>>> From: Rohit Yadav <ro...@shapeblue.com>
>>> Sent: 13 September 2021 14:30
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Thanks Alex, Wei. I've updated the docs here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing>
>>>
>>> I'll leave the thread open for futher discussion/ideas/feedback. I think we've completed the phase1 design doc including all feedback comments for adding IPv6 support in CloudStack and some initial poc/work can be started. My colleagues and I will keep everyone posted on this thread and/or on a Github PR as and when we're able to
> start our work on the same (after 4.16, potentially towards 4.17).
>>>
>>>
>>> Regards.
>>>
>>> ________________________________
>>> From: Wei ZHOU <us...@gmail.com>
>>> Sent: Friday, September 10, 2021 20:22
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Agree with Alex.
>>> We only need to know how many /64 are allocated. We do not care how many
>>> ipv6 addresses are used by VMs.
>>>
>>> -Wei
>>>
>>> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <Al...@shapeblue.com>
>>> wrote:
>>>
>>>> Hi Rohit,
>>>>
>>>> I'd go for option 2, don't see a point tracking anything smaller than a
>>>> /64 tbh.
>>>>
>>>> Cheers
>>>> Alex
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>> Sent: 09 September 2021 12:44
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>>>> agreements:
>>>>
>>>> * Allocate /64 for both isolated network and VPC tiers, no large
>>>> allocation of prefixes to VPC (cons: more static routing rules for upstream
>>>> router/admins)
>>>> * All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a
>>>> dedicated /64 prefix/block for systemvms
>>>>
>>>> The only outstanding question now is:
>>>>
>>>> * How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage
>>>> for shared network (design, implementation and use-cases?)
>>>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>>>> usage we don't track. For public VR/nics/networks, we can simply add the
>>>> IPv6 details for a related IPv4 address.
>>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>>>> tracking/management and usage for all VMs and systemvms nic (this means
>>>> account/domain level limits and new billing/records)
>>>> Option3: other thoughts?
>>>>
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Alex Mattioli <Al...@shapeblue.com>
>>>> Sent: Wednesday, September 8, 2021 23:24
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: RE: IPV6 in Isolated/VPC networks
>>>>
>>>> Hi Rohit, Kristaps,
>>>>
>>>> I'd say option 1 as well, it does create a bit more overhead with static
>>>> routes but if that's automated for a VPC it can also be easily automated
>>>> for several tiers of a VPC. We also don't constrain the amount of tiers in
>>>> a VPC.
>>>> It has the added advantage to be closer to the desired behaviour with
>>>> dynamic routing in the future, where a VPC VR can announce several subnets
>>>> upstream.
>>>>
>>>> Cheers
>>>> Alex
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>> Sent: 08 September 2021 19:04
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Hi Kristaps,
>>>>
>>>> Thanks for sharing, I suppose that means individual tiers should be
>>>> allocated /64 instead of larger ipv6 blocks to the whole VPC which could
>>>> cause wastage.
>>>>
>>>> Any objection from anybody?
>>>>
>>>> Regards.
>>>> ________________________________
>>>> From: Kristaps Cudars <kr...@gmail.com>
>>>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Hello,
>>>>
>>>> I asked networking team to comment on "How should the IPv6
>>>> block/allocation work in VPCs?"
>>>> Option1: They haven't seen lately devices with limits on how many static
>>>> routes can be created.
>>>> Option2: With /60 and /62 assignments and big quantity of routers IPv6
>>>> assignment from RIPE NNC can be drained fast.
>>>>
>>>> /48 contains 64000 /64
>>>> /60 contains 16 /64
>>>> 64000 / 16 = 4000 routers
>>>>
>>>>
>>>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
>>>>> All,
>>>>>
>>>>> After another iteration with Alex, I've updated the design doc. Kindly
>>>> review:
>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in>
>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>
>>>>>
>>>>> ... and advise on some outstanding questions:
>>>>>
>>>>> * How should the IPv6 block/allocation work in VPCs?
>>>>> Option1: Should this be simply /64 allocation on any new tier, the
>>>>> cons of this option is one static route/rule per VPC tier. (many
>>>>> upstream routers may have limit on no. of static routes?)
>>>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
>>>> (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks
>>>> global setting (3 is default). The pros of this option are less no. of
>>>> static route/rule and easy programming, but potential wastage of multiple
>>>> /64 prefix blocks for unused/uncreated tiers.
>>>>> * How do we manage IPv6 usage? Can anyone advise how we do IPv6
>>>> usage for shared network (design, implementation and use-cases?)
>>>>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>>>> usage we don't track. For public VR/nics/networks, we can simply add the
>>>> IPv6 details for a related IPv4 address.
>>>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>>>> tracking/management and usage for all VMs and systemvms nic (this means
>>>> account/domain level limits and new billing/records)
>>>>> * Enable IPv6 on systemvms (specifically SSVM and CPVM) by default
>>>> if zone has a IPv6 address block allocated/assigned for use for systemvms
>>>> (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons
>>>> of this?)
>>>>> *
>>>>>
>>>>> Regards.
>>>>>
>>>>> ________________________________
>>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>>> Sent: Thursday, August 19, 2021 15:45
>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I've taken feedback from this thread and wrote this design doc:
>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in>
>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>
>>>>> Kindly review and advise if I missed anything or anything that needs to
>>>> be changed/updated. You may comment on the wiki directly too.
>>>>>
>>>>> Kindly suggest your views on the following (also in the design doc
>>>> above):
>>>>>
>>>>> Outstanding Questions:
>>>>>
>>>>> * Should admin or user be able to specify how VPC super CIDRs are
>>>> created/needed; for example a user can ask for VPC with /60 super CIDR? Or
>>>> should CloudStack automatically find/allocate a /64 for a new VPC tier from
>>>> the root-admin configured /64-/48 block?
>>>>> * Should we explore FRR and iBGP or other strategies to do dynamic
>>>> routing which may not require advance/complex configuration in the VR or
>>>> for the users/admin?
>>>>> * With SLAAC and no dhcpv6, is there a way to support secondary IPv6
>>>> addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's
>>>> nics?
>>>>> * Any thoughts on UI/UX for firewall/routing management?
>>>>> * Any other feature/support for isolated network or VPC feature that
>>>> must be explored or supported such as PF, VPN, LB, vpc static routes, vpc
>>>> gateway etc.
>>>>> * For usage - should we have any consideration, or should we assume
>>>> that IPv4 and IPv6 address will go together for every nic; so IPv6 usage
>>>> for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new
>>>> biling/usage needed?
>>>>> * For smoketests, local dev-test should we explore ULA? Unique Local
>>>> Address - in the range fc00::/7. Typically only within the 'local' half
>>>> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
>>>> This prefix can be randomly generated at first install by CloudStack in a
>>>> zone using zoneid etc?
>>>>> * Should we expand support for VR diagnostic feature to include
>>>> support for ipv6, traceroute6?
>>>>> * Discuss - expand support/ability to allocate a /60, or /56 etc
>>>> prefix to an individual VM in shared network (Wido's suggestion)
>>>>>
>>>>>
>>>>> Regards.
>>>>>
>>>>> ________________________________
>>>>> From: Wei ZHOU <us...@gmail.com>
>>>>> Sent: Tuesday, August 17, 2021 21:16
>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>>>
>>>>> I am fine with not having stateful dhcpv6 and privacy
>>>>> extension/temporary address in phase 1. If community decides not to do
>>>>> eventually , it is also ok to me.
>>>>>
>>>>> We could explore how to better use secondary ipv6 addresses as Wido
>>>>> advised. It would be great if anyone share some user experience.
>>>>>
>>>>> -Wei
>>>>>
>>>>>
>>>>> On Tuesday, 17 August 2021, Kristaps Cudars
>>>>> <kr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Wei,
>>>>>>
>>>>>> Published this month's RFC 9099 and explains in different
>>>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>>>> https://www.rfc-editor.org/rfc/rfc9099.html
> <https://www.rfc-editor.org/rfc/rfc9099.html>
>>>>>>
>>>>>>
>>>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
>>>>>>> Hi Wido,
>>>>>>>
>>>>>>> (cc to Rohit and Alex)
>>>>>>>
>>>>>>> It is a good suggestion to use FRR for ipv6. The configuration is
>>>>>>> quite simple and the VMs can get SLAAC, routes, etc.
>>>>>>>
>>>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>>>> https://datatracker.ietf.org/doc/html/rfc4941
> <https://datatracker.ietf.org/doc/html/rfc4941>
>>>>>>>
>>>>>>> You are right. To use static routing, the admins need to configure
>>>>>>> the routes in the upstream router, and add some ipv6 ranges (eg
>>>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
>>>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
>>>>>>> pick up an IPv6 range
>>>>>> and
>>>>>>> assign it to an isolated network or vpc. @Rohit, correct me if I'm
>>>> wrong.
>>>>>>>
>>>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
>>>>>>> router/VR, router/dns etc via RA messages) will be the only option
>>>>>>> for now (related
>>>>>> to
>>>>>>> your pr https://github.com/apache/cloudstack/pull/3077)
> <https://github.com/apache/cloudstack/pull/3077)> . Would it
>>>>>>> be
>>>>>> good
>>>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
>>>>>>> as an option in cloudstack ? The advantages are
>>>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
>>>>>>> addresses can be changed
>>>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>>>> The problem is, stateful dhcpv6 does not support
>>>>>>> routers,nameservers,
>>>>>> etc.
>>>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>>>
>>>>>>> -Wei
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> See my inline responses:
>>>>>>>>
>>>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
>>>>>>>>> discussing
>>>>>> them
>>>>>>>> with Alex and came up with a PoC/design which can be implemented
>>>>>>>> in the following phases:
>>>>>>>>>
>>>>>>>>> * Phase1: implement ipv6 support in isolated networks and
>>>> VPC
>>>>>> with
>>>>>>>> static routing
>>>>>>>>> * Phase2: discuss and implement support for dynamic routing
>>>>>> (TBD)
>>>>>>>>>
>>>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>>>
>>>>>>>>> * IPv6 address management:
>>>>>>>>> * At the zone level root-admin specifies a /64 public
>>>> range
>>>>>> that
>>>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6 range
>>>>>>>> for
>>>>>> guest
>>>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>>>> * On creation of any IPv6 enabled isolated network or VPC
>>>>>> tier,
>>>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>>>> * We assume SLAAC and autoconfiguration, no DHCPv6 in the
>>>>>> zone
>>>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
>>>>>>>> of
>>>>>> slaac be
>>>>>>>> explored?)
>>>>>>>>
>>>>>>>> Privacy Extensions are only a concern for client devices which
>>>>>>>> roam between different IPv6 networks.
>>>>>>>>
>>>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>>>
>>>>>>>> This way a network like Google, Facebook, etc could track your
>>>>>>>> device moving from network to network if they only look at the
>>>>>>>> last 64-bits of the IPv6 address.
>>>>>>>>
>>>>>>>> For servers this is not a problem as you already know in which
>>>>>>>> network they are.
>>>>>>>>
>>>>>>>>> * Network offerings: root-admin can create new network
>>>> offerings
>>>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>>>> * ipv4 only (default, for backward compatibility all
>>>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>>>> * ipv4-and-ipv6
>>>>>>>>> * ipv6-only (this can be phase 1.b)
>>>>>>>>> * A new routing option: static (phase1), dynamic (phase2,
>>>>>> with
>>>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>>>
>>>>>>>> This means that the network admin will need to statically route
>>>>>>>> the
>>>>>> IPv6
>>>>>>>> subnet to the VR's outside IPv6 address, for example, on a JunOS
>>>>>> router:
>>>>>>>>
>>>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
>>>>>>>> next-hop
>>>>>>>> 2001:db8:100::50
>>>>>>>>
>>>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
>>>>>>>> the outside (/64) network. In reality this will probably be a
>>>>>>>> longer address, but this is for just the example.
>>>>>>>>
>>>>>>>>> * VR changes:
>>>>>>>>> * VR gets its guest and public nics set to inet6 auto
>>>>>>>>> * For each /64 allocated to guest network and VPC tiers,
>>>>>> radvd
>>>>>>>> is configured to do RA
>>>>>>>>
>>>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
>>>>>>>> might already want to look into FRRouting. FRR can also
>>>>>>>> advertise RAs while not doing any routing.
>>>>>>>>
>>>>>>>> interface ens4
>>>>>>>> no ipv6 nd suppress-ra
>>>>>>>> ipv6 nd prefix 2001:db8:500::/64
>>>>>>>> ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>>>
>>>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
> <http://docs.frrouting.org/en/latest/ipv6.html>
>>>>>>>>
>>>>>>>>> * Firewall: a new ipv6 zone/chain is created for ipv6
>>>> where
>>>>>> ipv6
>>>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
>>>>>>>> between
>>>>>> VPC
>>>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>>>
>>>>>>>> Please take a look at the existing security_group.py script
>>>>>>>> which implements RFC4890
>>>>>>>>
>>>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
> <https://datatracker.ietf.org/doc/html/rfc4890>
>>>>>>>>
>>>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should always
>>>>>>>> be allowed.
>>>>>>>>
>>>>>>>>> * It is assumed that static routes are created on the
>>>>>> core/main
>>>>>>>> router by the admin or automated using some scripts/tools; for
>>>>>>>> this CloudStack will announce events with details of /64
>>>>>>>> networks and VR's public IPv6 address that can be consumed by a
>>>>>>>> rabbitmq/message bus
>>>>>> client
>>>>>>>> (for example), or a custom cron job or script as part of
>>>> orchestration.
>>>>>>>> (this wouldn't be necessary for dynamic routing bgp with
>>>>>>>> phase2)\\
>>>>>>>>
>>>>>>>> You would only need to announce the /48 or /56 allocated to the
>>>>>>>> VR, that's all. You don't need to inform the upstream router
>>>>>>>> about the /64 subnets created within that larger subnet.
>>>>>>>>
>>>>>>>>> * Guest Networking: With SLAAC, it's easy for CloudStack to
>>>>>>>> calculate allocate and use a /64 and determine the IPv6 address
>>>>>>>> of VR
>>>>>> nics
>>>>>>>> and guest VM nics
>>>>>>>>> * A user create an isolated network/VPC with an offering
>>>>>> that is
>>>>>>>> ipv6 enabled
>>>>>>>>> * A user can manage firewall for the IPv6 address/guest
>>>> nics;
>>>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>>>> * A users can run workloads in the guest VMs that listen
>>>> on
>>>>>>>> publically routable ipv6 addresses
>>>>>>>>> * Usage/billing etc continue to work, no change needed
>>>>>>>>>
>>>>>>>>> Network layout:
>>>>>>>>>
>>>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
>>>>>>>>> VLAN] ->
>>>>>>>> [guest VMs/nics]
>>>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>>>
>>>>>>>>> Thoughts, feedback?
>>>>>>>>
>>>>>>>> Looks doable!
>>>>>>>>
>>>>>>>> Side-note: It would be very cool if you could use parts of this
>>>>>>>> implementation to also route /48, /56, or /60 subnets to
>>>>>>>> individual VMs in Shared networks.
>>>>>>>>
>>>>>>>> Why? This allows for running Docker containers with native IPv6
>>>>>>>> inside the VM or running a (Open)VPN server inside a VM which
>>>>>>>> then assigns public IPv6 addresses to clients connected.
>>>>>>>>
>>>>>>>> Instead of routing the subnet to a VR we route the subnet to a
>>>>>>>> single instance in a shared network.
>>>>>>>>
>>>>>>>> If we could then also move these subnets between Instances
>>>>>>>> easily one can quickly migrate to a different instance while
>>>>>>>> keeping the same IPv6 subnet.
>>>>>>>>
>>>>>>>> Wido
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>>>
>>>>>>>>> * Created an isolated network and deployed a VM in my home
>>>> lab
>>>>>>>>> The VR running on KVM has following nics
>>>>>>>>> eth0 - guest network
>>>>>>>>> eth1 - link local
>>>>>>>>> eth2 - public network
>>>>>>>>>
>>>>>>>>> * I setup a custom openwrt router on a RPi4 to serve as a
>>>>>> toy-core
>>>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker and
>>>>>>>> I got
>>>>>> a
>>>>>>>> /48 allocated. My configuration looks like:
>>>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>>>
>>>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>>>> subnet/prefix
>>>>>>>> 0:
>>>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>>>> *
>>>>>>>>> *
>>>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
>>>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
>>>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
>>>>>>>>> net.ipv6.conf.all.autoconf = 1
>>>>>>>>>
>>>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
>>>>>>>>> configured the nics:
>>>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
>>>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
>>>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
>>>>>>>>> network without cleanup to have it
>>>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>>>
>>>>>>>>> *
>>>>>>>>> Next, I created a /64 network for the isolated guest network
>>>>>>>>> on eth0
>>>>>> of
>>>>>>>> VR using radvd:
>>>>>>>>>
>>>>>>>>> # cat /etc/radvd.conf
>>>>>>>>> interface eth0
>>>>>>>>> {
>>>>>>>>> AdvSendAdvert on;
>>>>>>>>> MinRtrAdvInterval 5;
>>>>>>>>> MaxRtrAdvInterval 15;
>>>>>>>>> prefix 2001:470:ed36:1::/64
>>>>>>>>> {
>>>>>>>>> AdvOnLink on;
>>>>>>>>> AdvAutonomous on;
>>>>>>>>> };
>>>>>>>>> };
>>>>>>>>> systemctl restart radvd
>>>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
>>>>>>>>> this
>>>>>>>> ...:1::/64 network
>>>>>>>>> * Finally I added a static route in toy core-router for the
>>>> new
>>>>>> /64
>>>>>>>> IPv6 range in the isolated network
>>>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>>>> eth2> dev
>>>>>>>> <local lan nic>
>>>>>>>>> *
>>>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
>>>>>>>>> for the
>>>>>>>> new /64 network
>>>>>>>>>
>>>>>>>>> And voila all done! I create a domain AAAA record that points
>>>>>>>>> to my
>>>>>>>> guest VM IPv6 address a test webserver on
>>>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
> <http://ipv6-isolated-ntwk-demo.yadav.cloud/>
>>>>>>>>>
>>>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
>>>>>>>>> after a
>>>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
Re: IPV6 in Isolated/VPC networks
Posted by Wei ZHOU <us...@gmail.com>.
Hi Ahbishek,
Yes, the message should include the subnet.
The message should be sent when the IPv6 subnets are allocated or released,
right ? If so, there will not be multiple subnets/routes in the message.
-Wei
On Thu, 12 May 2022 at 09:56, Abhishek Kumar <Ab...@shapeblue.com>
wrote:
> Hi Wido,
>
> I do not understand what you mean by WAB address but
> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the public IP of the network (IPv6
> of the public NIC of the network VR) in the sample.
> Yes, route for fd23:313a:2f53:3cbf::/64 need to be added to this IP.
> fd23:313a:2f53:3cbf::/64 is guest IPv6 CIDR of the network here.
>
> Currently, the message on event bus does not include subnet. Should that
> be included?
> In case of VPCs, there could be multiple tiers which will need multiple
> routes to be added. Will that be an issue if we include current
> network/tier subnet in the event message?
>
> Regards,
> Abhishek
> ________________________________
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: 10 May 2022 19:01
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek Kumar
> <Ab...@shapeblue.com>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Hi,
>
> Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
> > Yes. When a public IPv6 is assigned or released, CloudStack will publish
> event with type NET.IP6ASSIGN, NET.IP6RELEASE.
> > These event notifications can be tracked. And with improvements in
> events framework, these event messages will have network uuid as entityuuid
> and Network as entitytype. Using this network can be queried using to list
> IPv6 routes that need to be added.
> >
> > Sample event message,
> >
> > {"eventDateTime":"2022-05-10 09:32:12
> +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":"14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned
> public IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID:
> 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":"bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-af19-1e00320001f3","entity":"Network","status":"Completed"}
> >
> >
> > ?Sample API call,
> >> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98
> filter=id,name,ip6routes
> > {
> > "count": 1,
> > "network": [
> > {
> > "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
> > "ip6routes": [
> > {
> > "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
> > "subnet": "fd23:313a:2f53:3cbf::/64"
> > }
>
> Looking at this JSON, does this mean that
> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
>
> And that I would need to (statically) route fd23:313a:2f53:3cbf::/64 to
> that IP?
>
> The event message does not include the subnet, that makes it a bit more
> difficult as you would then also need to do a API-call to gather that
> information.
>
> Wido
>
> P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an
> AAAA-record for IPv6!
>
> > ],
> > "name": "routing_test"
> > }
> > ]
> > }
> >
> >
> >
> >
> > ________________________________
> > From: Wido den Hollander <wi...@widodh.nl>
> > Sent: 10 May 2022 13:59
> > To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek
> Kumar <Ab...@shapeblue.com>
> > Subject: Re: IPV6 in Isolated/VPC networks
> >
> > Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
> >> Hi all,
> >>
> >> IPv6 Support in Isolated Network and VPC with Static Routing based on
> the design doc [1] has been implemented and is available in 4.17.0 RC2. I
> hope while testing 4.17.0 RC2 you will also try to test it ?
> >> Documentation for it is available at
> http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier
> (will be available in the official docs once 4.17.0 version of docs is
> built).
> >>
> >
> > Great work!
> >
> > I see only static routing is supported. But do we publish something on
> > the message bus once a new VR/VPC is created?
> >
> > This way you could pick up these messages and have the network create a
> > (static) route based on those.
> >
> > ExaBGP for example could be used to inject such routes.
> >
> > Wido
> >
> >> [1]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> >>
> >> Regards,
> >> Abhishek
> >>
> >> ________________________________
> >> From: Rohit Yadav <ro...@shapeblue.com>
> >> Sent: 13 September 2021 14:30
> >> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >> Subject: Re: IPV6 in Isolated/VPC networks
> >>
> >> Thanks Alex, Wei. I've updated the docs here:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> >>
> >> I'll leave the thread open for futher discussion/ideas/feedback. I
> think we've completed the phase1 design doc including all feedback comments
> for adding IPv6 support in CloudStack and some initial poc/work can be
> started. My colleagues and I will keep everyone posted on this thread
> and/or on a Github PR as and when we're able to start our work on the same
> (after 4.16, potentially towards 4.17).
> >>
> >>
> >> Regards.
> >>
> >> ________________________________
> >> From: Wei ZHOU <us...@gmail.com>
> >> Sent: Friday, September 10, 2021 20:22
> >> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >> Subject: Re: IPV6 in Isolated/VPC networks
> >>
> >> Agree with Alex.
> >> We only need to know how many /64 are allocated. We do not care how many
> >> ipv6 addresses are used by VMs.
> >>
> >> -Wei
> >>
> >> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <
> Alex.Mattioli@shapeblue.com>
> >> wrote:
> >>
> >>> Hi Rohit,
> >>>
> >>> I'd go for option 2, don't see a point tracking anything smaller than a
> >>> /64 tbh.
> >>>
> >>> Cheers
> >>> Alex
> >>>
> >>>
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Rohit Yadav <ro...@shapeblue.com>
> >>> Sent: 09 September 2021 12:44
> >>> To: dev@cloudstack.apache.org
> >>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>
> >>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
> >>> agreements:
> >>>
> >>> * Allocate /64 for both isolated network and VPC tiers, no large
> >>> allocation of prefixes to VPC (cons: more static routing rules for
> upstream
> >>> router/admins)
> >>> * All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone
> has a
> >>> dedicated /64 prefix/block for systemvms
> >>>
> >>> The only outstanding question now is:
> >>>
> >>> * How do we manage IPv6 usage? Can anyone advise how we do IPv6
> usage
> >>> for shared network (design, implementation and use-cases?)
> >>> Option1: We don't do it, all user VMs nics have ipv4 address which
> whose
> >>> usage we don't track. For public VR/nics/networks, we can simply add
> the
> >>> IPv6 details for a related IPv4 address.
> >>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
> >>> tracking/management and usage for all VMs and systemvms nic (this means
> >>> account/domain level limits and new billing/records)
> >>> Option3: other thoughts?
> >>>
> >>>
> >>> Regards.
> >>>
> >>> ________________________________
> >>> From: Alex Mattioli <Al...@shapeblue.com>
> >>> Sent: Wednesday, September 8, 2021 23:24
> >>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >>> Subject: RE: IPV6 in Isolated/VPC networks
> >>>
> >>> Hi Rohit, Kristaps,
> >>>
> >>> I'd say option 1 as well, it does create a bit more overhead with
> static
> >>> routes but if that's automated for a VPC it can also be easily
> automated
> >>> for several tiers of a VPC. We also don't constrain the amount of
> tiers in
> >>> a VPC.
> >>> It has the added advantage to be closer to the desired behaviour with
> >>> dynamic routing in the future, where a VPC VR can announce several
> subnets
> >>> upstream.
> >>>
> >>> Cheers
> >>> Alex
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Rohit Yadav <ro...@shapeblue.com>
> >>> Sent: 08 September 2021 19:04
> >>> To: dev@cloudstack.apache.org
> >>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>
> >>> Hi Kristaps,
> >>>
> >>> Thanks for sharing, I suppose that means individual tiers should be
> >>> allocated /64 instead of larger ipv6 blocks to the whole VPC which
> could
> >>> cause wastage.
> >>>
> >>> Any objection from anybody?
> >>>
> >>> Regards.
> >>> ________________________________
> >>> From: Kristaps Cudars <kr...@gmail.com>
> >>> Sent: Wednesday, September 8, 2021 9:24:01 PM
> >>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>
> >>> Hello,
> >>>
> >>> I asked networking team to comment on "How should the IPv6
> >>> block/allocation work in VPCs?"
> >>> Option1: They haven't seen lately devices with limits on how many
> static
> >>> routes can be created.
> >>> Option2: With /60 and /62 assignments and big quantity of routers IPv6
> >>> assignment from RIPE NNC can be drained fast.
> >>>
> >>> /48 contains 64000 /64
> >>> /60 contains 16 /64
> >>> 64000 / 16 = 4000 routers
> >>>
> >>>
> >>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
> >>>> All,
> >>>>
> >>>> After another iteration with Alex, I've updated the design doc. Kindly
> >>> review:
> >>>>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> >>>> +Isolated+Network+and+VPC+with+Static+Routing
> >>>>
> >>>>
> >>>> ... and advise on some outstanding questions:
> >>>>
> >>>> * How should the IPv6 block/allocation work in VPCs?
> >>>> Option1: Should this be simply /64 allocation on any new tier, the
> >>>> cons of this option is one static route/rule per VPC tier. (many
> >>>> upstream routers may have limit on no. of static routes?)
> >>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
> >>> (4 tiers) for the VPC, this can be filtered based on the
> vpc.max.networks
> >>> global setting (3 is default). The pros of this option are less no. of
> >>> static route/rule and easy programming, but potential wastage of
> multiple
> >>> /64 prefix blocks for unused/uncreated tiers.
> >>>> * How do we manage IPv6 usage? Can anyone advise how we do IPv6
> >>> usage for shared network (design, implementation and use-cases?)
> >>>> Option1: We don't do it, all user VMs nics have ipv4 address which
> whose
> >>> usage we don't track. For public VR/nics/networks, we can simply add
> the
> >>> IPv6 details for a related IPv4 address.
> >>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
> >>> tracking/management and usage for all VMs and systemvms nic (this means
> >>> account/domain level limits and new billing/records)
> >>>> * Enable IPv6 on systemvms (specifically SSVM and CPVM) by
> default
> >>> if zone has a IPv6 address block allocated/assigned for use for
> systemvms
> >>> (this was mainly thought for VRs, but why no ssvm and cpvms too - any
> cons
> >>> of this?)
> >>>> *
> >>>>
> >>>> Regards.
> >>>>
> >>>> ________________________________
> >>>> From: Rohit Yadav <ro...@shapeblue.com>
> >>>> Sent: Thursday, August 19, 2021 15:45
> >>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >>>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I've taken feedback from this thread and wrote this design doc:
> >>>>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> >>>> +Isolated+Network+and+VPC+with+Static+Routing
> >>>>
> >>>> Kindly review and advise if I missed anything or anything that needs
> to
> >>> be changed/updated. You may comment on the wiki directly too.
> >>>>
> >>>> Kindly suggest your views on the following (also in the design doc
> >>> above):
> >>>>
> >>>> Outstanding Questions:
> >>>>
> >>>> * Should admin or user be able to specify how VPC super CIDRs
> are
> >>> created/needed; for example a user can ask for VPC with /60 super
> CIDR? Or
> >>> should CloudStack automatically find/allocate a /64 for a new VPC tier
> from
> >>> the root-admin configured /64-/48 block?
> >>>> * Should we explore FRR and iBGP or other strategies to do
> dynamic
> >>> routing which may not require advance/complex configuration in the VR
> or
> >>> for the users/admin?
> >>>> * With SLAAC and no dhcpv6, is there a way to support secondary
> IPv6
> >>> addresses (or floating IPv6 addresses for VR/public traffic) for guest
> VM's
> >>> nics?
> >>>> * Any thoughts on UI/UX for firewall/routing management?
> >>>> * Any other feature/support for isolated network or VPC feature
> that
> >>> must be explored or supported such as PF, VPN, LB, vpc static routes,
> vpc
> >>> gateway etc.
> >>>> * For usage - should we have any consideration, or should we
> assume
> >>> that IPv4 and IPv6 address will go together for every nic; so IPv6
> usage
> >>> for a nic is in tandem with Ipv4 address for a nic, i.e. no
> explicit/new
> >>> biling/usage needed?
> >>>> * For smoketests, local dev-test should we explore ULA? Unique
> Local
> >>> Address - in the range fc00::/7. Typically only within the 'local' half
> >>> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
> >>> This prefix can be randomly generated at first install by CloudStack
> in a
> >>> zone using zoneid etc?
> >>>> * Should we expand support for VR diagnostic feature to include
> >>> support for ipv6, traceroute6?
> >>>> * Discuss - expand support/ability to allocate a /60, or /56 etc
> >>> prefix to an individual VM in shared network (Wido's suggestion)
> >>>>
> >>>>
> >>>> Regards.
> >>>>
> >>>> ________________________________
> >>>> From: Wei ZHOU <us...@gmail.com>
> >>>> Sent: Tuesday, August 17, 2021 21:16
> >>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >>>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>>
> >>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
> >>>>
> >>>> I am fine with not having stateful dhcpv6 and privacy
> >>>> extension/temporary address in phase 1. If community decides not to do
> >>>> eventually , it is also ok to me.
> >>>>
> >>>> We could explore how to better use secondary ipv6 addresses as Wido
> >>>> advised. It would be great if anyone share some user experience.
> >>>>
> >>>> -Wei
> >>>>
> >>>>
> >>>> On Tuesday, 17 August 2021, Kristaps Cudars
> >>>> <kr...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi Wei,
> >>>>>
> >>>>> Published this month's RFC 9099 and explains in different
> >>>>> words/perspective what has been written by Alex, Rohit and Wido.
> >>>>> https://www.rfc-editor.org/rfc/rfc9099.html
> >>>>>
> >>>>>
> >>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
> >>>>>> Hi Wido,
> >>>>>>
> >>>>>> (cc to Rohit and Alex)
> >>>>>>
> >>>>>> It is a good suggestion to use FRR for ipv6. The configuration is
> >>>>>> quite simple and the VMs can get SLAAC, routes, etc.
> >>>>>>
> >>>>>> Privacy extension looks not the same as what you mentioned. see
> >>>>>> https://datatracker.ietf.org/doc/html/rfc4941
> >>>>>>
> >>>>>> You are right. To use static routing, the admins need to configure
> >>>>>> the routes in the upstream router, and add some ipv6 ranges (eg
> >>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
> >>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
> >>>>>> pick up an IPv6 range
> >>>>> and
> >>>>>> assign it to an isolated network or vpc. @Rohit, correct me if I'm
> >>> wrong.
> >>>>>>
> >>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
> >>>>>> router/VR, router/dns etc via RA messages) will be the only option
> >>>>>> for now (related
> >>>>> to
> >>>>>> your pr https://github.com/apache/cloudstack/pull/3077) . Would it
> >>>>>> be
> >>>>> good
> >>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
> >>>>>> as an option in cloudstack ? The advantages are
> >>>>>> (1) support other ipv6 cidr sizes than /64.
> >>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
> >>>>>> addresses can be changed
> >>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
> >>>>>> The problem is, stateful dhcpv6 does not support
> >>>>>> routers,nameservers,
> >>>>> etc.
> >>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
> >>>>>>
> >>>>>> -Wei
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
> >
>
>
> > On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> See my inline responses:
> >>>>>>>
> >>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
> >>>>>>>> discussing
> >>>>> them
> >>>>>>> with Alex and came up with a PoC/design which can be implemented
> >>>>>>> in the following phases:
> >>>>>>>>
> >>>>>>>> * Phase1: implement ipv6 support in isolated networks and
> >>> VPC
> >>>>> with
> >>>>>>> static routing
> >>>>>>>> * Phase2: discuss and implement support for dynamic routing
> >>>>> (TBD)
> >>>>>>>>
> >>>>>>>> For Phase1 here's the high-level proposal:
> >>>>>>>>
> >>>>>>>> * IPv6 address management:
> >>>>>>>> * At the zone level root-admin specifies a /64 public
> >>> range
> >>>>> that
> >>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6 range
> >>>>>>> for
> >>>>> guest
> >>>>>>> networks (to be used by isolated networks and VPC tiers)
> >>>>>>>> * On creation of any IPv6 enabled isolated network or
> VPC
> >>>>> tier,
> >>>>>>> from the /48 or /56 block a /64 network is allocated/used
> >>>>>>>> * We assume SLAAC and autoconfiguration, no DHCPv6 in
> the
> >>>>> zone
> >>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
> >>>>>>> of
> >>>>> slaac be
> >>>>>>> explored?)
> >>>>>>>
> >>>>>>> Privacy Extensions are only a concern for client devices which
> >>>>>>> roam between different IPv6 networks.
> >>>>>>>
> >>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
> >>>>>>> based) and switches network then only the prefix (/64) will change.
> >>>>>>>
> >>>>>>> This way a network like Google, Facebook, etc could track your
> >>>>>>> device moving from network to network if they only look at the
> >>>>>>> last 64-bits of the IPv6 address.
> >>>>>>>
> >>>>>>> For servers this is not a problem as you already know in which
> >>>>>>> network they are.
> >>>>>>>
> >>>>>>>> * Network offerings: root-admin can create new network
> >>> offerings
> >>>>>>> (with VPC too) that specifies a network stack option:
> >>>>>>>> * ipv4 only (default, for backward compatibility all
> >>>>>>> networks/offerings post-upgrade migrate to this option)
> >>>>>>>> * ipv4-and-ipv6
> >>>>>>>> * ipv6-only (this can be phase 1.b)
> >>>>>>>> * A new routing option: static (phase1), dynamic
> (phase2,
> >>>>> with
> >>>>>>> multiple sub-options such as ospf/bgp etc...)
> >>>>>>>
> >>>>>>> This means that the network admin will need to statically route
> >>>>>>> the
> >>>>> IPv6
> >>>>>>> subnet to the VR's outside IPv6 address, for example, on a JunOS
> >>>>> router:
> >>>>>>>
> >>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
> >>>>>>> next-hop
> >>>>>>> 2001:db8:100::50
> >>>>>>>
> >>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
> >>>>>>> the outside (/64) network. In reality this will probably be a
> >>>>>>> longer address, but this is for just the example.
> >>>>>>>
> >>>>>>>> * VR changes:
> >>>>>>>> * VR gets its guest and public nics set to inet6 auto
> >>>>>>>> * For each /64 allocated to guest network and VPC tiers,
> >>>>> radvd
> >>>>>>> is configured to do RA
> >>>>>>>
> >>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
> >>>>>>> might already want to look into FRRouting. FRR can also
> >>>>>>> advertise RAs while not doing any routing.
> >>>>>>>
> >>>>>>> interface ens4
> >>>>>>> no ipv6 nd suppress-ra
> >>>>>>> ipv6 nd prefix 2001:db8:500::/64
> >>>>>>> ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> >>>>>>>
> >>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
> >>>>>>>
> >>>>>>>> * Firewall: a new ipv6 zone/chain is created for ipv6
> >>> where
> >>>>> ipv6
> >>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
> >>>>>>> between
> >>>>> VPC
> >>>>>>> tiers are managed/implemented by ipv6 firewall on VR
> >>>>>>>
> >>>>>>> Please take a look at the existing security_group.py script
> >>>>>>> which implements RFC4890
> >>>>>>>
> >>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
> >>>>>>>
> >>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should always
> >>>>>>> be allowed.
> >>>>>>>
> >>>>>>>> * It is assumed that static routes are created on the
> >>>>> core/main
> >>>>>>> router by the admin or automated using some scripts/tools; for
> >>>>>>> this CloudStack will announce events with details of /64
> >>>>>>> networks and VR's public IPv6 address that can be consumed by a
> >>>>>>> rabbitmq/message bus
> >>>>> client
> >>>>>>> (for example), or a custom cron job or script as part of
> >>> orchestration.
> >>>>>>> (this wouldn't be necessary for dynamic routing bgp with
> >>>>>>> phase2)\\
> >>>>>>>
> >>>>>>> You would only need to announce the /48 or /56 allocated to the
> >>>>>>> VR, that's all. You don't need to inform the upstream router
> >>>>>>> about the /64 subnets created within that larger subnet.
> >>>>>>>
> >>>>>>>> * Guest Networking: With SLAAC, it's easy for CloudStack to
> >>>>>>> calculate allocate and use a /64 and determine the IPv6 address
> >>>>>>> of VR
> >>>>> nics
> >>>>>>> and guest VM nics
> >>>>>>>> * A user create an isolated network/VPC with an offering
> >>>>> that is
> >>>>>>> ipv6 enabled
> >>>>>>>> * A user can manage firewall for the IPv6 address/guest
> >>> nics;
> >>>>>>> there'll be no port forward and LB feature though for IPv6
> >>>>>>>> * A users can run workloads in the guest VMs that listen
> >>> on
> >>>>>>> publically routable ipv6 addresses
> >>>>>>>> * Usage/billing etc continue to work, no change needed
> >>>>>>>>
> >>>>>>>> Network layout:
> >>>>>>>>
> >>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
> >>>>>>>> VLAN] ->
> >>>>>>> [guest VMs/nics]
> >>>>>>>> *core/ISP router needs static routes to be added (manually or
> >>>>>>> automated), assumes a /48 or /56 configured for the zone
> >>>>>>>>
> >>>>>>>> Thoughts, feedback?
> >>>>>>>
> >>>>>>> Looks doable!
> >>>>>>>
> >>>>>>> Side-note: It would be very cool if you could use parts of this
> >>>>>>> implementation to also route /48, /56, or /60 subnets to
> >>>>>>> individual VMs in Shared networks.
> >>>>>>>
> >>>>>>> Why? This allows for running Docker containers with native IPv6
> >>>>>>> inside the VM or running a (Open)VPN server inside a VM which
> >>>>>>> then assigns public IPv6 addresses to clients connected.
> >>>>>>>
> >>>>>>> Instead of routing the subnet to a VR we route the subnet to a
> >>>>>>> single instance in a shared network.
> >>>>>>>
> >>>>>>> If we could then also move these subnets between Instances
> >>>>>>> easily one can quickly migrate to a different instance while
> >>>>>>> keeping the same IPv6 subnet.
> >>>>>>>
> >>>>>>> Wido
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
> >>>>>>>>
> >>>>>>>> * Created an isolated network and deployed a VM in my home
> >>> lab
> >>>>>>>> The VR running on KVM has following nics
> >>>>>>>> eth0 - guest network
> >>>>>>>> eth1 - link local
> >>>>>>>> eth2 - public network
> >>>>>>>>
> >>>>>>>> * I setup a custom openwrt router on a RPi4 to serve as a
> >>>>> toy-core
> >>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker and
> >>>>>>> I got
> >>>>> a
> >>>>>>> /48 allocated. My configuration looks like:
> >>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> >>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
> >>>>>>>>
> >>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
> >>>>> subnet/prefix
> >>>>>>> 0:
> >>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
> >>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
> >>>>>>>> *
> >>>>>>>> *
> >>>>>>>> In the isolated VR, I enabled ipv6 as:
> >>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
> >>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
> >>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
> >>>>>>>> net.ipv6.conf.all.autoconf = 1
> >>>>>>>>
> >>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
> >>>>>>>> configured the nics:
> >>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
> >>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
> >>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
> >>>>>>>> network without cleanup to have it
> >>>>>>> reconfigure IPv4 nics, firewall, NAT etc
> >>>>>>>>
> >>>>>>>> *
> >>>>>>>> Next, I created a /64 network for the isolated guest network
> >>>>>>>> on eth0
> >>>>> of
> >>>>>>> VR using radvd:
> >>>>>>>>
> >>>>>>>> # cat /etc/radvd.conf
> >>>>>>>> interface eth0
> >>>>>>>> {
> >>>>>>>> AdvSendAdvert on;
> >>>>>>>> MinRtrAdvInterval 5;
> >>>>>>>> MaxRtrAdvInterval 15;
> >>>>>>>> prefix 2001:470:ed36:1::/64
> >>>>>>>> {
> >>>>>>>> AdvOnLink on;
> >>>>>>>> AdvAutonomous on;
> >>>>>>>> };
> >>>>>>>> };
> >>>>>>>> systemctl restart radvd
> >>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
> >>>>>>>> this
> >>>>>>> ...:1::/64 network
> >>>>>>>> * Finally I added a static route in toy core-router for the
> >>> new
> >>>>> /64
> >>>>>>> IPv6 range in the isolated network
> >>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
> >>>>>>>> eth2> dev
> >>>>>>> <local lan nic>
> >>>>>>>> *
> >>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
> >>>>>>>> for the
> >>>>>>> new /64 network
> >>>>>>>>
> >>>>>>>> And voila all done! I create a domain AAAA record that points
> >>>>>>>> to my
> >>>>>>> guest VM IPv6 address a test webserver on
> >>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
> >>>>>>>>
> >>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
> >>>>>>>> after a
> >>>>>>> few days, sharing this solely for testing purposes)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Regards.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >
>
Re: IPV6 in Isolated/VPC networks
Posted by Abhishek Kumar <Ab...@shapeblue.com>.
Hi Wido,
I do not understand what you mean by WAB address but fd23:313a:2f53:3000:1c00:baff:fe00:4 is the public IP of the network (IPv6 of the public NIC of the network VR) in the sample.
Yes, route for fd23:313a:2f53:3cbf::/64 need to be added to this IP. fd23:313a:2f53:3cbf::/64 is guest IPv6 CIDR of the network here.
Currently, the message on event bus does not include subnet. Should that be included?
In case of VPCs, there could be multiple tiers which will need multiple routes to be added. Will that be an issue if we include current network/tier subnet in the event message?
Regards,
Abhishek
________________________________
From: Wido den Hollander <wi...@widodh.nl>
Sent: 10 May 2022 19:01
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek Kumar <Ab...@shapeblue.com>
Subject: Re: IPV6 in Isolated/VPC networks
Hi,
Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
> Yes. When a public IPv6 is assigned or released, CloudStack will publish event with type NET.IP6ASSIGN, NET.IP6RELEASE.
> These event notifications can be tracked. And with improvements in events framework, these event messages will have network uuid as entityuuid and Network as entitytype. Using this network can be queried using to list IPv6 routes that need to be added.
>
> Sample event message,
>
> {"eventDateTime":"2022-05-10 09:32:12 +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":"14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned public IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID: 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":"bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-af19-1e00320001f3","entity":"Network","status":"Completed"}
>
>
> ?Sample API call,
>> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98 filter=id,name,ip6routes
> {
> "count": 1,
> "network": [
> {
> "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
> "ip6routes": [
> {
> "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
> "subnet": "fd23:313a:2f53:3cbf::/64"
> }
Looking at this JSON, does this mean that
fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
And that I would need to (statically) route fd23:313a:2f53:3cbf::/64 to
that IP?
The event message does not include the subnet, that makes it a bit more
difficult as you would then also need to do a API-call to gather that
information.
Wido
P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an
AAAA-record for IPv6!
> ],
> "name": "routing_test"
> }
> ]
> }
>
>
>
>
> ________________________________
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: 10 May 2022 13:59
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek Kumar <Ab...@shapeblue.com>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
>> Hi all,
>>
>> IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
>> Documentation for it is available at http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier (will be available in the official docs once 4.17.0 version of docs is built).
>>
>
> Great work!
>
> I see only static routing is supported. But do we publish something on
> the message bus once a new VR/VPC is created?
>
> This way you could pick up these messages and have the network create a
> (static) route based on those.
>
> ExaBGP for example could be used to inject such routes.
>
> Wido
>
>> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
>>
>> Regards,
>> Abhishek
>>
>> ________________________________
>> From: Rohit Yadav <ro...@shapeblue.com>
>> Sent: 13 September 2021 14:30
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Thanks Alex, Wei. I've updated the docs here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
>>
>> I'll leave the thread open for futher discussion/ideas/feedback. I think we've completed the phase1 design doc including all feedback comments for adding IPv6 support in CloudStack and some initial poc/work can be started. My colleagues and I will keep everyone posted on this thread and/or on a Github PR as and when we're able to start our work on the same (after 4.16, potentially towards 4.17).
>>
>>
>> Regards.
>>
>> ________________________________
>> From: Wei ZHOU <us...@gmail.com>
>> Sent: Friday, September 10, 2021 20:22
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Agree with Alex.
>> We only need to know how many /64 are allocated. We do not care how many
>> ipv6 addresses are used by VMs.
>>
>> -Wei
>>
>> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <Al...@shapeblue.com>
>> wrote:
>>
>>> Hi Rohit,
>>>
>>> I'd go for option 2, don't see a point tracking anything smaller than a
>>> /64 tbh.
>>>
>>> Cheers
>>> Alex
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Rohit Yadav <ro...@shapeblue.com>
>>> Sent: 09 September 2021 12:44
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>>> agreements:
>>>
>>> * Allocate /64 for both isolated network and VPC tiers, no large
>>> allocation of prefixes to VPC (cons: more static routing rules for upstream
>>> router/admins)
>>> * All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a
>>> dedicated /64 prefix/block for systemvms
>>>
>>> The only outstanding question now is:
>>>
>>> * How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage
>>> for shared network (design, implementation and use-cases?)
>>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>>> usage we don't track. For public VR/nics/networks, we can simply add the
>>> IPv6 details for a related IPv4 address.
>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>>> tracking/management and usage for all VMs and systemvms nic (this means
>>> account/domain level limits and new billing/records)
>>> Option3: other thoughts?
>>>
>>>
>>> Regards.
>>>
>>> ________________________________
>>> From: Alex Mattioli <Al...@shapeblue.com>
>>> Sent: Wednesday, September 8, 2021 23:24
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: RE: IPV6 in Isolated/VPC networks
>>>
>>> Hi Rohit, Kristaps,
>>>
>>> I'd say option 1 as well, it does create a bit more overhead with static
>>> routes but if that's automated for a VPC it can also be easily automated
>>> for several tiers of a VPC. We also don't constrain the amount of tiers in
>>> a VPC.
>>> It has the added advantage to be closer to the desired behaviour with
>>> dynamic routing in the future, where a VPC VR can announce several subnets
>>> upstream.
>>>
>>> Cheers
>>> Alex
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Rohit Yadav <ro...@shapeblue.com>
>>> Sent: 08 September 2021 19:04
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Hi Kristaps,
>>>
>>> Thanks for sharing, I suppose that means individual tiers should be
>>> allocated /64 instead of larger ipv6 blocks to the whole VPC which could
>>> cause wastage.
>>>
>>> Any objection from anybody?
>>>
>>> Regards.
>>> ________________________________
>>> From: Kristaps Cudars <kr...@gmail.com>
>>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Hello,
>>>
>>> I asked networking team to comment on "How should the IPv6
>>> block/allocation work in VPCs?"
>>> Option1: They haven't seen lately devices with limits on how many static
>>> routes can be created.
>>> Option2: With /60 and /62 assignments and big quantity of routers IPv6
>>> assignment from RIPE NNC can be drained fast.
>>>
>>> /48 contains 64000 /64
>>> /60 contains 16 /64
>>> 64000 / 16 = 4000 routers
>>>
>>>
>>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
>>>> All,
>>>>
>>>> After another iteration with Alex, I've updated the design doc. Kindly
>>> review:
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>
>>>>
>>>> ... and advise on some outstanding questions:
>>>>
>>>> * How should the IPv6 block/allocation work in VPCs?
>>>> Option1: Should this be simply /64 allocation on any new tier, the
>>>> cons of this option is one static route/rule per VPC tier. (many
>>>> upstream routers may have limit on no. of static routes?)
>>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
>>> (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks
>>> global setting (3 is default). The pros of this option are less no. of
>>> static route/rule and easy programming, but potential wastage of multiple
>>> /64 prefix blocks for unused/uncreated tiers.
>>>> * How do we manage IPv6 usage? Can anyone advise how we do IPv6
>>> usage for shared network (design, implementation and use-cases?)
>>>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>>> usage we don't track. For public VR/nics/networks, we can simply add the
>>> IPv6 details for a related IPv4 address.
>>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>>> tracking/management and usage for all VMs and systemvms nic (this means
>>> account/domain level limits and new billing/records)
>>>> * Enable IPv6 on systemvms (specifically SSVM and CPVM) by default
>>> if zone has a IPv6 address block allocated/assigned for use for systemvms
>>> (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons
>>> of this?)
>>>> *
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>> Sent: Thursday, August 19, 2021 15:45
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Hi all,
>>>>
>>>> I've taken feedback from this thread and wrote this design doc:
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>
>>>> Kindly review and advise if I missed anything or anything that needs to
>>> be changed/updated. You may comment on the wiki directly too.
>>>>
>>>> Kindly suggest your views on the following (also in the design doc
>>> above):
>>>>
>>>> Outstanding Questions:
>>>>
>>>> * Should admin or user be able to specify how VPC super CIDRs are
>>> created/needed; for example a user can ask for VPC with /60 super CIDR? Or
>>> should CloudStack automatically find/allocate a /64 for a new VPC tier from
>>> the root-admin configured /64-/48 block?
>>>> * Should we explore FRR and iBGP or other strategies to do dynamic
>>> routing which may not require advance/complex configuration in the VR or
>>> for the users/admin?
>>>> * With SLAAC and no dhcpv6, is there a way to support secondary IPv6
>>> addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's
>>> nics?
>>>> * Any thoughts on UI/UX for firewall/routing management?
>>>> * Any other feature/support for isolated network or VPC feature that
>>> must be explored or supported such as PF, VPN, LB, vpc static routes, vpc
>>> gateway etc.
>>>> * For usage - should we have any consideration, or should we assume
>>> that IPv4 and IPv6 address will go together for every nic; so IPv6 usage
>>> for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new
>>> biling/usage needed?
>>>> * For smoketests, local dev-test should we explore ULA? Unique Local
>>> Address - in the range fc00::/7. Typically only within the 'local' half
>>> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
>>> This prefix can be randomly generated at first install by CloudStack in a
>>> zone using zoneid etc?
>>>> * Should we expand support for VR diagnostic feature to include
>>> support for ipv6, traceroute6?
>>>> * Discuss - expand support/ability to allocate a /60, or /56 etc
>>> prefix to an individual VM in shared network (Wido's suggestion)
>>>>
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Wei ZHOU <us...@gmail.com>
>>>> Sent: Tuesday, August 17, 2021 21:16
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>>
>>>> I am fine with not having stateful dhcpv6 and privacy
>>>> extension/temporary address in phase 1. If community decides not to do
>>>> eventually , it is also ok to me.
>>>>
>>>> We could explore how to better use secondary ipv6 addresses as Wido
>>>> advised. It would be great if anyone share some user experience.
>>>>
>>>> -Wei
>>>>
>>>>
>>>> On Tuesday, 17 August 2021, Kristaps Cudars
>>>> <kr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Wei,
>>>>>
>>>>> Published this month's RFC 9099 and explains in different
>>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>>> https://www.rfc-editor.org/rfc/rfc9099.html
>>>>>
>>>>>
>>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
>>>>>> Hi Wido,
>>>>>>
>>>>>> (cc to Rohit and Alex)
>>>>>>
>>>>>> It is a good suggestion to use FRR for ipv6. The configuration is
>>>>>> quite simple and the VMs can get SLAAC, routes, etc.
>>>>>>
>>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>>> https://datatracker.ietf.org/doc/html/rfc4941
>>>>>>
>>>>>> You are right. To use static routing, the admins need to configure
>>>>>> the routes in the upstream router, and add some ipv6 ranges (eg
>>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
>>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
>>>>>> pick up an IPv6 range
>>>>> and
>>>>>> assign it to an isolated network or vpc. @Rohit, correct me if I'm
>>> wrong.
>>>>>>
>>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
>>>>>> router/VR, router/dns etc via RA messages) will be the only option
>>>>>> for now (related
>>>>> to
>>>>>> your pr https://github.com/apache/cloudstack/pull/3077) . Would it
>>>>>> be
>>>>> good
>>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
>>>>>> as an option in cloudstack ? The advantages are
>>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
>>>>>> addresses can be changed
>>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>>> The problem is, stateful dhcpv6 does not support
>>>>>> routers,nameservers,
>>>>> etc.
>>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>>
>>>>>> -Wei
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>>
>
>
>
> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> See my inline responses:
>>>>>>>
>>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
>>>>>>>> discussing
>>>>> them
>>>>>>> with Alex and came up with a PoC/design which can be implemented
>>>>>>> in the following phases:
>>>>>>>>
>>>>>>>> * Phase1: implement ipv6 support in isolated networks and
>>> VPC
>>>>> with
>>>>>>> static routing
>>>>>>>> * Phase2: discuss and implement support for dynamic routing
>>>>> (TBD)
>>>>>>>>
>>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>>
>>>>>>>> * IPv6 address management:
>>>>>>>> * At the zone level root-admin specifies a /64 public
>>> range
>>>>> that
>>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6 range
>>>>>>> for
>>>>> guest
>>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>>> * On creation of any IPv6 enabled isolated network or VPC
>>>>> tier,
>>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>>> * We assume SLAAC and autoconfiguration, no DHCPv6 in the
>>>>> zone
>>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
>>>>>>> of
>>>>> slaac be
>>>>>>> explored?)
>>>>>>>
>>>>>>> Privacy Extensions are only a concern for client devices which
>>>>>>> roam between different IPv6 networks.
>>>>>>>
>>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>>
>>>>>>> This way a network like Google, Facebook, etc could track your
>>>>>>> device moving from network to network if they only look at the
>>>>>>> last 64-bits of the IPv6 address.
>>>>>>>
>>>>>>> For servers this is not a problem as you already know in which
>>>>>>> network they are.
>>>>>>>
>>>>>>>> * Network offerings: root-admin can create new network
>>> offerings
>>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>>> * ipv4 only (default, for backward compatibility all
>>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>>> * ipv4-and-ipv6
>>>>>>>> * ipv6-only (this can be phase 1.b)
>>>>>>>> * A new routing option: static (phase1), dynamic (phase2,
>>>>> with
>>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>>
>>>>>>> This means that the network admin will need to statically route
>>>>>>> the
>>>>> IPv6
>>>>>>> subnet to the VR's outside IPv6 address, for example, on a JunOS
>>>>> router:
>>>>>>>
>>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
>>>>>>> next-hop
>>>>>>> 2001:db8:100::50
>>>>>>>
>>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
>>>>>>> the outside (/64) network. In reality this will probably be a
>>>>>>> longer address, but this is for just the example.
>>>>>>>
>>>>>>>> * VR changes:
>>>>>>>> * VR gets its guest and public nics set to inet6 auto
>>>>>>>> * For each /64 allocated to guest network and VPC tiers,
>>>>> radvd
>>>>>>> is configured to do RA
>>>>>>>
>>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
>>>>>>> might already want to look into FRRouting. FRR can also
>>>>>>> advertise RAs while not doing any routing.
>>>>>>>
>>>>>>> interface ens4
>>>>>>> no ipv6 nd suppress-ra
>>>>>>> ipv6 nd prefix 2001:db8:500::/64
>>>>>>> ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>>
>>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
>>>>>>>
>>>>>>>> * Firewall: a new ipv6 zone/chain is created for ipv6
>>> where
>>>>> ipv6
>>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
>>>>>>> between
>>>>> VPC
>>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>>
>>>>>>> Please take a look at the existing security_group.py script
>>>>>>> which implements RFC4890
>>>>>>>
>>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
>>>>>>>
>>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should always
>>>>>>> be allowed.
>>>>>>>
>>>>>>>> * It is assumed that static routes are created on the
>>>>> core/main
>>>>>>> router by the admin or automated using some scripts/tools; for
>>>>>>> this CloudStack will announce events with details of /64
>>>>>>> networks and VR's public IPv6 address that can be consumed by a
>>>>>>> rabbitmq/message bus
>>>>> client
>>>>>>> (for example), or a custom cron job or script as part of
>>> orchestration.
>>>>>>> (this wouldn't be necessary for dynamic routing bgp with
>>>>>>> phase2)\\
>>>>>>>
>>>>>>> You would only need to announce the /48 or /56 allocated to the
>>>>>>> VR, that's all. You don't need to inform the upstream router
>>>>>>> about the /64 subnets created within that larger subnet.
>>>>>>>
>>>>>>>> * Guest Networking: With SLAAC, it's easy for CloudStack to
>>>>>>> calculate allocate and use a /64 and determine the IPv6 address
>>>>>>> of VR
>>>>> nics
>>>>>>> and guest VM nics
>>>>>>>> * A user create an isolated network/VPC with an offering
>>>>> that is
>>>>>>> ipv6 enabled
>>>>>>>> * A user can manage firewall for the IPv6 address/guest
>>> nics;
>>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>>> * A users can run workloads in the guest VMs that listen
>>> on
>>>>>>> publically routable ipv6 addresses
>>>>>>>> * Usage/billing etc continue to work, no change needed
>>>>>>>>
>>>>>>>> Network layout:
>>>>>>>>
>>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
>>>>>>>> VLAN] ->
>>>>>>> [guest VMs/nics]
>>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>>
>>>>>>>> Thoughts, feedback?
>>>>>>>
>>>>>>> Looks doable!
>>>>>>>
>>>>>>> Side-note: It would be very cool if you could use parts of this
>>>>>>> implementation to also route /48, /56, or /60 subnets to
>>>>>>> individual VMs in Shared networks.
>>>>>>>
>>>>>>> Why? This allows for running Docker containers with native IPv6
>>>>>>> inside the VM or running a (Open)VPN server inside a VM which
>>>>>>> then assigns public IPv6 addresses to clients connected.
>>>>>>>
>>>>>>> Instead of routing the subnet to a VR we route the subnet to a
>>>>>>> single instance in a shared network.
>>>>>>>
>>>>>>> If we could then also move these subnets between Instances
>>>>>>> easily one can quickly migrate to a different instance while
>>>>>>> keeping the same IPv6 subnet.
>>>>>>>
>>>>>>> Wido
>>>>>>>
>>>>>>>>
>>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>>
>>>>>>>> * Created an isolated network and deployed a VM in my home
>>> lab
>>>>>>>> The VR running on KVM has following nics
>>>>>>>> eth0 - guest network
>>>>>>>> eth1 - link local
>>>>>>>> eth2 - public network
>>>>>>>>
>>>>>>>> * I setup a custom openwrt router on a RPi4 to serve as a
>>>>> toy-core
>>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker and
>>>>>>> I got
>>>>> a
>>>>>>> /48 allocated. My configuration looks like:
>>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>>
>>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>>> subnet/prefix
>>>>>>> 0:
>>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>>> *
>>>>>>>> *
>>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
>>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
>>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
>>>>>>>> net.ipv6.conf.all.autoconf = 1
>>>>>>>>
>>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
>>>>>>>> configured the nics:
>>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
>>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
>>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
>>>>>>>> network without cleanup to have it
>>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>>
>>>>>>>> *
>>>>>>>> Next, I created a /64 network for the isolated guest network
>>>>>>>> on eth0
>>>>> of
>>>>>>> VR using radvd:
>>>>>>>>
>>>>>>>> # cat /etc/radvd.conf
>>>>>>>> interface eth0
>>>>>>>> {
>>>>>>>> AdvSendAdvert on;
>>>>>>>> MinRtrAdvInterval 5;
>>>>>>>> MaxRtrAdvInterval 15;
>>>>>>>> prefix 2001:470:ed36:1::/64
>>>>>>>> {
>>>>>>>> AdvOnLink on;
>>>>>>>> AdvAutonomous on;
>>>>>>>> };
>>>>>>>> };
>>>>>>>> systemctl restart radvd
>>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
>>>>>>>> this
>>>>>>> ...:1::/64 network
>>>>>>>> * Finally I added a static route in toy core-router for the
>>> new
>>>>> /64
>>>>>>> IPv6 range in the isolated network
>>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>>> eth2> dev
>>>>>>> <local lan nic>
>>>>>>>> *
>>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
>>>>>>>> for the
>>>>>>> new /64 network
>>>>>>>>
>>>>>>>> And voila all done! I create a domain AAAA record that points
>>>>>>>> to my
>>>>>>> guest VM IPv6 address a test webserver on
>>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>>>>>>>>
>>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
>>>>>>>> after a
>>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>
Re: IPV6 in Isolated/VPC networks
Posted by Wei ZHOU <us...@gmail.com>.
Hi Wido,
In Abhishek's example, `fd23:313a:2f53:3000:1c00:baff:fe00:4` (gateway) is
used for communication between CloudStack VR and the upstream router.
In current implementation, it uses the same VLAN as the public IPv4 address
of the network, it is associated with the public NIC in CloudStack VR.
it can be a public or private IPv6 address. the upstream router should have
an Ipv6 in the network (e.g. upstream router has IP
fd23:313a:2f53:3000::1/64)
the guest vms are reachable when a static route is added in upstream
router, similar as `ip -6 route add <guest subnet> via <gateway>"
-Wei
On Tue, 10 May 2022 at 15:31, Wido den Hollander <wi...@widodh.nl> wrote:
> Hi,
>
> Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
> > Yes. When a public IPv6 is assigned or released, CloudStack will publish
> event with type NET.IP6ASSIGN, NET.IP6RELEASE.
> > These event notifications can be tracked. And with improvements in
> events framework, these event messages will have network uuid as entityuuid
> and Network as entitytype. Using this network can be queried using to list
> IPv6 routes that need to be added.
> >
> > Sample event message,
> >
> > {"eventDateTime":"2022-05-10 09:32:12
> +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":"14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned
> public IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID:
> 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":"bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-af19-1e00320001f3","entity":"Network","status":"Completed"}
> >
> >
> > ?Sample API call,
> >> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98
> filter=id,name,ip6routes
> > {
> > "count": 1,
> > "network": [
> > {
> > "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
> > "ip6routes": [
> > {
> > "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
> > "subnet": "fd23:313a:2f53:3cbf::/64"
> > }
>
> Looking at this JSON, does this mean that
> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
>
> And that I would need to (statically) route fd23:313a:2f53:3cbf::/64 to
> that IP?
>
> The event message does not include the subnet, that makes it a bit more
> difficult as you would then also need to do a API-call to gather that
> information.
>
> Wido
>
> P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an
> AAAA-record for IPv6!
>
> > ],
> > "name": "routing_test"
> > }
> > ]
> > }
> >
> >
> >
> >
> > ________________________________
> > From: Wido den Hollander <wi...@widodh.nl>
> > Sent: 10 May 2022 13:59
> > To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek
> Kumar <Ab...@shapeblue.com>
> > Subject: Re: IPV6 in Isolated/VPC networks
> >
> > Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
> >> Hi all,
> >>
> >> IPv6 Support in Isolated Network and VPC with Static Routing based on
> the design doc [1] has been implemented and is available in 4.17.0 RC2. I
> hope while testing 4.17.0 RC2 you will also try to test it ?
> >> Documentation for it is available at
> http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier
> (will be available in the official docs once 4.17.0 version of docs is
> built).
> >>
> >
> > Great work!
> >
> > I see only static routing is supported. But do we publish something on
> > the message bus once a new VR/VPC is created?
> >
> > This way you could pick up these messages and have the network create a
> > (static) route based on those.
> >
> > ExaBGP for example could be used to inject such routes.
> >
> > Wido
> >
> >> [1]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> >>
> >> Regards,
> >> Abhishek
> >>
> >> ________________________________
> >> From: Rohit Yadav <ro...@shapeblue.com>
> >> Sent: 13 September 2021 14:30
> >> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >> Subject: Re: IPV6 in Isolated/VPC networks
> >>
> >> Thanks Alex, Wei. I've updated the docs here:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> >>
> >> I'll leave the thread open for futher discussion/ideas/feedback. I
> think we've completed the phase1 design doc including all feedback comments
> for adding IPv6 support in CloudStack and some initial poc/work can be
> started. My colleagues and I will keep everyone posted on this thread
> and/or on a Github PR as and when we're able to start our work on the same
> (after 4.16, potentially towards 4.17).
> >>
> >>
> >> Regards.
> >>
> >> ________________________________
> >> From: Wei ZHOU <us...@gmail.com>
> >> Sent: Friday, September 10, 2021 20:22
> >> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >> Subject: Re: IPV6 in Isolated/VPC networks
> >>
> >> Agree with Alex.
> >> We only need to know how many /64 are allocated. We do not care how many
> >> ipv6 addresses are used by VMs.
> >>
> >> -Wei
> >>
> >> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <
> Alex.Mattioli@shapeblue.com>
> >> wrote:
> >>
> >>> Hi Rohit,
> >>>
> >>> I'd go for option 2, don't see a point tracking anything smaller than a
> >>> /64 tbh.
> >>>
> >>> Cheers
> >>> Alex
> >>>
> >>>
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Rohit Yadav <ro...@shapeblue.com>
> >>> Sent: 09 September 2021 12:44
> >>> To: dev@cloudstack.apache.org
> >>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>
> >>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
> >>> agreements:
> >>>
> >>> * Allocate /64 for both isolated network and VPC tiers, no large
> >>> allocation of prefixes to VPC (cons: more static routing rules for
> upstream
> >>> router/admins)
> >>> * All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone
> has a
> >>> dedicated /64 prefix/block for systemvms
> >>>
> >>> The only outstanding question now is:
> >>>
> >>> * How do we manage IPv6 usage? Can anyone advise how we do IPv6
> usage
> >>> for shared network (design, implementation and use-cases?)
> >>> Option1: We don't do it, all user VMs nics have ipv4 address which
> whose
> >>> usage we don't track. For public VR/nics/networks, we can simply add
> the
> >>> IPv6 details for a related IPv4 address.
> >>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
> >>> tracking/management and usage for all VMs and systemvms nic (this means
> >>> account/domain level limits and new billing/records)
> >>> Option3: other thoughts?
> >>>
> >>>
> >>> Regards.
> >>>
> >>> ________________________________
> >>> From: Alex Mattioli <Al...@shapeblue.com>
> >>> Sent: Wednesday, September 8, 2021 23:24
> >>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >>> Subject: RE: IPV6 in Isolated/VPC networks
> >>>
> >>> Hi Rohit, Kristaps,
> >>>
> >>> I'd say option 1 as well, it does create a bit more overhead with
> static
> >>> routes but if that's automated for a VPC it can also be easily
> automated
> >>> for several tiers of a VPC. We also don't constrain the amount of
> tiers in
> >>> a VPC.
> >>> It has the added advantage to be closer to the desired behaviour with
> >>> dynamic routing in the future, where a VPC VR can announce several
> subnets
> >>> upstream.
> >>>
> >>> Cheers
> >>> Alex
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Rohit Yadav <ro...@shapeblue.com>
> >>> Sent: 08 September 2021 19:04
> >>> To: dev@cloudstack.apache.org
> >>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>
> >>> Hi Kristaps,
> >>>
> >>> Thanks for sharing, I suppose that means individual tiers should be
> >>> allocated /64 instead of larger ipv6 blocks to the whole VPC which
> could
> >>> cause wastage.
> >>>
> >>> Any objection from anybody?
> >>>
> >>> Regards.
> >>> ________________________________
> >>> From: Kristaps Cudars <kr...@gmail.com>
> >>> Sent: Wednesday, September 8, 2021 9:24:01 PM
> >>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>
> >>> Hello,
> >>>
> >>> I asked networking team to comment on "How should the IPv6
> >>> block/allocation work in VPCs?"
> >>> Option1: They haven't seen lately devices with limits on how many
> static
> >>> routes can be created.
> >>> Option2: With /60 and /62 assignments and big quantity of routers IPv6
> >>> assignment from RIPE NNC can be drained fast.
> >>>
> >>> /48 contains 64000 /64
> >>> /60 contains 16 /64
> >>> 64000 / 16 = 4000 routers
> >>>
> >>>
> >>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
> >>>> All,
> >>>>
> >>>> After another iteration with Alex, I've updated the design doc. Kindly
> >>> review:
> >>>>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> >>>> +Isolated+Network+and+VPC+with+Static+Routing
> >>>>
> >>>>
> >>>> ... and advise on some outstanding questions:
> >>>>
> >>>> * How should the IPv6 block/allocation work in VPCs?
> >>>> Option1: Should this be simply /64 allocation on any new tier, the
> >>>> cons of this option is one static route/rule per VPC tier. (many
> >>>> upstream routers may have limit on no. of static routes?)
> >>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
> >>> (4 tiers) for the VPC, this can be filtered based on the
> vpc.max.networks
> >>> global setting (3 is default). The pros of this option are less no. of
> >>> static route/rule and easy programming, but potential wastage of
> multiple
> >>> /64 prefix blocks for unused/uncreated tiers.
> >>>> * How do we manage IPv6 usage? Can anyone advise how we do IPv6
> >>> usage for shared network (design, implementation and use-cases?)
> >>>> Option1: We don't do it, all user VMs nics have ipv4 address which
> whose
> >>> usage we don't track. For public VR/nics/networks, we can simply add
> the
> >>> IPv6 details for a related IPv4 address.
> >>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
> >>> tracking/management and usage for all VMs and systemvms nic (this means
> >>> account/domain level limits and new billing/records)
> >>>> * Enable IPv6 on systemvms (specifically SSVM and CPVM) by
> default
> >>> if zone has a IPv6 address block allocated/assigned for use for
> systemvms
> >>> (this was mainly thought for VRs, but why no ssvm and cpvms too - any
> cons
> >>> of this?)
> >>>> *
> >>>>
> >>>> Regards.
> >>>>
> >>>> ________________________________
> >>>> From: Rohit Yadav <ro...@shapeblue.com>
> >>>> Sent: Thursday, August 19, 2021 15:45
> >>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >>>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I've taken feedback from this thread and wrote this design doc:
> >>>>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> >>>> +Isolated+Network+and+VPC+with+Static+Routing
> >>>>
> >>>> Kindly review and advise if I missed anything or anything that needs
> to
> >>> be changed/updated. You may comment on the wiki directly too.
> >>>>
> >>>> Kindly suggest your views on the following (also in the design doc
> >>> above):
> >>>>
> >>>> Outstanding Questions:
> >>>>
> >>>> * Should admin or user be able to specify how VPC super CIDRs
> are
> >>> created/needed; for example a user can ask for VPC with /60 super
> CIDR? Or
> >>> should CloudStack automatically find/allocate a /64 for a new VPC tier
> from
> >>> the root-admin configured /64-/48 block?
> >>>> * Should we explore FRR and iBGP or other strategies to do
> dynamic
> >>> routing which may not require advance/complex configuration in the VR
> or
> >>> for the users/admin?
> >>>> * With SLAAC and no dhcpv6, is there a way to support secondary
> IPv6
> >>> addresses (or floating IPv6 addresses for VR/public traffic) for guest
> VM's
> >>> nics?
> >>>> * Any thoughts on UI/UX for firewall/routing management?
> >>>> * Any other feature/support for isolated network or VPC feature
> that
> >>> must be explored or supported such as PF, VPN, LB, vpc static routes,
> vpc
> >>> gateway etc.
> >>>> * For usage - should we have any consideration, or should we
> assume
> >>> that IPv4 and IPv6 address will go together for every nic; so IPv6
> usage
> >>> for a nic is in tandem with Ipv4 address for a nic, i.e. no
> explicit/new
> >>> biling/usage needed?
> >>>> * For smoketests, local dev-test should we explore ULA? Unique
> Local
> >>> Address - in the range fc00::/7. Typically only within the 'local' half
> >>> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
> >>> This prefix can be randomly generated at first install by CloudStack
> in a
> >>> zone using zoneid etc?
> >>>> * Should we expand support for VR diagnostic feature to include
> >>> support for ipv6, traceroute6?
> >>>> * Discuss - expand support/ability to allocate a /60, or /56 etc
> >>> prefix to an individual VM in shared network (Wido's suggestion)
> >>>>
> >>>>
> >>>> Regards.
> >>>>
> >>>> ________________________________
> >>>> From: Wei ZHOU <us...@gmail.com>
> >>>> Sent: Tuesday, August 17, 2021 21:16
> >>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >>>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>>
> >>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
> >>>>
> >>>> I am fine with not having stateful dhcpv6 and privacy
> >>>> extension/temporary address in phase 1. If community decides not to do
> >>>> eventually , it is also ok to me.
> >>>>
> >>>> We could explore how to better use secondary ipv6 addresses as Wido
> >>>> advised. It would be great if anyone share some user experience.
> >>>>
> >>>> -Wei
> >>>>
> >>>>
> >>>> On Tuesday, 17 August 2021, Kristaps Cudars
> >>>> <kr...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi Wei,
> >>>>>
> >>>>> Published this month's RFC 9099 and explains in different
> >>>>> words/perspective what has been written by Alex, Rohit and Wido.
> >>>>> https://www.rfc-editor.org/rfc/rfc9099.html
> >>>>>
> >>>>>
> >>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
> >>>>>> Hi Wido,
> >>>>>>
> >>>>>> (cc to Rohit and Alex)
> >>>>>>
> >>>>>> It is a good suggestion to use FRR for ipv6. The configuration is
> >>>>>> quite simple and the VMs can get SLAAC, routes, etc.
> >>>>>>
> >>>>>> Privacy extension looks not the same as what you mentioned. see
> >>>>>> https://datatracker.ietf.org/doc/html/rfc4941
> >>>>>>
> >>>>>> You are right. To use static routing, the admins need to configure
> >>>>>> the routes in the upstream router, and add some ipv6 ranges (eg
> >>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
> >>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
> >>>>>> pick up an IPv6 range
> >>>>> and
> >>>>>> assign it to an isolated network or vpc. @Rohit, correct me if I'm
> >>> wrong.
> >>>>>>
> >>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
> >>>>>> router/VR, router/dns etc via RA messages) will be the only option
> >>>>>> for now (related
> >>>>> to
> >>>>>> your pr https://github.com/apache/cloudstack/pull/3077) . Would it
> >>>>>> be
> >>>>> good
> >>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
> >>>>>> as an option in cloudstack ? The advantages are
> >>>>>> (1) support other ipv6 cidr sizes than /64.
> >>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
> >>>>>> addresses can be changed
> >>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
> >>>>>> The problem is, stateful dhcpv6 does not support
> >>>>>> routers,nameservers,
> >>>>> etc.
> >>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
> >>>>>>
> >>>>>> -Wei
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
> >> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl>
> wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> See my inline responses:
> >>>>>>>
> >>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
> >>>>>>>> discussing
> >>>>> them
> >>>>>>> with Alex and came up with a PoC/design which can be implemented
> >>>>>>> in the following phases:
> >>>>>>>>
> >>>>>>>> * Phase1: implement ipv6 support in isolated networks and
> >>> VPC
> >>>>> with
> >>>>>>> static routing
> >>>>>>>> * Phase2: discuss and implement support for dynamic routing
> >>>>> (TBD)
> >>>>>>>>
> >>>>>>>> For Phase1 here's the high-level proposal:
> >>>>>>>>
> >>>>>>>> * IPv6 address management:
> >>>>>>>> * At the zone level root-admin specifies a /64 public
> >>> range
> >>>>> that
> >>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6 range
> >>>>>>> for
> >>>>> guest
> >>>>>>> networks (to be used by isolated networks and VPC tiers)
> >>>>>>>> * On creation of any IPv6 enabled isolated network or
> VPC
> >>>>> tier,
> >>>>>>> from the /48 or /56 block a /64 network is allocated/used
> >>>>>>>> * We assume SLAAC and autoconfiguration, no DHCPv6 in
> the
> >>>>> zone
> >>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
> >>>>>>> of
> >>>>> slaac be
> >>>>>>> explored?)
> >>>>>>>
> >>>>>>> Privacy Extensions are only a concern for client devices which
> >>>>>>> roam between different IPv6 networks.
> >>>>>>>
> >>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
> >>>>>>> based) and switches network then only the prefix (/64) will change.
> >>>>>>>
> >>>>>>> This way a network like Google, Facebook, etc could track your
> >>>>>>> device moving from network to network if they only look at the
> >>>>>>> last 64-bits of the IPv6 address.
> >>>>>>>
> >>>>>>> For servers this is not a problem as you already know in which
> >>>>>>> network they are.
> >>>>>>>
> >>>>>>>> * Network offerings: root-admin can create new network
> >>> offerings
> >>>>>>> (with VPC too) that specifies a network stack option:
> >>>>>>>> * ipv4 only (default, for backward compatibility all
> >>>>>>> networks/offerings post-upgrade migrate to this option)
> >>>>>>>> * ipv4-and-ipv6
> >>>>>>>> * ipv6-only (this can be phase 1.b)
> >>>>>>>> * A new routing option: static (phase1), dynamic
> (phase2,
> >>>>> with
> >>>>>>> multiple sub-options such as ospf/bgp etc...)
> >>>>>>>
> >>>>>>> This means that the network admin will need to statically route
> >>>>>>> the
> >>>>> IPv6
> >>>>>>> subnet to the VR's outside IPv6 address, for example, on a JunOS
> >>>>> router:
> >>>>>>>
> >>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
> >>>>>>> next-hop
> >>>>>>> 2001:db8:100::50
> >>>>>>>
> >>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
> >>>>>>> the outside (/64) network. In reality this will probably be a
> >>>>>>> longer address, but this is for just the example.
> >>>>>>>
> >>>>>>>> * VR changes:
> >>>>>>>> * VR gets its guest and public nics set to inet6 auto
> >>>>>>>> * For each /64 allocated to guest network and VPC tiers,
> >>>>> radvd
> >>>>>>> is configured to do RA
> >>>>>>>
> >>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
> >>>>>>> might already want to look into FRRouting. FRR can also
> >>>>>>> advertise RAs while not doing any routing.
> >>>>>>>
> >>>>>>> interface ens4
> >>>>>>> no ipv6 nd suppress-ra
> >>>>>>> ipv6 nd prefix 2001:db8:500::/64
> >>>>>>> ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> >>>>>>>
> >>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
> >>>>>>>
> >>>>>>>> * Firewall: a new ipv6 zone/chain is created for ipv6
> >>> where
> >>>>> ipv6
> >>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
> >>>>>>> between
> >>>>> VPC
> >>>>>>> tiers are managed/implemented by ipv6 firewall on VR
> >>>>>>>
> >>>>>>> Please take a look at the existing security_group.py script
> >>>>>>> which implements RFC4890
> >>>>>>>
> >>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
> >>>>>>>
> >>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should always
> >>>>>>> be allowed.
> >>>>>>>
> >>>>>>>> * It is assumed that static routes are created on the
> >>>>> core/main
> >>>>>>> router by the admin or automated using some scripts/tools; for
> >>>>>>> this CloudStack will announce events with details of /64
> >>>>>>> networks and VR's public IPv6 address that can be consumed by a
> >>>>>>> rabbitmq/message bus
> >>>>> client
> >>>>>>> (for example), or a custom cron job or script as part of
> >>> orchestration.
> >>>>>>> (this wouldn't be necessary for dynamic routing bgp with
> >>>>>>> phase2)\\
> >>>>>>>
> >>>>>>> You would only need to announce the /48 or /56 allocated to the
> >>>>>>> VR, that's all. You don't need to inform the upstream router
> >>>>>>> about the /64 subnets created within that larger subnet.
> >>>>>>>
> >>>>>>>> * Guest Networking: With SLAAC, it's easy for CloudStack to
> >>>>>>> calculate allocate and use a /64 and determine the IPv6 address
> >>>>>>> of VR
> >>>>> nics
> >>>>>>> and guest VM nics
> >>>>>>>> * A user create an isolated network/VPC with an offering
> >>>>> that is
> >>>>>>> ipv6 enabled
> >>>>>>>> * A user can manage firewall for the IPv6 address/guest
> >>> nics;
> >>>>>>> there'll be no port forward and LB feature though for IPv6
> >>>>>>>> * A users can run workloads in the guest VMs that listen
> >>> on
> >>>>>>> publically routable ipv6 addresses
> >>>>>>>> * Usage/billing etc continue to work, no change needed
> >>>>>>>>
> >>>>>>>> Network layout:
> >>>>>>>>
> >>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
> >>>>>>>> VLAN] ->
> >>>>>>> [guest VMs/nics]
> >>>>>>>> *core/ISP router needs static routes to be added (manually or
> >>>>>>> automated), assumes a /48 or /56 configured for the zone
> >>>>>>>>
> >>>>>>>> Thoughts, feedback?
> >>>>>>>
> >>>>>>> Looks doable!
> >>>>>>>
> >>>>>>> Side-note: It would be very cool if you could use parts of this
> >>>>>>> implementation to also route /48, /56, or /60 subnets to
> >>>>>>> individual VMs in Shared networks.
> >>>>>>>
> >>>>>>> Why? This allows for running Docker containers with native IPv6
> >>>>>>> inside the VM or running a (Open)VPN server inside a VM which
> >>>>>>> then assigns public IPv6 addresses to clients connected.
> >>>>>>>
> >>>>>>> Instead of routing the subnet to a VR we route the subnet to a
> >>>>>>> single instance in a shared network.
> >>>>>>>
> >>>>>>> If we could then also move these subnets between Instances
> >>>>>>> easily one can quickly migrate to a different instance while
> >>>>>>> keeping the same IPv6 subnet.
> >>>>>>>
> >>>>>>> Wido
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
> >>>>>>>>
> >>>>>>>> * Created an isolated network and deployed a VM in my home
> >>> lab
> >>>>>>>> The VR running on KVM has following nics
> >>>>>>>> eth0 - guest network
> >>>>>>>> eth1 - link local
> >>>>>>>> eth2 - public network
> >>>>>>>>
> >>>>>>>> * I setup a custom openwrt router on a RPi4 to serve as a
> >>>>> toy-core
> >>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker and
> >>>>>>> I got
> >>>>> a
> >>>>>>> /48 allocated. My configuration looks like:
> >>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> >>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
> >>>>>>>>
> >>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
> >>>>> subnet/prefix
> >>>>>>> 0:
> >>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
> >>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
> >>>>>>>> *
> >>>>>>>> *
> >>>>>>>> In the isolated VR, I enabled ipv6 as:
> >>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
> >>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
> >>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
> >>>>>>>> net.ipv6.conf.all.autoconf = 1
> >>>>>>>>
> >>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
> >>>>>>>> configured the nics:
> >>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
> >>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
> >>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
> >>>>>>>> network without cleanup to have it
> >>>>>>> reconfigure IPv4 nics, firewall, NAT etc
> >>>>>>>>
> >>>>>>>> *
> >>>>>>>> Next, I created a /64 network for the isolated guest network
> >>>>>>>> on eth0
> >>>>> of
> >>>>>>> VR using radvd:
> >>>>>>>>
> >>>>>>>> # cat /etc/radvd.conf
> >>>>>>>> interface eth0
> >>>>>>>> {
> >>>>>>>> AdvSendAdvert on;
> >>>>>>>> MinRtrAdvInterval 5;
> >>>>>>>> MaxRtrAdvInterval 15;
> >>>>>>>> prefix 2001:470:ed36:1::/64
> >>>>>>>> {
> >>>>>>>> AdvOnLink on;
> >>>>>>>> AdvAutonomous on;
> >>>>>>>> };
> >>>>>>>> };
> >>>>>>>> systemctl restart radvd
> >>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
> >>>>>>>> this
> >>>>>>> ...:1::/64 network
> >>>>>>>> * Finally I added a static route in toy core-router for the
> >>> new
> >>>>> /64
> >>>>>>> IPv6 range in the isolated network
> >>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
> >>>>>>>> eth2> dev
> >>>>>>> <local lan nic>
> >>>>>>>> *
> >>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
> >>>>>>>> for the
> >>>>>>> new /64 network
> >>>>>>>>
> >>>>>>>> And voila all done! I create a domain AAAA record that points
> >>>>>>>> to my
> >>>>>>> guest VM IPv6 address a test webserver on
> >>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
> >>>>>>>>
> >>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
> >>>>>>>> after a
> >>>>>>> few days, sharing this solely for testing purposes)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Regards.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >
>
Re: IPV6 in Isolated/VPC networks
Posted by Wido den Hollander <wi...@widodh.nl>.
Hi,
Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
> Yes. When a public IPv6 is assigned or released, CloudStack will publish event with type NET.IP6ASSIGN, NET.IP6RELEASE.
> These event notifications can be tracked. And with improvements in events framework, these event messages will have network uuid as entityuuid and Network as entitytype. Using this network can be queried using to list IPv6 routes that need to be added.
>
> Sample event message,
>
> {"eventDateTime":"2022-05-10 09:32:12 +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":"14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned public IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID: 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":"bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-af19-1e00320001f3","entity":"Network","status":"Completed"}
>
>
> ?Sample API call,
>> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98 filter=id,name,ip6routes
> {
> "count": 1,
> "network": [
> {
> "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
> "ip6routes": [
> {
> "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
> "subnet": "fd23:313a:2f53:3cbf::/64"
> }
Looking at this JSON, does this mean that
fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
And that I would need to (statically) route fd23:313a:2f53:3cbf::/64 to
that IP?
The event message does not include the subnet, that makes it a bit more
difficult as you would then also need to do a API-call to gather that
information.
Wido
P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an
AAAA-record for IPv6!
> ],
> "name": "routing_test"
> }
> ]
> }
>
>
>
>
> ________________________________
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: 10 May 2022 13:59
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek Kumar <Ab...@shapeblue.com>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
>> Hi all,
>>
>> IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
>> Documentation for it is available at http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier (will be available in the official docs once 4.17.0 version of docs is built).
>>
>
> Great work!
>
> I see only static routing is supported. But do we publish something on
> the message bus once a new VR/VPC is created?
>
> This way you could pick up these messages and have the network create a
> (static) route based on those.
>
> ExaBGP for example could be used to inject such routes.
>
> Wido
>
>> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
>>
>> Regards,
>> Abhishek
>>
>> ________________________________
>> From: Rohit Yadav <ro...@shapeblue.com>
>> Sent: 13 September 2021 14:30
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Thanks Alex, Wei. I've updated the docs here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
>>
>> I'll leave the thread open for futher discussion/ideas/feedback. I think we've completed the phase1 design doc including all feedback comments for adding IPv6 support in CloudStack and some initial poc/work can be started. My colleagues and I will keep everyone posted on this thread and/or on a Github PR as and when we're able to start our work on the same (after 4.16, potentially towards 4.17).
>>
>>
>> Regards.
>>
>> ________________________________
>> From: Wei ZHOU <us...@gmail.com>
>> Sent: Friday, September 10, 2021 20:22
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Agree with Alex.
>> We only need to know how many /64 are allocated. We do not care how many
>> ipv6 addresses are used by VMs.
>>
>> -Wei
>>
>> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <Al...@shapeblue.com>
>> wrote:
>>
>>> Hi Rohit,
>>>
>>> I'd go for option 2, don't see a point tracking anything smaller than a
>>> /64 tbh.
>>>
>>> Cheers
>>> Alex
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Rohit Yadav <ro...@shapeblue.com>
>>> Sent: 09 September 2021 12:44
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>>> agreements:
>>>
>>> * Allocate /64 for both isolated network and VPC tiers, no large
>>> allocation of prefixes to VPC (cons: more static routing rules for upstream
>>> router/admins)
>>> * All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a
>>> dedicated /64 prefix/block for systemvms
>>>
>>> The only outstanding question now is:
>>>
>>> * How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage
>>> for shared network (design, implementation and use-cases?)
>>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>>> usage we don't track. For public VR/nics/networks, we can simply add the
>>> IPv6 details for a related IPv4 address.
>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>>> tracking/management and usage for all VMs and systemvms nic (this means
>>> account/domain level limits and new billing/records)
>>> Option3: other thoughts?
>>>
>>>
>>> Regards.
>>>
>>> ________________________________
>>> From: Alex Mattioli <Al...@shapeblue.com>
>>> Sent: Wednesday, September 8, 2021 23:24
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: RE: IPV6 in Isolated/VPC networks
>>>
>>> Hi Rohit, Kristaps,
>>>
>>> I'd say option 1 as well, it does create a bit more overhead with static
>>> routes but if that's automated for a VPC it can also be easily automated
>>> for several tiers of a VPC. We also don't constrain the amount of tiers in
>>> a VPC.
>>> It has the added advantage to be closer to the desired behaviour with
>>> dynamic routing in the future, where a VPC VR can announce several subnets
>>> upstream.
>>>
>>> Cheers
>>> Alex
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Rohit Yadav <ro...@shapeblue.com>
>>> Sent: 08 September 2021 19:04
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Hi Kristaps,
>>>
>>> Thanks for sharing, I suppose that means individual tiers should be
>>> allocated /64 instead of larger ipv6 blocks to the whole VPC which could
>>> cause wastage.
>>>
>>> Any objection from anybody?
>>>
>>> Regards.
>>> ________________________________
>>> From: Kristaps Cudars <kr...@gmail.com>
>>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Hello,
>>>
>>> I asked networking team to comment on "How should the IPv6
>>> block/allocation work in VPCs?"
>>> Option1: They haven't seen lately devices with limits on how many static
>>> routes can be created.
>>> Option2: With /60 and /62 assignments and big quantity of routers IPv6
>>> assignment from RIPE NNC can be drained fast.
>>>
>>> /48 contains 64000 /64
>>> /60 contains 16 /64
>>> 64000 / 16 = 4000 routers
>>>
>>>
>>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
>>>> All,
>>>>
>>>> After another iteration with Alex, I've updated the design doc. Kindly
>>> review:
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>
>>>>
>>>> ... and advise on some outstanding questions:
>>>>
>>>> * How should the IPv6 block/allocation work in VPCs?
>>>> Option1: Should this be simply /64 allocation on any new tier, the
>>>> cons of this option is one static route/rule per VPC tier. (many
>>>> upstream routers may have limit on no. of static routes?)
>>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
>>> (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks
>>> global setting (3 is default). The pros of this option are less no. of
>>> static route/rule and easy programming, but potential wastage of multiple
>>> /64 prefix blocks for unused/uncreated tiers.
>>>> * How do we manage IPv6 usage? Can anyone advise how we do IPv6
>>> usage for shared network (design, implementation and use-cases?)
>>>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>>> usage we don't track. For public VR/nics/networks, we can simply add the
>>> IPv6 details for a related IPv4 address.
>>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>>> tracking/management and usage for all VMs and systemvms nic (this means
>>> account/domain level limits and new billing/records)
>>>> * Enable IPv6 on systemvms (specifically SSVM and CPVM) by default
>>> if zone has a IPv6 address block allocated/assigned for use for systemvms
>>> (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons
>>> of this?)
>>>> *
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>> Sent: Thursday, August 19, 2021 15:45
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Hi all,
>>>>
>>>> I've taken feedback from this thread and wrote this design doc:
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>
>>>> Kindly review and advise if I missed anything or anything that needs to
>>> be changed/updated. You may comment on the wiki directly too.
>>>>
>>>> Kindly suggest your views on the following (also in the design doc
>>> above):
>>>>
>>>> Outstanding Questions:
>>>>
>>>> * Should admin or user be able to specify how VPC super CIDRs are
>>> created/needed; for example a user can ask for VPC with /60 super CIDR? Or
>>> should CloudStack automatically find/allocate a /64 for a new VPC tier from
>>> the root-admin configured /64-/48 block?
>>>> * Should we explore FRR and iBGP or other strategies to do dynamic
>>> routing which may not require advance/complex configuration in the VR or
>>> for the users/admin?
>>>> * With SLAAC and no dhcpv6, is there a way to support secondary IPv6
>>> addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's
>>> nics?
>>>> * Any thoughts on UI/UX for firewall/routing management?
>>>> * Any other feature/support for isolated network or VPC feature that
>>> must be explored or supported such as PF, VPN, LB, vpc static routes, vpc
>>> gateway etc.
>>>> * For usage - should we have any consideration, or should we assume
>>> that IPv4 and IPv6 address will go together for every nic; so IPv6 usage
>>> for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new
>>> biling/usage needed?
>>>> * For smoketests, local dev-test should we explore ULA? Unique Local
>>> Address - in the range fc00::/7. Typically only within the 'local' half
>>> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
>>> This prefix can be randomly generated at first install by CloudStack in a
>>> zone using zoneid etc?
>>>> * Should we expand support for VR diagnostic feature to include
>>> support for ipv6, traceroute6?
>>>> * Discuss - expand support/ability to allocate a /60, or /56 etc
>>> prefix to an individual VM in shared network (Wido's suggestion)
>>>>
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Wei ZHOU <us...@gmail.com>
>>>> Sent: Tuesday, August 17, 2021 21:16
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>>
>>>> I am fine with not having stateful dhcpv6 and privacy
>>>> extension/temporary address in phase 1. If community decides not to do
>>>> eventually , it is also ok to me.
>>>>
>>>> We could explore how to better use secondary ipv6 addresses as Wido
>>>> advised. It would be great if anyone share some user experience.
>>>>
>>>> -Wei
>>>>
>>>>
>>>> On Tuesday, 17 August 2021, Kristaps Cudars
>>>> <kr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Wei,
>>>>>
>>>>> Published this month's RFC 9099 and explains in different
>>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>>> https://www.rfc-editor.org/rfc/rfc9099.html
>>>>>
>>>>>
>>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
>>>>>> Hi Wido,
>>>>>>
>>>>>> (cc to Rohit and Alex)
>>>>>>
>>>>>> It is a good suggestion to use FRR for ipv6. The configuration is
>>>>>> quite simple and the VMs can get SLAAC, routes, etc.
>>>>>>
>>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>>> https://datatracker.ietf.org/doc/html/rfc4941
>>>>>>
>>>>>> You are right. To use static routing, the admins need to configure
>>>>>> the routes in the upstream router, and add some ipv6 ranges (eg
>>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
>>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
>>>>>> pick up an IPv6 range
>>>>> and
>>>>>> assign it to an isolated network or vpc. @Rohit, correct me if I'm
>>> wrong.
>>>>>>
>>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
>>>>>> router/VR, router/dns etc via RA messages) will be the only option
>>>>>> for now (related
>>>>> to
>>>>>> your pr https://github.com/apache/cloudstack/pull/3077) . Would it
>>>>>> be
>>>>> good
>>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
>>>>>> as an option in cloudstack ? The advantages are
>>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
>>>>>> addresses can be changed
>>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>>> The problem is, stateful dhcpv6 does not support
>>>>>> routers,nameservers,
>>>>> etc.
>>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>>
>>>>>> -Wei
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>>
>
>
>> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> See my inline responses:
>>>>>>>
>>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
>>>>>>>> discussing
>>>>> them
>>>>>>> with Alex and came up with a PoC/design which can be implemented
>>>>>>> in the following phases:
>>>>>>>>
>>>>>>>> * Phase1: implement ipv6 support in isolated networks and
>>> VPC
>>>>> with
>>>>>>> static routing
>>>>>>>> * Phase2: discuss and implement support for dynamic routing
>>>>> (TBD)
>>>>>>>>
>>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>>
>>>>>>>> * IPv6 address management:
>>>>>>>> * At the zone level root-admin specifies a /64 public
>>> range
>>>>> that
>>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6 range
>>>>>>> for
>>>>> guest
>>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>>> * On creation of any IPv6 enabled isolated network or VPC
>>>>> tier,
>>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>>> * We assume SLAAC and autoconfiguration, no DHCPv6 in the
>>>>> zone
>>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
>>>>>>> of
>>>>> slaac be
>>>>>>> explored?)
>>>>>>>
>>>>>>> Privacy Extensions are only a concern for client devices which
>>>>>>> roam between different IPv6 networks.
>>>>>>>
>>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>>
>>>>>>> This way a network like Google, Facebook, etc could track your
>>>>>>> device moving from network to network if they only look at the
>>>>>>> last 64-bits of the IPv6 address.
>>>>>>>
>>>>>>> For servers this is not a problem as you already know in which
>>>>>>> network they are.
>>>>>>>
>>>>>>>> * Network offerings: root-admin can create new network
>>> offerings
>>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>>> * ipv4 only (default, for backward compatibility all
>>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>>> * ipv4-and-ipv6
>>>>>>>> * ipv6-only (this can be phase 1.b)
>>>>>>>> * A new routing option: static (phase1), dynamic (phase2,
>>>>> with
>>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>>
>>>>>>> This means that the network admin will need to statically route
>>>>>>> the
>>>>> IPv6
>>>>>>> subnet to the VR's outside IPv6 address, for example, on a JunOS
>>>>> router:
>>>>>>>
>>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
>>>>>>> next-hop
>>>>>>> 2001:db8:100::50
>>>>>>>
>>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
>>>>>>> the outside (/64) network. In reality this will probably be a
>>>>>>> longer address, but this is for just the example.
>>>>>>>
>>>>>>>> * VR changes:
>>>>>>>> * VR gets its guest and public nics set to inet6 auto
>>>>>>>> * For each /64 allocated to guest network and VPC tiers,
>>>>> radvd
>>>>>>> is configured to do RA
>>>>>>>
>>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
>>>>>>> might already want to look into FRRouting. FRR can also
>>>>>>> advertise RAs while not doing any routing.
>>>>>>>
>>>>>>> interface ens4
>>>>>>> no ipv6 nd suppress-ra
>>>>>>> ipv6 nd prefix 2001:db8:500::/64
>>>>>>> ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>>
>>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
>>>>>>>
>>>>>>>> * Firewall: a new ipv6 zone/chain is created for ipv6
>>> where
>>>>> ipv6
>>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
>>>>>>> between
>>>>> VPC
>>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>>
>>>>>>> Please take a look at the existing security_group.py script
>>>>>>> which implements RFC4890
>>>>>>>
>>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
>>>>>>>
>>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should always
>>>>>>> be allowed.
>>>>>>>
>>>>>>>> * It is assumed that static routes are created on the
>>>>> core/main
>>>>>>> router by the admin or automated using some scripts/tools; for
>>>>>>> this CloudStack will announce events with details of /64
>>>>>>> networks and VR's public IPv6 address that can be consumed by a
>>>>>>> rabbitmq/message bus
>>>>> client
>>>>>>> (for example), or a custom cron job or script as part of
>>> orchestration.
>>>>>>> (this wouldn't be necessary for dynamic routing bgp with
>>>>>>> phase2)\\
>>>>>>>
>>>>>>> You would only need to announce the /48 or /56 allocated to the
>>>>>>> VR, that's all. You don't need to inform the upstream router
>>>>>>> about the /64 subnets created within that larger subnet.
>>>>>>>
>>>>>>>> * Guest Networking: With SLAAC, it's easy for CloudStack to
>>>>>>> calculate allocate and use a /64 and determine the IPv6 address
>>>>>>> of VR
>>>>> nics
>>>>>>> and guest VM nics
>>>>>>>> * A user create an isolated network/VPC with an offering
>>>>> that is
>>>>>>> ipv6 enabled
>>>>>>>> * A user can manage firewall for the IPv6 address/guest
>>> nics;
>>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>>> * A users can run workloads in the guest VMs that listen
>>> on
>>>>>>> publically routable ipv6 addresses
>>>>>>>> * Usage/billing etc continue to work, no change needed
>>>>>>>>
>>>>>>>> Network layout:
>>>>>>>>
>>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
>>>>>>>> VLAN] ->
>>>>>>> [guest VMs/nics]
>>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>>
>>>>>>>> Thoughts, feedback?
>>>>>>>
>>>>>>> Looks doable!
>>>>>>>
>>>>>>> Side-note: It would be very cool if you could use parts of this
>>>>>>> implementation to also route /48, /56, or /60 subnets to
>>>>>>> individual VMs in Shared networks.
>>>>>>>
>>>>>>> Why? This allows for running Docker containers with native IPv6
>>>>>>> inside the VM or running a (Open)VPN server inside a VM which
>>>>>>> then assigns public IPv6 addresses to clients connected.
>>>>>>>
>>>>>>> Instead of routing the subnet to a VR we route the subnet to a
>>>>>>> single instance in a shared network.
>>>>>>>
>>>>>>> If we could then also move these subnets between Instances
>>>>>>> easily one can quickly migrate to a different instance while
>>>>>>> keeping the same IPv6 subnet.
>>>>>>>
>>>>>>> Wido
>>>>>>>
>>>>>>>>
>>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>>
>>>>>>>> * Created an isolated network and deployed a VM in my home
>>> lab
>>>>>>>> The VR running on KVM has following nics
>>>>>>>> eth0 - guest network
>>>>>>>> eth1 - link local
>>>>>>>> eth2 - public network
>>>>>>>>
>>>>>>>> * I setup a custom openwrt router on a RPi4 to serve as a
>>>>> toy-core
>>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker and
>>>>>>> I got
>>>>> a
>>>>>>> /48 allocated. My configuration looks like:
>>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>>
>>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>>> subnet/prefix
>>>>>>> 0:
>>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>>> *
>>>>>>>> *
>>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
>>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
>>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
>>>>>>>> net.ipv6.conf.all.autoconf = 1
>>>>>>>>
>>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
>>>>>>>> configured the nics:
>>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
>>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
>>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
>>>>>>>> network without cleanup to have it
>>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>>
>>>>>>>> *
>>>>>>>> Next, I created a /64 network for the isolated guest network
>>>>>>>> on eth0
>>>>> of
>>>>>>> VR using radvd:
>>>>>>>>
>>>>>>>> # cat /etc/radvd.conf
>>>>>>>> interface eth0
>>>>>>>> {
>>>>>>>> AdvSendAdvert on;
>>>>>>>> MinRtrAdvInterval 5;
>>>>>>>> MaxRtrAdvInterval 15;
>>>>>>>> prefix 2001:470:ed36:1::/64
>>>>>>>> {
>>>>>>>> AdvOnLink on;
>>>>>>>> AdvAutonomous on;
>>>>>>>> };
>>>>>>>> };
>>>>>>>> systemctl restart radvd
>>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
>>>>>>>> this
>>>>>>> ...:1::/64 network
>>>>>>>> * Finally I added a static route in toy core-router for the
>>> new
>>>>> /64
>>>>>>> IPv6 range in the isolated network
>>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>>> eth2> dev
>>>>>>> <local lan nic>
>>>>>>>> *
>>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
>>>>>>>> for the
>>>>>>> new /64 network
>>>>>>>>
>>>>>>>> And voila all done! I create a domain AAAA record that points
>>>>>>>> to my
>>>>>>> guest VM IPv6 address a test webserver on
>>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>>>>>>>>
>>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
>>>>>>>> after a
>>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>
Re: IPV6 in Isolated/VPC networks
Posted by Abhishek Kumar <Ab...@shapeblue.com>.
Yes. When a public IPv6 is assigned or released, CloudStack will publish event with type NET.IP6ASSIGN, NET.IP6RELEASE.
These event notifications can be tracked. And with improvements in events framework, these event messages will have network uuid as entityuuid and Network as entitytype. Using this network can be queried using to list IPv6 routes that need to be added.
Sample event message,
{"eventDateTime":"2022-05-10 09:32:12 +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":"14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned public IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID: 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":"bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-af19-1e00320001f3","entity":"Network","status":"Completed"}
?Sample API call,
> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98 filter=id,name,ip6routes
{
"count": 1,
"network": [
{
"id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
"ip6routes": [
{
"gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
"subnet": "fd23:313a:2f53:3cbf::/64"
}
],
"name": "routing_test"
}
]
}
________________________________
From: Wido den Hollander <wi...@widodh.nl>
Sent: 10 May 2022 13:59
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek Kumar <Ab...@shapeblue.com>
Subject: Re: IPV6 in Isolated/VPC networks
Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
> Hi all,
>
> IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
> Documentation for it is available at http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier (will be available in the official docs once 4.17.0 version of docs is built).
>
Great work!
I see only static routing is supported. But do we publish something on
the message bus once a new VR/VPC is created?
This way you could pick up these messages and have the network create a
(static) route based on those.
ExaBGP for example could be used to inject such routes.
Wido
> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
>
> Regards,
> Abhishek
>
> ________________________________
> From: Rohit Yadav <ro...@shapeblue.com>
> Sent: 13 September 2021 14:30
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Thanks Alex, Wei. I've updated the docs here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
>
> I'll leave the thread open for futher discussion/ideas/feedback. I think we've completed the phase1 design doc including all feedback comments for adding IPv6 support in CloudStack and some initial poc/work can be started. My colleagues and I will keep everyone posted on this thread and/or on a Github PR as and when we're able to start our work on the same (after 4.16, potentially towards 4.17).
>
>
> Regards.
>
> ________________________________
> From: Wei ZHOU <us...@gmail.com>
> Sent: Friday, September 10, 2021 20:22
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Agree with Alex.
> We only need to know how many /64 are allocated. We do not care how many
> ipv6 addresses are used by VMs.
>
> -Wei
>
> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <Al...@shapeblue.com>
> wrote:
>
>> Hi Rohit,
>>
>> I'd go for option 2, don't see a point tracking anything smaller than a
>> /64 tbh.
>>
>> Cheers
>> Alex
>>
>>
>>
>>
>> -----Original Message-----
>> From: Rohit Yadav <ro...@shapeblue.com>
>> Sent: 09 September 2021 12:44
>> To: dev@cloudstack.apache.org
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>> agreements:
>>
>> * Allocate /64 for both isolated network and VPC tiers, no large
>> allocation of prefixes to VPC (cons: more static routing rules for upstream
>> router/admins)
>> * All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a
>> dedicated /64 prefix/block for systemvms
>>
>> The only outstanding question now is:
>>
>> * How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage
>> for shared network (design, implementation and use-cases?)
>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>> usage we don't track. For public VR/nics/networks, we can simply add the
>> IPv6 details for a related IPv4 address.
>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>> tracking/management and usage for all VMs and systemvms nic (this means
>> account/domain level limits and new billing/records)
>> Option3: other thoughts?
>>
>>
>> Regards.
>>
>> ________________________________
>> From: Alex Mattioli <Al...@shapeblue.com>
>> Sent: Wednesday, September 8, 2021 23:24
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: RE: IPV6 in Isolated/VPC networks
>>
>> Hi Rohit, Kristaps,
>>
>> I'd say option 1 as well, it does create a bit more overhead with static
>> routes but if that's automated for a VPC it can also be easily automated
>> for several tiers of a VPC. We also don't constrain the amount of tiers in
>> a VPC.
>> It has the added advantage to be closer to the desired behaviour with
>> dynamic routing in the future, where a VPC VR can announce several subnets
>> upstream.
>>
>> Cheers
>> Alex
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Rohit Yadav <ro...@shapeblue.com>
>> Sent: 08 September 2021 19:04
>> To: dev@cloudstack.apache.org
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Hi Kristaps,
>>
>> Thanks for sharing, I suppose that means individual tiers should be
>> allocated /64 instead of larger ipv6 blocks to the whole VPC which could
>> cause wastage.
>>
>> Any objection from anybody?
>>
>> Regards.
>> ________________________________
>> From: Kristaps Cudars <kr...@gmail.com>
>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Hello,
>>
>> I asked networking team to comment on "How should the IPv6
>> block/allocation work in VPCs?"
>> Option1: They haven't seen lately devices with limits on how many static
>> routes can be created.
>> Option2: With /60 and /62 assignments and big quantity of routers IPv6
>> assignment from RIPE NNC can be drained fast.
>>
>> /48 contains 64000 /64
>> /60 contains 16 /64
>> 64000 / 16 = 4000 routers
>>
>>
>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
>>> All,
>>>
>>> After another iteration with Alex, I've updated the design doc. Kindly
>> review:
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>
>>>
>>> ... and advise on some outstanding questions:
>>>
>>> * How should the IPv6 block/allocation work in VPCs?
>>> Option1: Should this be simply /64 allocation on any new tier, the
>>> cons of this option is one static route/rule per VPC tier. (many
>>> upstream routers may have limit on no. of static routes?)
>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
>> (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks
>> global setting (3 is default). The pros of this option are less no. of
>> static route/rule and easy programming, but potential wastage of multiple
>> /64 prefix blocks for unused/uncreated tiers.
>>> * How do we manage IPv6 usage? Can anyone advise how we do IPv6
>> usage for shared network (design, implementation and use-cases?)
>>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>> usage we don't track. For public VR/nics/networks, we can simply add the
>> IPv6 details for a related IPv4 address.
>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>> tracking/management and usage for all VMs and systemvms nic (this means
>> account/domain level limits and new billing/records)
>>> * Enable IPv6 on systemvms (specifically SSVM and CPVM) by default
>> if zone has a IPv6 address block allocated/assigned for use for systemvms
>> (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons
>> of this?)
>>> *
>>>
>>> Regards.
>>>
>>> ________________________________
>>> From: Rohit Yadav <ro...@shapeblue.com>
>>> Sent: Thursday, August 19, 2021 15:45
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Hi all,
>>>
>>> I've taken feedback from this thread and wrote this design doc:
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>
>>> Kindly review and advise if I missed anything or anything that needs to
>> be changed/updated. You may comment on the wiki directly too.
>>>
>>> Kindly suggest your views on the following (also in the design doc
>> above):
>>>
>>> Outstanding Questions:
>>>
>>> * Should admin or user be able to specify how VPC super CIDRs are
>> created/needed; for example a user can ask for VPC with /60 super CIDR? Or
>> should CloudStack automatically find/allocate a /64 for a new VPC tier from
>> the root-admin configured /64-/48 block?
>>> * Should we explore FRR and iBGP or other strategies to do dynamic
>> routing which may not require advance/complex configuration in the VR or
>> for the users/admin?
>>> * With SLAAC and no dhcpv6, is there a way to support secondary IPv6
>> addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's
>> nics?
>>> * Any thoughts on UI/UX for firewall/routing management?
>>> * Any other feature/support for isolated network or VPC feature that
>> must be explored or supported such as PF, VPN, LB, vpc static routes, vpc
>> gateway etc.
>>> * For usage - should we have any consideration, or should we assume
>> that IPv4 and IPv6 address will go together for every nic; so IPv6 usage
>> for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new
>> biling/usage needed?
>>> * For smoketests, local dev-test should we explore ULA? Unique Local
>> Address - in the range fc00::/7. Typically only within the 'local' half
>> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
>> This prefix can be randomly generated at first install by CloudStack in a
>> zone using zoneid etc?
>>> * Should we expand support for VR diagnostic feature to include
>> support for ipv6, traceroute6?
>>> * Discuss - expand support/ability to allocate a /60, or /56 etc
>> prefix to an individual VM in shared network (Wido's suggestion)
>>>
>>>
>>> Regards.
>>>
>>> ________________________________
>>> From: Wei ZHOU <us...@gmail.com>
>>> Sent: Tuesday, August 17, 2021 21:16
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>
>>> I am fine with not having stateful dhcpv6 and privacy
>>> extension/temporary address in phase 1. If community decides not to do
>>> eventually , it is also ok to me.
>>>
>>> We could explore how to better use secondary ipv6 addresses as Wido
>>> advised. It would be great if anyone share some user experience.
>>>
>>> -Wei
>>>
>>>
>>> On Tuesday, 17 August 2021, Kristaps Cudars
>>> <kr...@gmail.com>
>>> wrote:
>>>
>>>> Hi Wei,
>>>>
>>>> Published this month's RFC 9099 and explains in different
>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>> https://www.rfc-editor.org/rfc/rfc9099.html
>>>>
>>>>
>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
>>>>> Hi Wido,
>>>>>
>>>>> (cc to Rohit and Alex)
>>>>>
>>>>> It is a good suggestion to use FRR for ipv6. The configuration is
>>>>> quite simple and the VMs can get SLAAC, routes, etc.
>>>>>
>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>> https://datatracker.ietf.org/doc/html/rfc4941
>>>>>
>>>>> You are right. To use static routing, the admins need to configure
>>>>> the routes in the upstream router, and add some ipv6 ranges (eg
>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
>>>>> pick up an IPv6 range
>>>> and
>>>>> assign it to an isolated network or vpc. @Rohit, correct me if I'm
>> wrong.
>>>>>
>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
>>>>> router/VR, router/dns etc via RA messages) will be the only option
>>>>> for now (related
>>>> to
>>>>> your pr https://github.com/apache/cloudstack/pull/3077) . Would it
>>>>> be
>>>> good
>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
>>>>> as an option in cloudstack ? The advantages are
>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
>>>>> addresses can be changed
>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>> The problem is, stateful dhcpv6 does not support
>>>>> routers,nameservers,
>>>> etc.
>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>
>>>>> -Wei
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>
>
>
>
> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> See my inline responses:
>>>>>>
>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
>>>>>>> discussing
>>>> them
>>>>>> with Alex and came up with a PoC/design which can be implemented
>>>>>> in the following phases:
>>>>>>>
>>>>>>> * Phase1: implement ipv6 support in isolated networks and
>> VPC
>>>> with
>>>>>> static routing
>>>>>>> * Phase2: discuss and implement support for dynamic routing
>>>> (TBD)
>>>>>>>
>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>
>>>>>>> * IPv6 address management:
>>>>>>> * At the zone level root-admin specifies a /64 public
>> range
>>>> that
>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6 range
>>>>>> for
>>>> guest
>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>> * On creation of any IPv6 enabled isolated network or VPC
>>>> tier,
>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>> * We assume SLAAC and autoconfiguration, no DHCPv6 in the
>>>> zone
>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
>>>>>> of
>>>> slaac be
>>>>>> explored?)
>>>>>>
>>>>>> Privacy Extensions are only a concern for client devices which
>>>>>> roam between different IPv6 networks.
>>>>>>
>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>
>>>>>> This way a network like Google, Facebook, etc could track your
>>>>>> device moving from network to network if they only look at the
>>>>>> last 64-bits of the IPv6 address.
>>>>>>
>>>>>> For servers this is not a problem as you already know in which
>>>>>> network they are.
>>>>>>
>>>>>>> * Network offerings: root-admin can create new network
>> offerings
>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>> * ipv4 only (default, for backward compatibility all
>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>> * ipv4-and-ipv6
>>>>>>> * ipv6-only (this can be phase 1.b)
>>>>>>> * A new routing option: static (phase1), dynamic (phase2,
>>>> with
>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>
>>>>>> This means that the network admin will need to statically route
>>>>>> the
>>>> IPv6
>>>>>> subnet to the VR's outside IPv6 address, for example, on a JunOS
>>>> router:
>>>>>>
>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
>>>>>> next-hop
>>>>>> 2001:db8:100::50
>>>>>>
>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
>>>>>> the outside (/64) network. In reality this will probably be a
>>>>>> longer address, but this is for just the example.
>>>>>>
>>>>>>> * VR changes:
>>>>>>> * VR gets its guest and public nics set to inet6 auto
>>>>>>> * For each /64 allocated to guest network and VPC tiers,
>>>> radvd
>>>>>> is configured to do RA
>>>>>>
>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
>>>>>> might already want to look into FRRouting. FRR can also
>>>>>> advertise RAs while not doing any routing.
>>>>>>
>>>>>> interface ens4
>>>>>> no ipv6 nd suppress-ra
>>>>>> ipv6 nd prefix 2001:db8:500::/64
>>>>>> ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>
>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
>>>>>>
>>>>>>> * Firewall: a new ipv6 zone/chain is created for ipv6
>> where
>>>> ipv6
>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
>>>>>> between
>>>> VPC
>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>
>>>>>> Please take a look at the existing security_group.py script
>>>>>> which implements RFC4890
>>>>>>
>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
>>>>>>
>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should always
>>>>>> be allowed.
>>>>>>
>>>>>>> * It is assumed that static routes are created on the
>>>> core/main
>>>>>> router by the admin or automated using some scripts/tools; for
>>>>>> this CloudStack will announce events with details of /64
>>>>>> networks and VR's public IPv6 address that can be consumed by a
>>>>>> rabbitmq/message bus
>>>> client
>>>>>> (for example), or a custom cron job or script as part of
>> orchestration.
>>>>>> (this wouldn't be necessary for dynamic routing bgp with
>>>>>> phase2)\\
>>>>>>
>>>>>> You would only need to announce the /48 or /56 allocated to the
>>>>>> VR, that's all. You don't need to inform the upstream router
>>>>>> about the /64 subnets created within that larger subnet.
>>>>>>
>>>>>>> * Guest Networking: With SLAAC, it's easy for CloudStack to
>>>>>> calculate allocate and use a /64 and determine the IPv6 address
>>>>>> of VR
>>>> nics
>>>>>> and guest VM nics
>>>>>>> * A user create an isolated network/VPC with an offering
>>>> that is
>>>>>> ipv6 enabled
>>>>>>> * A user can manage firewall for the IPv6 address/guest
>> nics;
>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>> * A users can run workloads in the guest VMs that listen
>> on
>>>>>> publically routable ipv6 addresses
>>>>>>> * Usage/billing etc continue to work, no change needed
>>>>>>>
>>>>>>> Network layout:
>>>>>>>
>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
>>>>>>> VLAN] ->
>>>>>> [guest VMs/nics]
>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>
>>>>>>> Thoughts, feedback?
>>>>>>
>>>>>> Looks doable!
>>>>>>
>>>>>> Side-note: It would be very cool if you could use parts of this
>>>>>> implementation to also route /48, /56, or /60 subnets to
>>>>>> individual VMs in Shared networks.
>>>>>>
>>>>>> Why? This allows for running Docker containers with native IPv6
>>>>>> inside the VM or running a (Open)VPN server inside a VM which
>>>>>> then assigns public IPv6 addresses to clients connected.
>>>>>>
>>>>>> Instead of routing the subnet to a VR we route the subnet to a
>>>>>> single instance in a shared network.
>>>>>>
>>>>>> If we could then also move these subnets between Instances
>>>>>> easily one can quickly migrate to a different instance while
>>>>>> keeping the same IPv6 subnet.
>>>>>>
>>>>>> Wido
>>>>>>
>>>>>>>
>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>
>>>>>>> * Created an isolated network and deployed a VM in my home
>> lab
>>>>>>> The VR running on KVM has following nics
>>>>>>> eth0 - guest network
>>>>>>> eth1 - link local
>>>>>>> eth2 - public network
>>>>>>>
>>>>>>> * I setup a custom openwrt router on a RPi4 to serve as a
>>>> toy-core
>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker and
>>>>>> I got
>>>> a
>>>>>> /48 allocated. My configuration looks like:
>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>
>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>> subnet/prefix
>>>>>> 0:
>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>> *
>>>>>>> *
>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
>>>>>>> net.ipv6.conf.all.autoconf = 1
>>>>>>>
>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
>>>>>>> configured the nics:
>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
>>>>>>> network without cleanup to have it
>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>
>>>>>>> *
>>>>>>> Next, I created a /64 network for the isolated guest network
>>>>>>> on eth0
>>>> of
>>>>>> VR using radvd:
>>>>>>>
>>>>>>> # cat /etc/radvd.conf
>>>>>>> interface eth0
>>>>>>> {
>>>>>>> AdvSendAdvert on;
>>>>>>> MinRtrAdvInterval 5;
>>>>>>> MaxRtrAdvInterval 15;
>>>>>>> prefix 2001:470:ed36:1::/64
>>>>>>> {
>>>>>>> AdvOnLink on;
>>>>>>> AdvAutonomous on;
>>>>>>> };
>>>>>>> };
>>>>>>> systemctl restart radvd
>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
>>>>>>> this
>>>>>> ...:1::/64 network
>>>>>>> * Finally I added a static route in toy core-router for the
>> new
>>>> /64
>>>>>> IPv6 range in the isolated network
>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>> eth2> dev
>>>>>> <local lan nic>
>>>>>>> *
>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
>>>>>>> for the
>>>>>> new /64 network
>>>>>>>
>>>>>>> And voila all done! I create a domain AAAA record that points
>>>>>>> to my
>>>>>> guest VM IPv6 address a test webserver on
>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>>>>>>>
>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
>>>>>>> after a
>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>
>>>>>>>
>>>>>>> Regards.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>>
>
Re: IPV6 in Isolated/VPC networks
Posted by Wido den Hollander <wi...@widodh.nl>.
Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
> Hi all,
>
> IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
> Documentation for it is available at http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier (will be available in the official docs once 4.17.0 version of docs is built).
>
Great work!
I see only static routing is supported. But do we publish something on
the message bus once a new VR/VPC is created?
This way you could pick up these messages and have the network create a
(static) route based on those.
ExaBGP for example could be used to inject such routes.
Wido
> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
>
> Regards,
> Abhishek
>
> ________________________________
> From: Rohit Yadav <ro...@shapeblue.com>
> Sent: 13 September 2021 14:30
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Thanks Alex, Wei. I've updated the docs here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
>
> I'll leave the thread open for futher discussion/ideas/feedback. I think we've completed the phase1 design doc including all feedback comments for adding IPv6 support in CloudStack and some initial poc/work can be started. My colleagues and I will keep everyone posted on this thread and/or on a Github PR as and when we're able to start our work on the same (after 4.16, potentially towards 4.17).
>
>
> Regards.
>
> ________________________________
> From: Wei ZHOU <us...@gmail.com>
> Sent: Friday, September 10, 2021 20:22
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Agree with Alex.
> We only need to know how many /64 are allocated. We do not care how many
> ipv6 addresses are used by VMs.
>
> -Wei
>
> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <Al...@shapeblue.com>
> wrote:
>
>> Hi Rohit,
>>
>> I'd go for option 2, don't see a point tracking anything smaller than a
>> /64 tbh.
>>
>> Cheers
>> Alex
>>
>>
>>
>>
>> -----Original Message-----
>> From: Rohit Yadav <ro...@shapeblue.com>
>> Sent: 09 September 2021 12:44
>> To: dev@cloudstack.apache.org
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>> agreements:
>>
>> * Allocate /64 for both isolated network and VPC tiers, no large
>> allocation of prefixes to VPC (cons: more static routing rules for upstream
>> router/admins)
>> * All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a
>> dedicated /64 prefix/block for systemvms
>>
>> The only outstanding question now is:
>>
>> * How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage
>> for shared network (design, implementation and use-cases?)
>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>> usage we don't track. For public VR/nics/networks, we can simply add the
>> IPv6 details for a related IPv4 address.
>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>> tracking/management and usage for all VMs and systemvms nic (this means
>> account/domain level limits and new billing/records)
>> Option3: other thoughts?
>>
>>
>> Regards.
>>
>> ________________________________
>> From: Alex Mattioli <Al...@shapeblue.com>
>> Sent: Wednesday, September 8, 2021 23:24
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: RE: IPV6 in Isolated/VPC networks
>>
>> Hi Rohit, Kristaps,
>>
>> I'd say option 1 as well, it does create a bit more overhead with static
>> routes but if that's automated for a VPC it can also be easily automated
>> for several tiers of a VPC. We also don't constrain the amount of tiers in
>> a VPC.
>> It has the added advantage to be closer to the desired behaviour with
>> dynamic routing in the future, where a VPC VR can announce several subnets
>> upstream.
>>
>> Cheers
>> Alex
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Rohit Yadav <ro...@shapeblue.com>
>> Sent: 08 September 2021 19:04
>> To: dev@cloudstack.apache.org
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Hi Kristaps,
>>
>> Thanks for sharing, I suppose that means individual tiers should be
>> allocated /64 instead of larger ipv6 blocks to the whole VPC which could
>> cause wastage.
>>
>> Any objection from anybody?
>>
>> Regards.
>> ________________________________
>> From: Kristaps Cudars <kr...@gmail.com>
>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Hello,
>>
>> I asked networking team to comment on "How should the IPv6
>> block/allocation work in VPCs?"
>> Option1: They haven't seen lately devices with limits on how many static
>> routes can be created.
>> Option2: With /60 and /62 assignments and big quantity of routers IPv6
>> assignment from RIPE NNC can be drained fast.
>>
>> /48 contains 64000 /64
>> /60 contains 16 /64
>> 64000 / 16 = 4000 routers
>>
>>
>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
>>> All,
>>>
>>> After another iteration with Alex, I've updated the design doc. Kindly
>> review:
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>
>>>
>>> ... and advise on some outstanding questions:
>>>
>>> * How should the IPv6 block/allocation work in VPCs?
>>> Option1: Should this be simply /64 allocation on any new tier, the
>>> cons of this option is one static route/rule per VPC tier. (many
>>> upstream routers may have limit on no. of static routes?)
>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
>> (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks
>> global setting (3 is default). The pros of this option are less no. of
>> static route/rule and easy programming, but potential wastage of multiple
>> /64 prefix blocks for unused/uncreated tiers.
>>> * How do we manage IPv6 usage? Can anyone advise how we do IPv6
>> usage for shared network (design, implementation and use-cases?)
>>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>> usage we don't track. For public VR/nics/networks, we can simply add the
>> IPv6 details for a related IPv4 address.
>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>> tracking/management and usage for all VMs and systemvms nic (this means
>> account/domain level limits and new billing/records)
>>> * Enable IPv6 on systemvms (specifically SSVM and CPVM) by default
>> if zone has a IPv6 address block allocated/assigned for use for systemvms
>> (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons
>> of this?)
>>> *
>>>
>>> Regards.
>>>
>>> ________________________________
>>> From: Rohit Yadav <ro...@shapeblue.com>
>>> Sent: Thursday, August 19, 2021 15:45
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Hi all,
>>>
>>> I've taken feedback from this thread and wrote this design doc:
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>
>>> Kindly review and advise if I missed anything or anything that needs to
>> be changed/updated. You may comment on the wiki directly too.
>>>
>>> Kindly suggest your views on the following (also in the design doc
>> above):
>>>
>>> Outstanding Questions:
>>>
>>> * Should admin or user be able to specify how VPC super CIDRs are
>> created/needed; for example a user can ask for VPC with /60 super CIDR? Or
>> should CloudStack automatically find/allocate a /64 for a new VPC tier from
>> the root-admin configured /64-/48 block?
>>> * Should we explore FRR and iBGP or other strategies to do dynamic
>> routing which may not require advance/complex configuration in the VR or
>> for the users/admin?
>>> * With SLAAC and no dhcpv6, is there a way to support secondary IPv6
>> addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's
>> nics?
>>> * Any thoughts on UI/UX for firewall/routing management?
>>> * Any other feature/support for isolated network or VPC feature that
>> must be explored or supported such as PF, VPN, LB, vpc static routes, vpc
>> gateway etc.
>>> * For usage - should we have any consideration, or should we assume
>> that IPv4 and IPv6 address will go together for every nic; so IPv6 usage
>> for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new
>> biling/usage needed?
>>> * For smoketests, local dev-test should we explore ULA? Unique Local
>> Address - in the range fc00::/7. Typically only within the 'local' half
>> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
>> This prefix can be randomly generated at first install by CloudStack in a
>> zone using zoneid etc?
>>> * Should we expand support for VR diagnostic feature to include
>> support for ipv6, traceroute6?
>>> * Discuss - expand support/ability to allocate a /60, or /56 etc
>> prefix to an individual VM in shared network (Wido's suggestion)
>>>
>>>
>>> Regards.
>>>
>>> ________________________________
>>> From: Wei ZHOU <us...@gmail.com>
>>> Sent: Tuesday, August 17, 2021 21:16
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>
>>> I am fine with not having stateful dhcpv6 and privacy
>>> extension/temporary address in phase 1. If community decides not to do
>>> eventually , it is also ok to me.
>>>
>>> We could explore how to better use secondary ipv6 addresses as Wido
>>> advised. It would be great if anyone share some user experience.
>>>
>>> -Wei
>>>
>>>
>>> On Tuesday, 17 August 2021, Kristaps Cudars
>>> <kr...@gmail.com>
>>> wrote:
>>>
>>>> Hi Wei,
>>>>
>>>> Published this month's RFC 9099 and explains in different
>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>> https://www.rfc-editor.org/rfc/rfc9099.html
>>>>
>>>>
>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
>>>>> Hi Wido,
>>>>>
>>>>> (cc to Rohit and Alex)
>>>>>
>>>>> It is a good suggestion to use FRR for ipv6. The configuration is
>>>>> quite simple and the VMs can get SLAAC, routes, etc.
>>>>>
>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>> https://datatracker.ietf.org/doc/html/rfc4941
>>>>>
>>>>> You are right. To use static routing, the admins need to configure
>>>>> the routes in the upstream router, and add some ipv6 ranges (eg
>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
>>>>> pick up an IPv6 range
>>>> and
>>>>> assign it to an isolated network or vpc. @Rohit, correct me if I'm
>> wrong.
>>>>>
>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
>>>>> router/VR, router/dns etc via RA messages) will be the only option
>>>>> for now (related
>>>> to
>>>>> your pr https://github.com/apache/cloudstack/pull/3077) . Would it
>>>>> be
>>>> good
>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
>>>>> as an option in cloudstack ? The advantages are
>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
>>>>> addresses can be changed
>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>> The problem is, stateful dhcpv6 does not support
>>>>> routers,nameservers,
>>>> etc.
>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>
>>>>> -Wei
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>
>
>
>> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> See my inline responses:
>>>>>>
>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
>>>>>>> discussing
>>>> them
>>>>>> with Alex and came up with a PoC/design which can be implemented
>>>>>> in the following phases:
>>>>>>>
>>>>>>> * Phase1: implement ipv6 support in isolated networks and
>> VPC
>>>> with
>>>>>> static routing
>>>>>>> * Phase2: discuss and implement support for dynamic routing
>>>> (TBD)
>>>>>>>
>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>
>>>>>>> * IPv6 address management:
>>>>>>> * At the zone level root-admin specifies a /64 public
>> range
>>>> that
>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6 range
>>>>>> for
>>>> guest
>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>> * On creation of any IPv6 enabled isolated network or VPC
>>>> tier,
>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>> * We assume SLAAC and autoconfiguration, no DHCPv6 in the
>>>> zone
>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
>>>>>> of
>>>> slaac be
>>>>>> explored?)
>>>>>>
>>>>>> Privacy Extensions are only a concern for client devices which
>>>>>> roam between different IPv6 networks.
>>>>>>
>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>
>>>>>> This way a network like Google, Facebook, etc could track your
>>>>>> device moving from network to network if they only look at the
>>>>>> last 64-bits of the IPv6 address.
>>>>>>
>>>>>> For servers this is not a problem as you already know in which
>>>>>> network they are.
>>>>>>
>>>>>>> * Network offerings: root-admin can create new network
>> offerings
>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>> * ipv4 only (default, for backward compatibility all
>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>> * ipv4-and-ipv6
>>>>>>> * ipv6-only (this can be phase 1.b)
>>>>>>> * A new routing option: static (phase1), dynamic (phase2,
>>>> with
>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>
>>>>>> This means that the network admin will need to statically route
>>>>>> the
>>>> IPv6
>>>>>> subnet to the VR's outside IPv6 address, for example, on a JunOS
>>>> router:
>>>>>>
>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
>>>>>> next-hop
>>>>>> 2001:db8:100::50
>>>>>>
>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
>>>>>> the outside (/64) network. In reality this will probably be a
>>>>>> longer address, but this is for just the example.
>>>>>>
>>>>>>> * VR changes:
>>>>>>> * VR gets its guest and public nics set to inet6 auto
>>>>>>> * For each /64 allocated to guest network and VPC tiers,
>>>> radvd
>>>>>> is configured to do RA
>>>>>>
>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
>>>>>> might already want to look into FRRouting. FRR can also
>>>>>> advertise RAs while not doing any routing.
>>>>>>
>>>>>> interface ens4
>>>>>> no ipv6 nd suppress-ra
>>>>>> ipv6 nd prefix 2001:db8:500::/64
>>>>>> ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>
>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
>>>>>>
>>>>>>> * Firewall: a new ipv6 zone/chain is created for ipv6
>> where
>>>> ipv6
>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
>>>>>> between
>>>> VPC
>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>
>>>>>> Please take a look at the existing security_group.py script
>>>>>> which implements RFC4890
>>>>>>
>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
>>>>>>
>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should always
>>>>>> be allowed.
>>>>>>
>>>>>>> * It is assumed that static routes are created on the
>>>> core/main
>>>>>> router by the admin or automated using some scripts/tools; for
>>>>>> this CloudStack will announce events with details of /64
>>>>>> networks and VR's public IPv6 address that can be consumed by a
>>>>>> rabbitmq/message bus
>>>> client
>>>>>> (for example), or a custom cron job or script as part of
>> orchestration.
>>>>>> (this wouldn't be necessary for dynamic routing bgp with
>>>>>> phase2)\\
>>>>>>
>>>>>> You would only need to announce the /48 or /56 allocated to the
>>>>>> VR, that's all. You don't need to inform the upstream router
>>>>>> about the /64 subnets created within that larger subnet.
>>>>>>
>>>>>>> * Guest Networking: With SLAAC, it's easy for CloudStack to
>>>>>> calculate allocate and use a /64 and determine the IPv6 address
>>>>>> of VR
>>>> nics
>>>>>> and guest VM nics
>>>>>>> * A user create an isolated network/VPC with an offering
>>>> that is
>>>>>> ipv6 enabled
>>>>>>> * A user can manage firewall for the IPv6 address/guest
>> nics;
>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>> * A users can run workloads in the guest VMs that listen
>> on
>>>>>> publically routable ipv6 addresses
>>>>>>> * Usage/billing etc continue to work, no change needed
>>>>>>>
>>>>>>> Network layout:
>>>>>>>
>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
>>>>>>> VLAN] ->
>>>>>> [guest VMs/nics]
>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>
>>>>>>> Thoughts, feedback?
>>>>>>
>>>>>> Looks doable!
>>>>>>
>>>>>> Side-note: It would be very cool if you could use parts of this
>>>>>> implementation to also route /48, /56, or /60 subnets to
>>>>>> individual VMs in Shared networks.
>>>>>>
>>>>>> Why? This allows for running Docker containers with native IPv6
>>>>>> inside the VM or running a (Open)VPN server inside a VM which
>>>>>> then assigns public IPv6 addresses to clients connected.
>>>>>>
>>>>>> Instead of routing the subnet to a VR we route the subnet to a
>>>>>> single instance in a shared network.
>>>>>>
>>>>>> If we could then also move these subnets between Instances
>>>>>> easily one can quickly migrate to a different instance while
>>>>>> keeping the same IPv6 subnet.
>>>>>>
>>>>>> Wido
>>>>>>
>>>>>>>
>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>
>>>>>>> * Created an isolated network and deployed a VM in my home
>> lab
>>>>>>> The VR running on KVM has following nics
>>>>>>> eth0 - guest network
>>>>>>> eth1 - link local
>>>>>>> eth2 - public network
>>>>>>>
>>>>>>> * I setup a custom openwrt router on a RPi4 to serve as a
>>>> toy-core
>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker and
>>>>>> I got
>>>> a
>>>>>> /48 allocated. My configuration looks like:
>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>
>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>> subnet/prefix
>>>>>> 0:
>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>> *
>>>>>>> *
>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
>>>>>>> net.ipv6.conf.all.autoconf = 1
>>>>>>>
>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
>>>>>>> configured the nics:
>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
>>>>>>> network without cleanup to have it
>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>
>>>>>>> *
>>>>>>> Next, I created a /64 network for the isolated guest network
>>>>>>> on eth0
>>>> of
>>>>>> VR using radvd:
>>>>>>>
>>>>>>> # cat /etc/radvd.conf
>>>>>>> interface eth0
>>>>>>> {
>>>>>>> AdvSendAdvert on;
>>>>>>> MinRtrAdvInterval 5;
>>>>>>> MaxRtrAdvInterval 15;
>>>>>>> prefix 2001:470:ed36:1::/64
>>>>>>> {
>>>>>>> AdvOnLink on;
>>>>>>> AdvAutonomous on;
>>>>>>> };
>>>>>>> };
>>>>>>> systemctl restart radvd
>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
>>>>>>> this
>>>>>> ...:1::/64 network
>>>>>>> * Finally I added a static route in toy core-router for the
>> new
>>>> /64
>>>>>> IPv6 range in the isolated network
>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>> eth2> dev
>>>>>> <local lan nic>
>>>>>>> *
>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
>>>>>>> for the
>>>>>> new /64 network
>>>>>>>
>>>>>>> And voila all done! I create a domain AAAA record that points
>>>>>>> to my
>>>>>> guest VM IPv6 address a test webserver on
>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>>>>>>>
>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
>>>>>>> after a
>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>
>>>>>>>
>>>>>>> Regards.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>>
>