You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Rohit Yadav <ro...@shapeblue.com> on 2021/08/11 12:26:32 UTC

Re: IPV6 in Isolated/VPC networks

Hi all,

Thanks for your feedback and ideas, I've gone ahead with discussing them with Alex and came up with a PoC/design which can be implemented in the following phases:

  *   Phase1: implement ipv6 support in isolated networks and VPC with static routing
  *   Phase2: discuss and implement support for dynamic routing (TBD)

For Phase1 here's the high-level proposal:

  *   IPv6 address management:
     *   At the zone level root-admin specifies a /64 public range that will be used for VRs, then they can add a /48, or /56 IPv6 range for guest networks (to be used by isolated networks and VPC tiers)
     *   On creation of any IPv6 enabled isolated network or VPC tier, from the /48 or /56 block a /64 network is allocated/used
     *   We assume SLAAC and autoconfiguration, no DHCPv6 in the zone (discuss: is privacy a concern, can privacy extensions rfc4941 of slaac be explored?)
  *   Network offerings: root-admin can create new network offerings (with VPC too) that specifies a network stack option:
     *   ipv4 only (default, for backward compatibility all networks/offerings post-upgrade migrate to this option)
     *   ipv4-and-ipv6
     *   ipv6-only (this can be phase 1.b)
     *   A new routing option: static (phase1), dynamic (phase2, with multiple sub-options such as ospf/bgp etc...)
  *   VR changes:
     *   VR gets its guest and public nics set to inet6 auto
     *   For each /64 allocated to guest network and VPC tiers, radvd is configured to do RA
     *   Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6 firewall rules (ACLs, ingress, egress) are implemented; ACLs between VPC tiers are managed/implemented by ipv6 firewall on VR
     *   It is assumed that static routes are created on the core/main router by the admin or automated using some scripts/tools; for this CloudStack will announce events with details of /64 networks and VR's public IPv6 address that can be consumed by a rabbitmq/message bus client (for example), or a custom cron job or script as part of orchestration. (this wouldn't be necessary for dynamic routing bgp with phase2)
  *   Guest Networking: With SLAAC, it's easy for CloudStack to calculate allocate and use a /64 and determine the IPv6 address of VR nics and guest VM nics
     *   A user create an isolated network/VPC with an offering that is ipv6 enabled
     *   A user can manage firewall for the IPv6 address/guest nics; there'll be no port forward and LB feature though for IPv6
     *   A users can run workloads in the guest VMs that listen on publically routable ipv6 addresses
     *   Usage/billing etc continue to work, no change needed

Network layout:

[core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] -> [guest VMs/nics]
*core/ISP router needs static routes to be added (manually or automated), assumes a /48 or /56 configured for the zone

Thoughts, feedback?

Proof-of-concept commentary: here's what I did to test the idea:

  *   Created an isolated network and deployed a VM in my home lab
The VR running on KVM has following nics
eth0 - guest network
eth1 - link local
eth2 - public network

  *   I setup a custom openwrt router on a RPi4 to serve as a toy-core router where I create a wan6 IPv6 tunnel using tunnel broker and I got a /48 allocated. My configuration looks like:
/48 - 2001:470:ed36::/48 (allocated by tunnel broker)
/64 - 2001:470:36:3e2::/64 (default allocated by)

I create a LAN ipv6 (public network for CloudStack VR): at subnet/prefix 0:
LAN IPv6 address: 2001:470:ed36:0::1/64
Address mode: SLAAC+stateless DHCP (no dhcpv6)
  *
  *
In the isolated VR, I enabled ipv6 as:
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.all.accept_ra = 1
net.ipv6.conf.all.accept_redirects = 1
net.ipv6.conf.all.autoconf = 1

Set up a IPv6 nameserver/dns in /etc/resolve.conf
And configured the nics:
echo iface eth0 inet6 auto >> /etc/network/interfaces
echo iface eth2 inet6 auto >> /etc/network/interfaces
/etc/init.d/networking restart
Next, restart ACS isolated network without cleanup to have it reconfigure IPv4 nics, firewall, NAT etc

  *
Next, I created a /64 network for the isolated guest network on eth0 of VR using radvd:

# cat /etc/radvd.conf
interface eth0
{
    AdvSendAdvert on;
    MinRtrAdvInterval 5;
    MaxRtrAdvInterval 15;
    prefix 2001:470:ed36:1::/64
    {
        AdvOnLink on;
        AdvAutonomous on;
    };
};
systemctl restart radvd
All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this ...:1::/64 network
  *   Finally I added a static route in toy core-router for the new /64 IPv6 range in the isolated network
2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev <local lan nic>
  *
... and I enabled firewall rules to allow any traffic to pass for the new /64 network

And voila all done! I create a domain AAAA record that points to my guest VM IPv6 address a test webserver on
http://ipv6-isolated-ntwk-demo.yadav.cloud/

(Note: I'll get rid of the tunnel and request a new /48 block after a few days, sharing this solely for testing purposes)


Regards.

________________________________
From: Wido den Hollander <wi...@widodh.nl>
Sent: Tuesday, July 20, 2021 12:46
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks



Op 19-07-2021 om 20:38 schreef Kristaps Cudars:
> Hi Wido,
>
> I assume that flouting ip will not work grate with ingress/egress acl on VR.
>
>  From regular ACS user perspective:
> I have Instance with dualstack its running web app on 443.
> I want to swap instances for whatever reason.
> In case of IPv4 change d-nat rule.
> In case of IPv6 if flouting IP was not created upfront he will need to change dns entry that usually has 24h ttl. Inconvenience degradation in experience.
>

Yes, but, keep in mind that the IP you are using can also be terminated
on the VR where HAProxy proxies request to the backend VM (could even be
v4!)

I'm not against DHCPv6, but I have seen many issues with implementing
it. Therefor I always stick to SLAAC.

>  From ACS admin perspective:
> I don’t want to have these tickets in helpdesk.
> You needed to create another flouting IP that it would be seamless- will not work as answer.
>

I understand that as well.

Wido

>
> On 2021/07/19 09:05:54, Wido den Hollander <wi...@widodh.nl> wrote:
>>
>>
>> Op 16-07-2021 om 21:46 schreef Kristaps Cudars:
>>> Hi Wido,
>>>
>>> Your proposal is to sacrifice ability to reassign IPv6 to instance, have internal domain prefix, and list/db in ACS what IPv6 has been assigned to what instance and go with RA and SLAAC. For route signaling to switch use BGP/OSPFv3 or manual pre-creation.
>>>
>>
>> You can still list the IPs which have been assigned. You'll know exactly
>> what IPv6 address a VM has because of the prefix + MAC. Privacy
>> Extensions need to be disabled in the VM.
>>
>> This already works in CloudStack in Shared Networks in this way.
>>
>> Using secondary IPs you can always have 'floating' IPv6 addressess.
>>
>> Wido
>>
>>> Option with RA and managed flag that DHCPv6 is in use to support preset information and ability to create route information from ACS is not an option as DHCPv6 its failing?
>>>
>>>
>>> On 2021/07/16 15:17:42, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>
>>>>
>>>> Op 16-07-2021 om 16:42 schreef Hean Seng:
>>>>> Hi Wido,
>>>>>
>>>>> In current setup,  each Cloudstack have own VR, so in this new  IPv6 subnet
>>>>> allocation , each VR (which have Frr) will need to have peering with ISP
>>>>> router (and either BGP or Static Route) , and there is 1000 Acocunts,  it
>>>>> will 1000 BGP session with ISP router ,  Am I right for this ? or I
>>>>> understand wrong .
>>>>>
>>>>
>>>> Yes, that is correct. A /56 would also be sufficient or a /60 which is
>>>> enough to allocate a few /64 subnets.
>>>>
>>>> 1000 BGP connections isn't really a problem for a proper router at the
>>>> ISP. OSPF(v3) would be better, but as I said that's poorly supported.
>>>>
>>>> The ISP could also install 1000 static routes, but that means that the
>>>> ISP's router needs to have those configured.
>>>>
>>>> http://docs.frrouting.org/en/latest/ospf6d.html
>>>> (While looking up this URL I see that Frr recently put in a lot of work
>>>> in OSPFv3, seems better now)
>>>>
>>>>> I understand IPv6 is different then IPv4, and in IPv6 it suppose each
>>>>> devices have own IP. It just how to realize in easy way.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
 

> On Fri, Jul 16, 2021 at 8:17 PM Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> Op 16-07-2021 om 05:54 schreef Hean Seng:
>>>>>>> Hi Wido,
>>>>>>>
>>>>>>> My initial thought is not like this,  it is the /48 at ISP router, and
>>>>>> /64
>>>>>>> subnet assign to AdvanceZoneVR,   AdvanceZoneVR responsible is
>>>>>>> distribule IPv6 ip (from the assigned /64 sunet) to VM,  and not routing
>>>>>>> the traffic,   in the VM that get the IPv6 IP will default route to ISP
>>>>>>> router as gw.   It can may be a bridge over via Advancezone-VR.
>>>>>>>
>>>>>>
>>>>>> How would you bridge this? That sounds like NAT?
>>>>>>
>>>>>> IPv6 is meant to be routed. Not to be translated or bridged in any way.
>>>>>>
>>>>>> The way a made the drawing is exactly how IPv6 should work in a VPC
>>>>>> environment.
>>>>>>
>>>>>> Traffic flows through the VR where it can do firewalling of the traffic.
>>>>>>
>>>>>>> However, If do as the way described in the drawing, then i suppose will
>>>>>> be
>>>>>>> another kind of virtual router going to introduce , to get hold the /48
>>>>>> in
>>>>>>> this virtual router right ?
>>>>>>>
>>>>>>
>>>>>> It can be the same VR. But keep in mind that IPv6 != IPv4.
>>>>>>
>>>>>> The VR will get Frr as a new daemon which can talk BGP with the upper
>>>>>> network to route traffic.
>>>>>>
>>>>>>> After this,  The Advance Zone, NAT's  VR will peer with this new IPv6 VR
>>>>>>> for getting the IPv6 /64 prefix ?
>>>>>>>
>>>>>>
>>>>>> IPv4 will be behind NAT, but IPv6 will not be behind NAT.
>>>>>>
>>>>>>> If do in this way, then I guess  you just only need Static route, with
>>>>>>> peering ip both end  as one /48 can have a lot of /64 on it.  And
>>>>>> hardware
>>>>>>> budgeting for new IPv6-VR will become very important, as all traffic will
>>>>>>> need to pass over it .
>>>>>>>
>>>>>>
>>>>>> Routing or NAT is the same for the VR. You don't need a very beefy VR
>>>>>> for this.
>>>>>>
>>>>>>> It will be like
>>>>>>>
>>>>>>> ISP Router  ------ >  (new IPV6-VR ) ---- > AdvanceZone-VR ----> VM
>>>>>>>
>>>>>>> Relationship of (new IPv6 VR) and AdvanceZone-VR , may be considering on
>>>>>>> OSPF instead of  BGP , otherwise few thousand of AdvanceZone-VR wil have
>>>>>>> few thousand of BGP session. on new-IPv6-VR
>>>>>>>
>>>>>>> Also, I suppose we cannot do ISP router. -->. Advancezone VR direct,   ,
>>>>>>> otherwise ISP router will be full of /64 prefix route either on BGP( Many
>>>>>>> BGP Session) , or  Many Static route .   If few thousand account, ti will
>>>>>>> be few thousand of BGP session with ISP router or few thousand static
>>>>>> route
>>>>>>> which  is not possible .
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 15, 2021 at 10:47 PM Wido den Hollander <wi...@widodh.nl>
>>>>>> wrote:
>>>>>>>
>>>>>>>> But you still need routing. See the attached PNG (and draw.io XML).
>>>>>>>>
>>>>>>>> You need to route the /48 subnet TO the VR which can then route it to
>>>>>>>> the Virtual Networks behind the VR.
>>>>>>>>
>>>>>>>> There is no other way then routing with either BGP or a Static route.
>>>>>>>>
>>>>>>>> Wido
>>>>>>>>
>>>>>>>> Op 15-07-2021 om 12:39 schreef Hean Seng:
>>>>>>>>> Or explain like this :
>>>>>>>>>
>>>>>>>>> 1) Cloudstack generate list of /64 subnet from /48 that Network admin
>>>>>>>>> assigned to Cloudstack
>>>>>>>>> 2) Cloudsack allocated the subnet (that generated from step1) to
>>>>>> Virtual
>>>>>>>>> Router, one Virtual Router have one subniet /64
>>>>>>>>> 3) Virtual Router allocate single IPv6 (within the range of /64
>>>>>>>>> allocated to VR)  to VM
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng <heanseng@gmail.com
>>>>>>>>> <ma...@gmail.com>> wrote:
>>>>>>>>>
>>>>>>>>>         Hi Wido,
>>>>>>>>>
>>>>>>>>>         I think the /48 is at physical router as gateway , and subnet of
>>>>>> /64
>>>>>>>>>         at VR of Cloudstack.   Cloudstack only keep which /48 prefix and
>>>>>>>>>         vlan information of this /48 to be later split the  /64. to VR.
>>>>>>>>>
>>>>>>>>>         And the instances is getting singe IPv6 of /64  IP.   The VR is
>>>>>>>>>         getting /64.  The default gateway shall goes to /48 of physical
>>>>>>>>>         router ip .   In this case ,does not need any BGP router .
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         Similar concept as IPv4 :
>>>>>>>>>
>>>>>>>>>         /48 subnet of IPv6 is equivalent to current /24 subnet of IPv4
>>>>>> that
>>>>>>>>>         created in Network.
>>>>>>>>>         and /64  of IPv6 is equivalent to single IP of IPv4 assign to VM.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander <
>>>>>> wido@widodh.nl
>>>>>>>>>         <ma...@widodh.nl>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>             Op 14-07-2021 om 16:44 schreef Hean Seng:
>>>>>>>>>              > Hi
>>>>>>>>>              >
>>>>>>>>>              > I replied in another thread, i think do not need implement
>>>>>>>>>             BGP or OSPF,
>>>>>>>>>              > that would be complicated .
>>>>>>>>>              >
>>>>>>>>>              > We only need assign  IPv6 's /64 prefix to Virtual Router
>>>>>>>>>             (VR) in NAT
>>>>>>>>>              > zone, and the VR responsible to deliver single IPv6 to VM
>>>>>> via
>>>>>>>>>             DHCP6.
>>>>>>>>>              >
>>>>>>>>>              > In VR, you need to have Default IPv6 route to  Physical
>>>>>>>>>             Router's /48. IP
>>>>>>>>>              > as IPv6 Gateway.  Thens should be done .
>>>>>>>>>              >
>>>>>>>>>              > Example :
>>>>>>>>>              > Physical Router Interface
>>>>>>>>>              >   IPv6 IP : 2000:aaaa::1/48
>>>>>>>>>              >
>>>>>>>>>              > Cloudstack  virtual router : 2000:aaaa:200:201::1/64 with
>>>>>>>>>             default ipv6
>>>>>>>>>              > route to router ip 2000:aaaa::1
>>>>>>>>>              > and Clodustack Virtual router dhcp allocate IP to VM , and
>>>>>>>>>             VM will have
>>>>>>>>>              > default route to VR. IPv6 2000:aaaa:200:201::1
>>>>>>>>>              >
>>>>>>>>>              > So in cloudstack need to allow  user to enter ,  IPv6
>>>>>>>>>             gwateway , and
>>>>>>>>>              > the  /48 Ipv6 prefix , then it will self allocate the /64
>>>>>> ip
>>>>>>>>>             to the VR ,
>>>>>>>>>              > and maintain make sure not ovelap allocation
>>>>>>>>>              >
>>>>>>>>>              >
>>>>>>>>>
>>>>>>>>>             But NAT is truly not the solution with IPv6. IPv6 is supposed
>>>>>> to
>>>>>>>> be
>>>>>>>>>             routable. In addition you should avoid DHCPv6 as much as
>>>>>>>>>             possible as
>>>>>>>>>             that's not really the intended use-case for address allocation
>>>>>>>>>             with IPv6.
>>>>>>>>>
>>>>>>>>>             In order to route an /48 IPv6 subnet to the VR you have a few
>>>>>>>>>             possibilities:
>>>>>>>>>
>>>>>>>>>             - Static route from the upperlying routers which are outside
>>>>>> of
>>>>>>>>>             CloudStack
>>>>>>>>>             - BGP
>>>>>>>>>             - OSPFv3 (broken in most cases!)
>>>>>>>>>             - DHCPv6 Prefix Delegation
>>>>>>>>>
>>>>>>>>>             BGP and/or Static routes are still the best bet here.
>>>>>>>>>
>>>>>>>>>             So what you do is that you tell CloudStack that you will route
>>>>>>>>>             2001:db8::/48 to the VR, the VR can then use that to split it
>>>>>> up
>>>>>>>>>             into
>>>>>>>>>             multiple /64 subnets going towards the instances:
>>>>>>>>>
>>>>>>>>>             - 2001:db8::/64
>>>>>>>>>             - 2001:db8:1::/64
>>>>>>>>>             - 2001:db8:2::/64
>>>>>>>>>             ...
>>>>>>>>>             - 2001:db8:f::/64
>>>>>>>>>
>>>>>>>>>             And go on.
>>>>>>>>>
>>>>>>>>>             In case of BGP you indeed have to tell the VR a few things:
>>>>>>>>>
>>>>>>>>>             - It's own AS number
>>>>>>>>>             - The peer's address(es)
>>>>>>>>>
>>>>>>>>>             With FRR you can simply say:
>>>>>>>>>
>>>>>>>>>             neighbor 2001:db8:4fa::179 remote-as external
>>>>>>>>>
>>>>>>>>>             The /48 you need to have at the VR anyway in case of either a
>>>>>>>>>             static
>>>>>>>>>             route or BGP.
>>>>>>>>>
>>>>>>>>>             We just need to add a NullRoute on the VR for that /48 so that
>>>>>>>>>             traffic
>>>>>>>>>             will not be routed to the upper gateway in case of the VR
>>>>>> can't
>>>>>>>>>             find a
>>>>>>>>>             route.
>>>>>>>>>
>>>>>>>>>             Wido
>>>>>>>>>
>>>>>>>>>              >
>>>>>>>>>              >
>>>>>>>>>              >
>>>>>>>>>              >
>>>>>>>>>              >
>>>>>>>>>              > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli
>>>>>>>>>              > <Alex.Mattioli@shapeblue.com
>>>>>>>>>             <ma...@shapeblue.com>
>>>>>>>>>             <mailto:Alex.Mattioli@shapeblue.com
>>>>>>>>>             <ma...@shapeblue.com>>> wrote:
>>>>>>>>>              >
>>>>>>>>>              >     Hi Wido,
>>>>>>>>>              >     That's pretty much in line with our thoughts, thanks
>>>>>> for
>>>>>>>>>             the input.
>>>>>>>>>              >     I believe we agree on the following points then:
>>>>>>>>>              >
>>>>>>>>>              >     - FRR with BGP (no OSPF)
>>>>>>>>>              >     - Route /48 (or/56) down to the VR
>>>>>>>>>              >     - /64 per network
>>>>>>>>>              >     - SLACC for IP addressing
>>>>>>>>>              >
>>>>>>>>>              >     I believe the next big question is then "on which level
>>>>>>>>>             of ACS do we
>>>>>>>>>              >     manage AS numbers?".  I see two options:
>>>>>>>>>              >     1) Private AS number on a per-zone basis
>>>>>>>>>              >     2) Root Admin assigned AS number on a domain/account
>>>>>> basis
>>>>>>>>>              >     3) End-user driven AS number on a per network basis
>>>>>> (for
>>>>>>>>>             bring your
>>>>>>>>>              >     own AS and IP scenario)
>>>>>>>>>              >
>>>>>>>>>              >     Thoughts?
>>>>>>>>>              >
>>>>>>>>>              >     Cheers
>>>>>>>>>              >     Alex
>>>>>>>>>              >
>>>>>>>>>              >
>>>>>>>>>              >
>>>>>>>>>              >
>>>>>>>>>              >     -----Original Message-----
>>>>>>>>>              >     From: Wido den Hollander <wido@widodh.nl
>>>>>>>>>             <ma...@widodh.nl> <mailto:wido@widodh.nl
>>>>>>>>>             <ma...@widodh.nl>>>
>>>>>>>>>              >     Sent: 13 July 2021 15:08
>>>>>>>>>              >     To: dev@cloudstack.apache.org
>>>>>>>>>             <ma...@cloudstack.apache.org>
>>>>>>>>>             <mailto:dev@cloudstack.apache.org
>>>>>>>>>             <ma...@cloudstack.apache.org>>;
>>>>>>>>>              >     Alex Mattioli <Alex.Mattioli@shapeblue.com
>>>>>>>>>             <ma...@shapeblue.com>
>>>>>>>>>              >     <mailto:Alex.Mattioli@shapeblue.com
>>>>>>>>>             <ma...@shapeblue.com>>>
>>>>>>>>>              >     Cc: Wei Zhou <Wei.Zhou@shapeblue.com
>>>>>>>>>             <ma...@shapeblue.com>
>>>>>>>>>              >     <mailto:Wei.Zhou@shapeblue.com
>>>>>>>>>             <ma...@shapeblue.com>>>; Rohit Yadav
>>>>>>>>>              >     <rohit.yadav@shapeblue.com
>>>>>>>>>             <ma...@shapeblue.com>
>>>>>>>>>             <mailto:rohit.yadav@shapeblue.com
>>>>>>>>>             <ma...@shapeblue.com>>>;
>>>>>>>>>              >     Gabriel Beims Bräscher <gabriel@pcextreme.nl
>>>>>>>>>             <ma...@pcextreme.nl>
>>>>>>>>>              >     <mailto:gabriel@pcextreme.nl <mailto:
>>>>>> gabriel@pcextreme.nl
>>>>>>>>>>>
>>>>>>>>>              >     Subject: Re: IPV6 in Isolated/VPC networks
>>>>>>>>>              >
>>>>>>>>>              >
>>>>>>>>>              >
>>>>>>>>>              >     On 7/7/21 1:16 PM, Alex Mattioli wrote:
>>>>>>>>>              >      > Hi all,
>>>>>>>>>              >      > @Wei Zhou<mailto:Wei.Zhou@shapeblue.com
>>>>>>>>>             <ma...@shapeblue.com>
>>>>>>>>>              >     <mailto:Wei.Zhou@shapeblue.com
>>>>>>>>>             <ma...@shapeblue.com>>> @Rohit
>>>>>>>>>              >     Yadav<mailto:rohit.yadav@shapeblue.com
>>>>>>>>>             <ma...@shapeblue.com>
>>>>>>>>>              >     <mailto:rohit.yadav@shapeblue.com
>>>>>>>>>             <ma...@shapeblue.com>>> and myself are
>>>>>>>>>             investigating how
>>>>>>>>>              >     to enable IPV6 support on Isolated and VPC networks and
>>>>>>>>>             would like
>>>>>>>>>              >     your input on it.
>>>>>>>>>              >      > At the moment we are looking at implementing FRR
>>>>>> with
>>>>>>>>>             BGP (and
>>>>>>>>>              >     possibly OSPF) on the ACS VR.
>>>>>>>>>              >      >
>>>>>>>>>              >      > We are looking for requirements, recommendations,
>>>>>>>>>             ideas, rants,
>>>>>>>>>              >     etc...etc...
>>>>>>>>>              >      >
>>>>>>>>>              >
>>>>>>>>>              >     Ok! Here we go.
>>>>>>>>>              >
>>>>>>>>>              >     I think that you mean that the VR will actually route
>>>>>> the
>>>>>>>>>             IPv6
>>>>>>>>>              >     traffic and for that you need to have a way of getting
>>>>>> a
>>>>>>>>>             subnet
>>>>>>>>>              >     routed to the VR.
>>>>>>>>>              >
>>>>>>>>>              >     BGP is probably you best bet here. Although OSPFv3
>>>>>>>>>             technically
>>>>>>>>>              >     supports this it is very badly implemented in Frr for
>>>>>>>>>             example.
>>>>>>>>>              >
>>>>>>>>>              >     Now FRR is a very good router and one of the fancy
>>>>>>>>>             features it
>>>>>>>>>              >     supports is BGP Unnumered. This allows for auto
>>>>>>>>>             configuration of BGP
>>>>>>>>>              >     over a L2 network when both sides are sending Router
>>>>>>>>>             Advertisements.
>>>>>>>>>              >     This is very easy for flexible BGP configurations where
>>>>>>>>>             both sides
>>>>>>>>>              >     have dynamic IPs.
>>>>>>>>>              >
>>>>>>>>>              >     What you want to do is that you get a /56, /48 or
>>>>>>>>>             something which is
>>>>>>>>>              >      >/64 bits routed to the VR.
>>>>>>>>>              >
>>>>>>>>>              >     Now you can sub-segment this into separate /64 subnets.
>>>>>>>>>             You don't
>>>>>>>>>              >     want to go smaller then a /64 is that prevents you from
>>>>>>>>>             using SLAAC
>>>>>>>>>              >     for IPv6 address configuration. This is how it works
>>>>>> for
>>>>>>>>>             Shared
>>>>>>>>>              >     Networks now in Basic and Advanced Zones.
>>>>>>>>>              >
>>>>>>>>>              >     FRR can now also send out the Router Advertisements on
>>>>>>>>>             the downlinks
>>>>>>>>>              >     sending out:
>>>>>>>>>              >
>>>>>>>>>              >     - DNS servers
>>>>>>>>>              >     - DNS domain
>>>>>>>>>              >     - Prefix (/64) to be used
>>>>>>>>>              >
>>>>>>>>>              >     There is no need for DHCPv6. You can calculate the IPv6
>>>>>>>>>             address the
>>>>>>>>>              >     VM will obtain by using the MAC and the prefix.
>>>>>>>>>              >
>>>>>>>>>              >     So in short:
>>>>>>>>>              >
>>>>>>>>>              >     - Using BGP you routed a /48 to the VR
>>>>>>>>>              >     - Now you split this into /64 subnets towards the
>>>>>>>>>             isolated networks
>>>>>>>>>              >
>>>>>>>>>              >     Wido
>>>>>>>>>              >
>>>>>>>>>              >      > Alex Mattioli
>>>>>>>>>              >      >
>>>>>>>>>              >      >
>>>>>>>>>              >      >
>>>>>>>>>              >      >
>>>>>>>>>              >
>>>>>>>>>              >
>>>>>>>>>              >
>>>>>>>>>              > --
>>>>>>>>>              > Regards,
>>>>>>>>>              > Hean Seng
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>         --
>>>>>>>>>         Regards,
>>>>>>>>>         Hean Seng
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Hean Seng
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>

RE: IPV6 in Isolated/VPC networks

Posted by Alex Mattioli <Al...@shapeblue.com>.
Hi Kristaps,

The operator will need to add two ranges to ACS.

1) /64 for the physical network (just as we do with public IPV4 at the moment), this IPs will be used for the outside interface of the VRs
2) 1 or more /64 or larger (ideally /56s I'd say), which ACS will subnet into /64s and assign them to the inside interfaces of the VRs.

Cheers
Alex

 


-----Original Message-----
From: Kristaps Cudars <kr...@gmail.com> 
Sent: 12 August 2021 10:37
To: dev@cloudstack.apache.org
Cc: users@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks

Hi Rohit,

I think its excellent and satisfies all involved sides.

We will be in front seats bringing this to production.


Suggestion:

This

*   At the zone level root-admin specifies a /64 public range that will be
used for VRs, then

they can add a /48, or /56 IPv6 range for guest networks (to be used by isolated networks and VPC tiers)


To

*   At the zone level root-admin specifies a /64 public range that will be
used for VRs, then

they can add one or more ranges equal or larger than /64 IPv6 and ACS will subnet it in /64s

for Isolated networks or VPC tiers.


Way:

RIPE NCC in case of PI (Provider Independent) range is granting /48 and its important to have

possibility of assigning several subnets, for example /58 to be used for internal /64 allocations.

If you want to extend your /48 you will need to have good explanation why you want it.
RIPE NCC: Organisations requesting an IPv6 PI assignment larger than a /48 need to provide

additional documentation to justify the larger assignment size.

The RIPE NCC will evaluate the PI assignment request and assign an independent IPv6 prefix

to the End User organization directly.

On Wed, 11 Aug 2021 at 15:26, Rohit Yadav <ro...@shapeblue.com> wrote:

> Hi all,
>
> Thanks for your feedback and ideas, I've gone ahead with discussing 
> them with Alex and came up with a PoC/design which can be implemented 
> in the following phases:
>
>   *   Phase1: implement ipv6 support in isolated networks and VPC with
> static routing
>   *   Phase2: discuss and implement support for dynamic routing (TBD)
>
> For Phase1 here's the high-level proposal:
>
>   *   IPv6 address management:
>      *   At the zone level root-admin specifies a /64 public range that
> will be used for VRs, then they can add a /48, or /56 IPv6 range for 
> guest networks (to be used by isolated networks and VPC tiers)
>      *   On creation of any IPv6 enabled isolated network or VPC tier,
> from the /48 or /56 block a /64 network is allocated/used
>      *   We assume SLAAC and autoconfiguration, no DHCPv6 in the zone
> (discuss: is privacy a concern, can privacy extensions rfc4941 of 
> slaac be
> explored?)
>   *   Network offerings: root-admin can create new network offerings (with
> VPC too) that specifies a network stack option:
>      *   ipv4 only (default, for backward compatibility all
> networks/offerings post-upgrade migrate to this option)
>      *   ipv4-and-ipv6
>      *   ipv6-only (this can be phase 1.b)
>      *   A new routing option: static (phase1), dynamic (phase2, with
> multiple sub-options such as ospf/bgp etc...)
>   *   VR changes:
>      *   VR gets its guest and public nics set to inet6 auto
>      *   For each /64 allocated to guest network and VPC tiers, radvd is
> configured to do RA
>      *   Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6
> firewall rules (ACLs, ingress, egress) are implemented; ACLs between 
> VPC tiers are managed/implemented by ipv6 firewall on VR
>      *   It is assumed that static routes are created on the core/main
> router by the admin or automated using some scripts/tools; for this 
> CloudStack will announce events with details of /64 networks and VR's 
> public IPv6 address that can be consumed by a rabbitmq/message bus 
> client (for example), or a custom cron job or script as part of orchestration.
> (this wouldn't be necessary for dynamic routing bgp with phase2)
>   *   Guest Networking: With SLAAC, it's easy for CloudStack to calculate
> allocate and use a /64 and determine the IPv6 address of VR nics and 
> guest VM nics
>      *   A user create an isolated network/VPC with an offering that is
> ipv6 enabled
>      *   A user can manage firewall for the IPv6 address/guest nics;
> there'll be no port forward and LB feature though for IPv6
>      *   A users can run workloads in the guest VMs that listen on
> publically routable ipv6 addresses
>      *   Usage/billing etc continue to work, no change needed
>
> Network layout:
>
> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] -> 
> [guest VMs/nics] *core/ISP router needs static routes to be added 
> (manually or automated), assumes a /48 or /56 configured for the zone
>
> Thoughts, feedback?
>
> Proof-of-concept commentary: here's what I did to test the idea:
>
>   *   Created an isolated network and deployed a VM in my home lab
> The VR running on KVM has following nics
> eth0 - guest network
> eth1 - link local
> eth2 - public network
>
>   *   I setup a custom openwrt router on a RPi4 to serve as a toy-core
> router where I create a wan6 IPv6 tunnel using tunnel broker and I got 
> a
> /48 allocated. My configuration looks like:
> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> /64 - 2001:470:36:3e2::/64 (default allocated by)
>
> I create a LAN ipv6 (public network for CloudStack VR): at subnet/prefix 0:
> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode: SLAAC+stateless 
> DHCP (no dhcpv6)
>   *
>   *
> In the isolated VR, I enabled ipv6 as:
> net.ipv6.conf.all.disable_ipv6 = 0
> net.ipv6.conf.all.forwarding = 1
> net.ipv6.conf.all.accept_ra = 1
> net.ipv6.conf.all.accept_redirects = 1 net.ipv6.conf.all.autoconf = 1
>
> Set up a IPv6 nameserver/dns in /etc/resolve.conf And configured the 
> nics:
> echo iface eth0 inet6 auto >> /etc/network/interfaces echo iface eth2 
> inet6 auto >> /etc/network/interfaces /etc/init.d/networking restart 
> Next, restart ACS isolated network without cleanup to have it 
> reconfigure
> IPv4 nics, firewall, NAT etc
>
>   *
> Next, I created a /64 network for the isolated guest network on eth0 
> of VR using radvd:
>
> # cat /etc/radvd.conf
> interface eth0
> {
>     AdvSendAdvert on;
>     MinRtrAdvInterval 5;
>     MaxRtrAdvInterval 15;
>     prefix 2001:470:ed36:1::/64
>     {
>         AdvOnLink on;
>         AdvAutonomous on;
>     };
> };
> systemctl restart radvd
> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this
> ...:1::/64 network
>   *   Finally I added a static route in toy core-router for the new /64
> IPv6 range in the isolated network
> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev 
> <local lan nic>
>   *
> ... and I enabled firewall rules to allow any traffic to pass for the 
> new
> /64 network
>
> And voila all done! I create a domain AAAA record that points to my 
> guest VM IPv6 address a test webserver on 
> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>
> (Note: I'll get rid of the tunnel and request a new /48 block after a 
> few days, sharing this solely for testing purposes)
>
>
> Regards.
>
> ________________________________
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: Tuesday, July 20, 2021 12:46
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
>
>
> Op 19-07-2021 om 20:38 schreef Kristaps Cudars:
> > Hi Wido,
> >
> > I assume that flouting ip will not work grate with ingress/egress 
> > acl on
> VR.
> >
> >  From regular ACS user perspective:
> > I have Instance with dualstack its running web app on 443.
> > I want to swap instances for whatever reason.
> > In case of IPv4 change d-nat rule.
> > In case of IPv6 if flouting IP was not created upfront he will need 
> > to
> change dns entry that usually has 24h ttl. Inconvenience degradation 
> in experience.
> >
>
> Yes, but, keep in mind that the IP you are using can also be 
> terminated on the VR where HAProxy proxies request to the backend VM 
> (could even be
> v4!)
>
> I'm not against DHCPv6, but I have seen many issues with implementing 
> it. Therefor I always stick to SLAAC.
>
> >  From ACS admin perspective:
> > I don’t want to have these tickets in helpdesk.
> > You needed to create another flouting IP that it would be seamless- 
> > will
> not work as answer.
> >
>
> I understand that as well.
>
> Wido
>
> >
> > On 2021/07/19 09:05:54, Wido den Hollander <wi...@widodh.nl> wrote:
> >>
> >>
> >> Op 16-07-2021 om 21:46 schreef Kristaps Cudars:
> >>> Hi Wido,
> >>>
> >>> Your proposal is to sacrifice ability to reassign IPv6 to 
> >>> instance,
> have internal domain prefix, and list/db in ACS what IPv6 has been 
> assigned to what instance and go with RA and SLAAC. For route 
> signaling to switch use BGP/OSPFv3 or manual pre-creation.
> >>>
> >>
> >> You can still list the IPs which have been assigned. You'll know 
> >> exactly what IPv6 address a VM has because of the prefix + MAC. 
> >> Privacy Extensions need to be disabled in the VM.
> >>
> >> This already works in CloudStack in Shared Networks in this way.
> >>
> >> Using secondary IPs you can always have 'floating' IPv6 addressess.
> >>
> >> Wido
> >>
> >>> Option with RA and managed flag that DHCPv6 is in use to support
> preset information and ability to create route information from ACS is 
> not an option as DHCPv6 its failing?
> >>>
> >>>
> >>> On 2021/07/16 15:17:42, Wido den Hollander <wi...@widodh.nl> wrote:
> >>>>
> >>>>
> >>>> Op 16-07-2021 om 16:42 schreef Hean Seng:
> >>>>> Hi Wido,
> >>>>>
> >>>>> In current setup,  each Cloudstack have own VR, so in this new  
> >>>>> IPv6
> subnet
> >>>>> allocation , each VR (which have Frr) will need to have peering 
> >>>>> with
> ISP
> >>>>> router (and either BGP or Static Route) , and there is 1000
> Acocunts,  it
> >>>>> will 1000 BGP session with ISP router ,  Am I right for this ? 
> >>>>> or I understand wrong .
> >>>>>
> >>>>
> >>>> Yes, that is correct. A /56 would also be sufficient or a /60 
> >>>> which is enough to allocate a few /64 subnets.
> >>>>
> >>>> 1000 BGP connections isn't really a problem for a proper router 
> >>>> at the ISP. OSPF(v3) would be better, but as I said that's poorly supported.
> >>>>
> >>>> The ISP could also install 1000 static routes, but that means 
> >>>> that the ISP's router needs to have those configured.
> >>>>
> >>>> http://docs.frrouting.org/en/latest/ospf6d.html
> >>>> (While looking up this URL I see that Frr recently put in a lot 
> >>>> of
> work
> >>>> in OSPFv3, seems better now)
> >>>>
> >>>>> I understand IPv6 is different then IPv4, and in IPv6 it suppose 
> >>>>> each devices have own IP. It just how to realize in easy way.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
>
>
> > On Fri, Jul 16, 2021 at 8:17 PM Wido den Hollander <wi...@widodh.nl>
> wrote:
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> Op 16-07-2021 om 05:54 schreef Hean Seng:
> >>>>>>> Hi Wido,
> >>>>>>>
> >>>>>>> My initial thought is not like this,  it is the /48 at ISP 
> >>>>>>> router,
> and
> >>>>>> /64
> >>>>>>> subnet assign to AdvanceZoneVR,   AdvanceZoneVR responsible is
> >>>>>>> distribule IPv6 ip (from the assigned /64 sunet) to VM,  and 
> >>>>>>> not
> routing
> >>>>>>> the traffic,   in the VM that get the IPv6 IP will default route
> to ISP
> >>>>>>> router as gw.   It can may be a bridge over via Advancezone-VR.
> >>>>>>>
> >>>>>>
> >>>>>> How would you bridge this? That sounds like NAT?
> >>>>>>
> >>>>>> IPv6 is meant to be routed. Not to be translated or bridged in 
> >>>>>> any
> way.
> >>>>>>
> >>>>>> The way a made the drawing is exactly how IPv6 should work in a 
> >>>>>> VPC environment.
> >>>>>>
> >>>>>> Traffic flows through the VR where it can do firewalling of the
> traffic.
> >>>>>>
> >>>>>>> However, If do as the way described in the drawing, then i 
> >>>>>>> suppose
> will
> >>>>>> be
> >>>>>>> another kind of virtual router going to introduce , to get 
> >>>>>>> hold
> the /48
> >>>>>> in
> >>>>>>> this virtual router right ?
> >>>>>>>
> >>>>>>
> >>>>>> It can be the same VR. But keep in mind that IPv6 != IPv4.
> >>>>>>
> >>>>>> The VR will get Frr as a new daemon which can talk BGP with the
> upper
> >>>>>> network to route traffic.
> >>>>>>
> >>>>>>> After this,  The Advance Zone, NAT's  VR will peer with this 
> >>>>>>> new
> IPv6 VR
> >>>>>>> for getting the IPv6 /64 prefix ?
> >>>>>>>
> >>>>>>
> >>>>>> IPv4 will be behind NAT, but IPv6 will not be behind NAT.
> >>>>>>
> >>>>>>> If do in this way, then I guess  you just only need Static 
> >>>>>>> route,
> with
> >>>>>>> peering ip both end  as one /48 can have a lot of /64 on it.  
> >>>>>>> And
> >>>>>> hardware
> >>>>>>> budgeting for new IPv6-VR will become very important, as all
> traffic will
> >>>>>>> need to pass over it .
> >>>>>>>
> >>>>>>
> >>>>>> Routing or NAT is the same for the VR. You don't need a very 
> >>>>>> beefy
> VR
> >>>>>> for this.
> >>>>>>
> >>>>>>> It will be like
> >>>>>>>
> >>>>>>> ISP Router  ------ >  (new IPV6-VR ) ---- > AdvanceZone-VR 
> >>>>>>> ----> VM
> >>>>>>>
> >>>>>>> Relationship of (new IPv6 VR) and AdvanceZone-VR , may be
> considering on
> >>>>>>> OSPF instead of  BGP , otherwise few thousand of 
> >>>>>>> AdvanceZone-VR
> wil have
> >>>>>>> few thousand of BGP session. on new-IPv6-VR
> >>>>>>>
> >>>>>>> Also, I suppose we cannot do ISP router. -->. Advancezone VR
> direct,   ,
> >>>>>>> otherwise ISP router will be full of /64 prefix route either 
> >>>>>>> on
> BGP( Many
> >>>>>>> BGP Session) , or  Many Static route .   If few thousand account,
> ti will
> >>>>>>> be few thousand of BGP session with ISP router or few thousand
> static
> >>>>>> route
> >>>>>>> which  is not possible .
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Jul 15, 2021 at 10:47 PM Wido den Hollander <
> wido@widodh.nl>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> But you still need routing. See the attached PNG (and draw.io
> XML).
> >>>>>>>>
> >>>>>>>> You need to route the /48 subnet TO the VR which can then 
> >>>>>>>> route
> it to
> >>>>>>>> the Virtual Networks behind the VR.
> >>>>>>>>
> >>>>>>>> There is no other way then routing with either BGP or a 
> >>>>>>>> Static
> route.
> >>>>>>>>
> >>>>>>>> Wido
> >>>>>>>>
> >>>>>>>> Op 15-07-2021 om 12:39 schreef Hean Seng:
> >>>>>>>>> Or explain like this :
> >>>>>>>>>
> >>>>>>>>> 1) Cloudstack generate list of /64 subnet from /48 that 
> >>>>>>>>> Network
> admin
> >>>>>>>>> assigned to Cloudstack
> >>>>>>>>> 2) Cloudsack allocated the subnet (that generated from 
> >>>>>>>>> step1) to
> >>>>>> Virtual
> >>>>>>>>> Router, one Virtual Router have one subniet /64
> >>>>>>>>> 3) Virtual Router allocate single IPv6 (within the range of 
> >>>>>>>>> /64 allocated to VR)  to VM
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng 
> >>>>>>>>> <heanseng@gmail.com <ma...@gmail.com>> wrote:
> >>>>>>>>>
> >>>>>>>>>         Hi Wido,
> >>>>>>>>>
> >>>>>>>>>         I think the /48 is at physical router as gateway , 
> >>>>>>>>> and
> subnet of
> >>>>>> /64
> >>>>>>>>>         at VR of Cloudstack.   Cloudstack only keep which /48
> prefix and
> >>>>>>>>>         vlan information of this /48 to be later split the  /64.
> to VR.
> >>>>>>>>>
> >>>>>>>>>         And the instances is getting singe IPv6 of /64  IP.
>  The VR is
> >>>>>>>>>         getting /64.  The default gateway shall goes to /48 
> >>>>>>>>> of
> physical
> >>>>>>>>>         router ip .   In this case ,does not need any BGP router
> .
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         Similar concept as IPv4 :
> >>>>>>>>>
> >>>>>>>>>         /48 subnet of IPv6 is equivalent to current /24 
> >>>>>>>>> subnet
> of IPv4
> >>>>>> that
> >>>>>>>>>         created in Network.
> >>>>>>>>>         and /64  of IPv6 is equivalent to single IP of IPv4
> assign to VM.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander <
> >>>>>> wido@widodh.nl
> >>>>>>>>>         <ma...@widodh.nl>> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>             Op 14-07-2021 om 16:44 schreef Hean Seng:
> >>>>>>>>>              > Hi
> >>>>>>>>>              >
> >>>>>>>>>              > I replied in another thread, i think do not 
> >>>>>>>>> need
> implement
> >>>>>>>>>             BGP or OSPF,
> >>>>>>>>>              > that would be complicated .
> >>>>>>>>>              >
> >>>>>>>>>              > We only need assign  IPv6 's /64 prefix to
> Virtual Router
> >>>>>>>>>             (VR) in NAT
> >>>>>>>>>              > zone, and the VR responsible to deliver 
> >>>>>>>>> single
> IPv6 to VM
> >>>>>> via
> >>>>>>>>>             DHCP6.
> >>>>>>>>>              >
> >>>>>>>>>              > In VR, you need to have Default IPv6 route to
> Physical
> >>>>>>>>>             Router's /48. IP
> >>>>>>>>>              > as IPv6 Gateway.  Thens should be done .
> >>>>>>>>>              >
> >>>>>>>>>              > Example :
> >>>>>>>>>              > Physical Router Interface
> >>>>>>>>>              >   IPv6 IP : 2000:aaaa::1/48
> >>>>>>>>>              >
> >>>>>>>>>              > Cloudstack  virtual router :
> 2000:aaaa:200:201::1/64 with
> >>>>>>>>>             default ipv6
> >>>>>>>>>              > route to router ip 2000:aaaa::1
> >>>>>>>>>              > and Clodustack Virtual router dhcp allocate 
> >>>>>>>>> IP to
> VM , and
> >>>>>>>>>             VM will have
> >>>>>>>>>              > default route to VR. IPv6 2000:aaaa:200:201::1
> >>>>>>>>>              >
> >>>>>>>>>              > So in cloudstack need to allow  user to enter 
> >>>>>>>>> ,
> IPv6
> >>>>>>>>>             gwateway , and
> >>>>>>>>>              > the  /48 Ipv6 prefix , then it will self 
> >>>>>>>>> allocate
> the /64
> >>>>>> ip
> >>>>>>>>>             to the VR ,
> >>>>>>>>>              > and maintain make sure not ovelap allocation
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>
> >>>>>>>>>             But NAT is truly not the solution with IPv6. 
> >>>>>>>>> IPv6 is
> supposed
> >>>>>> to
> >>>>>>>> be
> >>>>>>>>>             routable. In addition you should avoid DHCPv6 as
> much as
> >>>>>>>>>             possible as
> >>>>>>>>>             that's not really the intended use-case for 
> >>>>>>>>> address
> allocation
> >>>>>>>>>             with IPv6.
> >>>>>>>>>
> >>>>>>>>>             In order to route an /48 IPv6 subnet to the VR 
> >>>>>>>>> you
> have a few
> >>>>>>>>>             possibilities:
> >>>>>>>>>
> >>>>>>>>>             - Static route from the upperlying routers which 
> >>>>>>>>> are
> outside
> >>>>>> of
> >>>>>>>>>             CloudStack
> >>>>>>>>>             - BGP
> >>>>>>>>>             - OSPFv3 (broken in most cases!)
> >>>>>>>>>             - DHCPv6 Prefix Delegation
> >>>>>>>>>
> >>>>>>>>>             BGP and/or Static routes are still the best bet here.
> >>>>>>>>>
> >>>>>>>>>             So what you do is that you tell CloudStack that 
> >>>>>>>>> you
> will route
> >>>>>>>>>             2001:db8::/48 to the VR, the VR can then use 
> >>>>>>>>> that to
> split it
> >>>>>> up
> >>>>>>>>>             into
> >>>>>>>>>             multiple /64 subnets going towards the instances:
> >>>>>>>>>
> >>>>>>>>>             - 2001:db8::/64
> >>>>>>>>>             - 2001:db8:1::/64
> >>>>>>>>>             - 2001:db8:2::/64
> >>>>>>>>>             ...
> >>>>>>>>>             - 2001:db8:f::/64
> >>>>>>>>>
> >>>>>>>>>             And go on.
> >>>>>>>>>
> >>>>>>>>>             In case of BGP you indeed have to tell the VR a 
> >>>>>>>>> few
> things:
> >>>>>>>>>
> >>>>>>>>>             - It's own AS number
> >>>>>>>>>             - The peer's address(es)
> >>>>>>>>>
> >>>>>>>>>             With FRR you can simply say:
> >>>>>>>>>
> >>>>>>>>>             neighbor 2001:db8:4fa::179 remote-as external
> >>>>>>>>>
> >>>>>>>>>             The /48 you need to have at the VR anyway in 
> >>>>>>>>> case of
> either a
> >>>>>>>>>             static
> >>>>>>>>>             route or BGP.
> >>>>>>>>>
> >>>>>>>>>             We just need to add a NullRoute on the VR for 
> >>>>>>>>> that
> /48 so that
> >>>>>>>>>             traffic
> >>>>>>>>>             will not be routed to the upper gateway in case 
> >>>>>>>>> of
> the VR
> >>>>>> can't
> >>>>>>>>>             find a
> >>>>>>>>>             route.
> >>>>>>>>>
> >>>>>>>>>             Wido
> >>>>>>>>>
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli
> >>>>>>>>>              > <Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>             <mailto:Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> wrote:
> >>>>>>>>>              >
> >>>>>>>>>              >     Hi Wido,
> >>>>>>>>>              >     That's pretty much in line with our thoughts,
> thanks
> >>>>>> for
> >>>>>>>>>             the input.
> >>>>>>>>>              >     I believe we agree on the following points
> then:
> >>>>>>>>>              >
> >>>>>>>>>              >     - FRR with BGP (no OSPF)
> >>>>>>>>>              >     - Route /48 (or/56) down to the VR
> >>>>>>>>>              >     - /64 per network
> >>>>>>>>>              >     - SLACC for IP addressing
> >>>>>>>>>              >
> >>>>>>>>>              >     I believe the next big question is then "on
> which level
> >>>>>>>>>             of ACS do we
> >>>>>>>>>              >     manage AS numbers?".  I see two options:
> >>>>>>>>>              >     1) Private AS number on a per-zone basis
> >>>>>>>>>              >     2) Root Admin assigned AS number on a
> domain/account
> >>>>>> basis
> >>>>>>>>>              >     3) End-user driven AS number on a per network
> basis
> >>>>>> (for
> >>>>>>>>>             bring your
> >>>>>>>>>              >     own AS and IP scenario)
> >>>>>>>>>              >
> >>>>>>>>>              >     Thoughts?
> >>>>>>>>>              >
> >>>>>>>>>              >     Cheers
> >>>>>>>>>              >     Alex
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >     -----Original Message-----
> >>>>>>>>>              >     From: Wido den Hollander <wido@widodh.nl
> >>>>>>>>>             <ma...@widodh.nl> <mailto:wido@widodh.nl
> >>>>>>>>>             <ma...@widodh.nl>>>
> >>>>>>>>>              >     Sent: 13 July 2021 15:08
> >>>>>>>>>              >     To: dev@cloudstack.apache.org
> >>>>>>>>>             <ma...@cloudstack.apache.org>
> >>>>>>>>>             <mailto:dev@cloudstack.apache.org
> >>>>>>>>>             <ma...@cloudstack.apache.org>>;
> >>>>>>>>>              >     Alex Mattioli <Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>
> >>>>>>>>>              >     Cc: Wei Zhou <Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>; Rohit Yadav
> >>>>>>>>>              >     <rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>             <mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>;
> >>>>>>>>>              >     Gabriel Beims Bräscher <gabriel@pcextreme.nl
> >>>>>>>>>             <ma...@pcextreme.nl>
> >>>>>>>>>              >     <mailto:gabriel@pcextreme.nl <mailto:
> >>>>>> gabriel@pcextreme.nl
> >>>>>>>>>>>
> >>>>>>>>>              >     Subject: Re: IPV6 in Isolated/VPC networks
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >     On 7/7/21 1:16 PM, Alex Mattioli wrote:
> >>>>>>>>>              >      > Hi all,
> >>>>>>>>>              >      > @Wei Zhou<mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> @Rohit
> >>>>>>>>>              >     Yadav<mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> and myself are
> >>>>>>>>>             investigating how
> >>>>>>>>>              >     to enable IPV6 support on Isolated and VPC
> networks and
> >>>>>>>>>             would like
> >>>>>>>>>              >     your input on it.
> >>>>>>>>>              >      > At the moment we are looking at
> implementing FRR
> >>>>>> with
> >>>>>>>>>             BGP (and
> >>>>>>>>>              >     possibly OSPF) on the ACS VR.
> >>>>>>>>>              >      >
> >>>>>>>>>              >      > We are looking for requirements,
> recommendations,
> >>>>>>>>>             ideas, rants,
> >>>>>>>>>              >     etc...etc...
> >>>>>>>>>              >      >
> >>>>>>>>>              >
> >>>>>>>>>              >     Ok! Here we go.
> >>>>>>>>>              >
> >>>>>>>>>              >     I think that you mean that the VR will
> actually route
> >>>>>> the
> >>>>>>>>>             IPv6
> >>>>>>>>>              >     traffic and for that you need to have a way
> of getting
> >>>>>> a
> >>>>>>>>>             subnet
> >>>>>>>>>              >     routed to the VR.
> >>>>>>>>>              >
> >>>>>>>>>              >     BGP is probably you best bet here. Although
> OSPFv3
> >>>>>>>>>             technically
> >>>>>>>>>              >     supports this it is very badly implemented in
> Frr for
> >>>>>>>>>             example.
> >>>>>>>>>              >
> >>>>>>>>>              >     Now FRR is a very good router and one of the
> fancy
> >>>>>>>>>             features it
> >>>>>>>>>              >     supports is BGP Unnumered. This allows for
> auto
> >>>>>>>>>             configuration of BGP
> >>>>>>>>>              >     over a L2 network when both sides are sending
> Router
> >>>>>>>>>             Advertisements.
> >>>>>>>>>              >     This is very easy for flexible BGP
> configurations where
> >>>>>>>>>             both sides
> >>>>>>>>>              >     have dynamic IPs.
> >>>>>>>>>              >
> >>>>>>>>>              >     What you want to do is that you get a /56,
> /48 or
> >>>>>>>>>             something which is
> >>>>>>>>>              >      >/64 bits routed to the VR.
> >>>>>>>>>              >
> >>>>>>>>>              >     Now you can sub-segment this into separate
> /64 subnets.
> >>>>>>>>>             You don't
> >>>>>>>>>              >     want to go smaller then a /64 is that
> prevents you from
> >>>>>>>>>             using SLAAC
> >>>>>>>>>              >     for IPv6 address configuration. This is how
> it works
> >>>>>> for
> >>>>>>>>>             Shared
> >>>>>>>>>              >     Networks now in Basic and Advanced Zones.
> >>>>>>>>>              >
> >>>>>>>>>              >     FRR can now also send out the Router
> Advertisements on
> >>>>>>>>>             the downlinks
> >>>>>>>>>              >     sending out:
> >>>>>>>>>              >
> >>>>>>>>>              >     - DNS servers
> >>>>>>>>>              >     - DNS domain
> >>>>>>>>>              >     - Prefix (/64) to be used
> >>>>>>>>>              >
> >>>>>>>>>              >     There is no need for DHCPv6. You can
> calculate the IPv6
> >>>>>>>>>             address the
> >>>>>>>>>              >     VM will obtain by using the MAC and the
> prefix.
> >>>>>>>>>              >
> >>>>>>>>>              >     So in short:
> >>>>>>>>>              >
> >>>>>>>>>              >     - Using BGP you routed a /48 to the VR
> >>>>>>>>>              >     - Now you split this into /64 subnets towards
> the
> >>>>>>>>>             isolated networks
> >>>>>>>>>              >
> >>>>>>>>>              >     Wido
> >>>>>>>>>              >
> >>>>>>>>>              >      > Alex Mattioli
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              > --
> >>>>>>>>>              > Regards,
> >>>>>>>>>              > Hean Seng
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         --
> >>>>>>>>>         Regards,
> >>>>>>>>>         Hean Seng
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Regards,
> >>>>>>>>> Hean Seng
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>

RE: IPV6 in Isolated/VPC networks

Posted by Alex Mattioli <Al...@shapeblue.com>.
Hi Kristaps,

The operator will need to add two ranges to ACS.

1) /64 for the physical network (just as we do with public IPV4 at the moment), this IPs will be used for the outside interface of the VRs
2) 1 or more /64 or larger (ideally /56s I'd say), which ACS will subnet into /64s and assign them to the inside interfaces of the VRs.

Cheers
Alex

 


-----Original Message-----
From: Kristaps Cudars <kr...@gmail.com> 
Sent: 12 August 2021 10:37
To: dev@cloudstack.apache.org
Cc: users@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks

Hi Rohit,

I think its excellent and satisfies all involved sides.

We will be in front seats bringing this to production.


Suggestion:

This

*   At the zone level root-admin specifies a /64 public range that will be
used for VRs, then

they can add a /48, or /56 IPv6 range for guest networks (to be used by isolated networks and VPC tiers)


To

*   At the zone level root-admin specifies a /64 public range that will be
used for VRs, then

they can add one or more ranges equal or larger than /64 IPv6 and ACS will subnet it in /64s

for Isolated networks or VPC tiers.


Way:

RIPE NCC in case of PI (Provider Independent) range is granting /48 and its important to have

possibility of assigning several subnets, for example /58 to be used for internal /64 allocations.

If you want to extend your /48 you will need to have good explanation why you want it.
RIPE NCC: Organisations requesting an IPv6 PI assignment larger than a /48 need to provide

additional documentation to justify the larger assignment size.

The RIPE NCC will evaluate the PI assignment request and assign an independent IPv6 prefix

to the End User organization directly.

On Wed, 11 Aug 2021 at 15:26, Rohit Yadav <ro...@shapeblue.com> wrote:

> Hi all,
>
> Thanks for your feedback and ideas, I've gone ahead with discussing 
> them with Alex and came up with a PoC/design which can be implemented 
> in the following phases:
>
>   *   Phase1: implement ipv6 support in isolated networks and VPC with
> static routing
>   *   Phase2: discuss and implement support for dynamic routing (TBD)
>
> For Phase1 here's the high-level proposal:
>
>   *   IPv6 address management:
>      *   At the zone level root-admin specifies a /64 public range that
> will be used for VRs, then they can add a /48, or /56 IPv6 range for 
> guest networks (to be used by isolated networks and VPC tiers)
>      *   On creation of any IPv6 enabled isolated network or VPC tier,
> from the /48 or /56 block a /64 network is allocated/used
>      *   We assume SLAAC and autoconfiguration, no DHCPv6 in the zone
> (discuss: is privacy a concern, can privacy extensions rfc4941 of 
> slaac be
> explored?)
>   *   Network offerings: root-admin can create new network offerings (with
> VPC too) that specifies a network stack option:
>      *   ipv4 only (default, for backward compatibility all
> networks/offerings post-upgrade migrate to this option)
>      *   ipv4-and-ipv6
>      *   ipv6-only (this can be phase 1.b)
>      *   A new routing option: static (phase1), dynamic (phase2, with
> multiple sub-options such as ospf/bgp etc...)
>   *   VR changes:
>      *   VR gets its guest and public nics set to inet6 auto
>      *   For each /64 allocated to guest network and VPC tiers, radvd is
> configured to do RA
>      *   Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6
> firewall rules (ACLs, ingress, egress) are implemented; ACLs between 
> VPC tiers are managed/implemented by ipv6 firewall on VR
>      *   It is assumed that static routes are created on the core/main
> router by the admin or automated using some scripts/tools; for this 
> CloudStack will announce events with details of /64 networks and VR's 
> public IPv6 address that can be consumed by a rabbitmq/message bus 
> client (for example), or a custom cron job or script as part of orchestration.
> (this wouldn't be necessary for dynamic routing bgp with phase2)
>   *   Guest Networking: With SLAAC, it's easy for CloudStack to calculate
> allocate and use a /64 and determine the IPv6 address of VR nics and 
> guest VM nics
>      *   A user create an isolated network/VPC with an offering that is
> ipv6 enabled
>      *   A user can manage firewall for the IPv6 address/guest nics;
> there'll be no port forward and LB feature though for IPv6
>      *   A users can run workloads in the guest VMs that listen on
> publically routable ipv6 addresses
>      *   Usage/billing etc continue to work, no change needed
>
> Network layout:
>
> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] -> 
> [guest VMs/nics] *core/ISP router needs static routes to be added 
> (manually or automated), assumes a /48 or /56 configured for the zone
>
> Thoughts, feedback?
>
> Proof-of-concept commentary: here's what I did to test the idea:
>
>   *   Created an isolated network and deployed a VM in my home lab
> The VR running on KVM has following nics
> eth0 - guest network
> eth1 - link local
> eth2 - public network
>
>   *   I setup a custom openwrt router on a RPi4 to serve as a toy-core
> router where I create a wan6 IPv6 tunnel using tunnel broker and I got 
> a
> /48 allocated. My configuration looks like:
> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> /64 - 2001:470:36:3e2::/64 (default allocated by)
>
> I create a LAN ipv6 (public network for CloudStack VR): at subnet/prefix 0:
> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode: SLAAC+stateless 
> DHCP (no dhcpv6)
>   *
>   *
> In the isolated VR, I enabled ipv6 as:
> net.ipv6.conf.all.disable_ipv6 = 0
> net.ipv6.conf.all.forwarding = 1
> net.ipv6.conf.all.accept_ra = 1
> net.ipv6.conf.all.accept_redirects = 1 net.ipv6.conf.all.autoconf = 1
>
> Set up a IPv6 nameserver/dns in /etc/resolve.conf And configured the 
> nics:
> echo iface eth0 inet6 auto >> /etc/network/interfaces echo iface eth2 
> inet6 auto >> /etc/network/interfaces /etc/init.d/networking restart 
> Next, restart ACS isolated network without cleanup to have it 
> reconfigure
> IPv4 nics, firewall, NAT etc
>
>   *
> Next, I created a /64 network for the isolated guest network on eth0 
> of VR using radvd:
>
> # cat /etc/radvd.conf
> interface eth0
> {
>     AdvSendAdvert on;
>     MinRtrAdvInterval 5;
>     MaxRtrAdvInterval 15;
>     prefix 2001:470:ed36:1::/64
>     {
>         AdvOnLink on;
>         AdvAutonomous on;
>     };
> };
> systemctl restart radvd
> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this
> ...:1::/64 network
>   *   Finally I added a static route in toy core-router for the new /64
> IPv6 range in the isolated network
> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev 
> <local lan nic>
>   *
> ... and I enabled firewall rules to allow any traffic to pass for the 
> new
> /64 network
>
> And voila all done! I create a domain AAAA record that points to my 
> guest VM IPv6 address a test webserver on 
> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>
> (Note: I'll get rid of the tunnel and request a new /48 block after a 
> few days, sharing this solely for testing purposes)
>
>
> Regards.
>
> ________________________________
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: Tuesday, July 20, 2021 12:46
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
>
>
> Op 19-07-2021 om 20:38 schreef Kristaps Cudars:
> > Hi Wido,
> >
> > I assume that flouting ip will not work grate with ingress/egress 
> > acl on
> VR.
> >
> >  From regular ACS user perspective:
> > I have Instance with dualstack its running web app on 443.
> > I want to swap instances for whatever reason.
> > In case of IPv4 change d-nat rule.
> > In case of IPv6 if flouting IP was not created upfront he will need 
> > to
> change dns entry that usually has 24h ttl. Inconvenience degradation 
> in experience.
> >
>
> Yes, but, keep in mind that the IP you are using can also be 
> terminated on the VR where HAProxy proxies request to the backend VM 
> (could even be
> v4!)
>
> I'm not against DHCPv6, but I have seen many issues with implementing 
> it. Therefor I always stick to SLAAC.
>
> >  From ACS admin perspective:
> > I don’t want to have these tickets in helpdesk.
> > You needed to create another flouting IP that it would be seamless- 
> > will
> not work as answer.
> >
>
> I understand that as well.
>
> Wido
>
> >
> > On 2021/07/19 09:05:54, Wido den Hollander <wi...@widodh.nl> wrote:
> >>
> >>
> >> Op 16-07-2021 om 21:46 schreef Kristaps Cudars:
> >>> Hi Wido,
> >>>
> >>> Your proposal is to sacrifice ability to reassign IPv6 to 
> >>> instance,
> have internal domain prefix, and list/db in ACS what IPv6 has been 
> assigned to what instance and go with RA and SLAAC. For route 
> signaling to switch use BGP/OSPFv3 or manual pre-creation.
> >>>
> >>
> >> You can still list the IPs which have been assigned. You'll know 
> >> exactly what IPv6 address a VM has because of the prefix + MAC. 
> >> Privacy Extensions need to be disabled in the VM.
> >>
> >> This already works in CloudStack in Shared Networks in this way.
> >>
> >> Using secondary IPs you can always have 'floating' IPv6 addressess.
> >>
> >> Wido
> >>
> >>> Option with RA and managed flag that DHCPv6 is in use to support
> preset information and ability to create route information from ACS is 
> not an option as DHCPv6 its failing?
> >>>
> >>>
> >>> On 2021/07/16 15:17:42, Wido den Hollander <wi...@widodh.nl> wrote:
> >>>>
> >>>>
> >>>> Op 16-07-2021 om 16:42 schreef Hean Seng:
> >>>>> Hi Wido,
> >>>>>
> >>>>> In current setup,  each Cloudstack have own VR, so in this new  
> >>>>> IPv6
> subnet
> >>>>> allocation , each VR (which have Frr) will need to have peering 
> >>>>> with
> ISP
> >>>>> router (and either BGP or Static Route) , and there is 1000
> Acocunts,  it
> >>>>> will 1000 BGP session with ISP router ,  Am I right for this ? 
> >>>>> or I understand wrong .
> >>>>>
> >>>>
> >>>> Yes, that is correct. A /56 would also be sufficient or a /60 
> >>>> which is enough to allocate a few /64 subnets.
> >>>>
> >>>> 1000 BGP connections isn't really a problem for a proper router 
> >>>> at the ISP. OSPF(v3) would be better, but as I said that's poorly supported.
> >>>>
> >>>> The ISP could also install 1000 static routes, but that means 
> >>>> that the ISP's router needs to have those configured.
> >>>>
> >>>> http://docs.frrouting.org/en/latest/ospf6d.html
> >>>> (While looking up this URL I see that Frr recently put in a lot 
> >>>> of
> work
> >>>> in OSPFv3, seems better now)
> >>>>
> >>>>> I understand IPv6 is different then IPv4, and in IPv6 it suppose 
> >>>>> each devices have own IP. It just how to realize in easy way.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
>
>
> > On Fri, Jul 16, 2021 at 8:17 PM Wido den Hollander <wi...@widodh.nl>
> wrote:
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> Op 16-07-2021 om 05:54 schreef Hean Seng:
> >>>>>>> Hi Wido,
> >>>>>>>
> >>>>>>> My initial thought is not like this,  it is the /48 at ISP 
> >>>>>>> router,
> and
> >>>>>> /64
> >>>>>>> subnet assign to AdvanceZoneVR,   AdvanceZoneVR responsible is
> >>>>>>> distribule IPv6 ip (from the assigned /64 sunet) to VM,  and 
> >>>>>>> not
> routing
> >>>>>>> the traffic,   in the VM that get the IPv6 IP will default route
> to ISP
> >>>>>>> router as gw.   It can may be a bridge over via Advancezone-VR.
> >>>>>>>
> >>>>>>
> >>>>>> How would you bridge this? That sounds like NAT?
> >>>>>>
> >>>>>> IPv6 is meant to be routed. Not to be translated or bridged in 
> >>>>>> any
> way.
> >>>>>>
> >>>>>> The way a made the drawing is exactly how IPv6 should work in a 
> >>>>>> VPC environment.
> >>>>>>
> >>>>>> Traffic flows through the VR where it can do firewalling of the
> traffic.
> >>>>>>
> >>>>>>> However, If do as the way described in the drawing, then i 
> >>>>>>> suppose
> will
> >>>>>> be
> >>>>>>> another kind of virtual router going to introduce , to get 
> >>>>>>> hold
> the /48
> >>>>>> in
> >>>>>>> this virtual router right ?
> >>>>>>>
> >>>>>>
> >>>>>> It can be the same VR. But keep in mind that IPv6 != IPv4.
> >>>>>>
> >>>>>> The VR will get Frr as a new daemon which can talk BGP with the
> upper
> >>>>>> network to route traffic.
> >>>>>>
> >>>>>>> After this,  The Advance Zone, NAT's  VR will peer with this 
> >>>>>>> new
> IPv6 VR
> >>>>>>> for getting the IPv6 /64 prefix ?
> >>>>>>>
> >>>>>>
> >>>>>> IPv4 will be behind NAT, but IPv6 will not be behind NAT.
> >>>>>>
> >>>>>>> If do in this way, then I guess  you just only need Static 
> >>>>>>> route,
> with
> >>>>>>> peering ip both end  as one /48 can have a lot of /64 on it.  
> >>>>>>> And
> >>>>>> hardware
> >>>>>>> budgeting for new IPv6-VR will become very important, as all
> traffic will
> >>>>>>> need to pass over it .
> >>>>>>>
> >>>>>>
> >>>>>> Routing or NAT is the same for the VR. You don't need a very 
> >>>>>> beefy
> VR
> >>>>>> for this.
> >>>>>>
> >>>>>>> It will be like
> >>>>>>>
> >>>>>>> ISP Router  ------ >  (new IPV6-VR ) ---- > AdvanceZone-VR 
> >>>>>>> ----> VM
> >>>>>>>
> >>>>>>> Relationship of (new IPv6 VR) and AdvanceZone-VR , may be
> considering on
> >>>>>>> OSPF instead of  BGP , otherwise few thousand of 
> >>>>>>> AdvanceZone-VR
> wil have
> >>>>>>> few thousand of BGP session. on new-IPv6-VR
> >>>>>>>
> >>>>>>> Also, I suppose we cannot do ISP router. -->. Advancezone VR
> direct,   ,
> >>>>>>> otherwise ISP router will be full of /64 prefix route either 
> >>>>>>> on
> BGP( Many
> >>>>>>> BGP Session) , or  Many Static route .   If few thousand account,
> ti will
> >>>>>>> be few thousand of BGP session with ISP router or few thousand
> static
> >>>>>> route
> >>>>>>> which  is not possible .
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Jul 15, 2021 at 10:47 PM Wido den Hollander <
> wido@widodh.nl>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> But you still need routing. See the attached PNG (and draw.io
> XML).
> >>>>>>>>
> >>>>>>>> You need to route the /48 subnet TO the VR which can then 
> >>>>>>>> route
> it to
> >>>>>>>> the Virtual Networks behind the VR.
> >>>>>>>>
> >>>>>>>> There is no other way then routing with either BGP or a 
> >>>>>>>> Static
> route.
> >>>>>>>>
> >>>>>>>> Wido
> >>>>>>>>
> >>>>>>>> Op 15-07-2021 om 12:39 schreef Hean Seng:
> >>>>>>>>> Or explain like this :
> >>>>>>>>>
> >>>>>>>>> 1) Cloudstack generate list of /64 subnet from /48 that 
> >>>>>>>>> Network
> admin
> >>>>>>>>> assigned to Cloudstack
> >>>>>>>>> 2) Cloudsack allocated the subnet (that generated from 
> >>>>>>>>> step1) to
> >>>>>> Virtual
> >>>>>>>>> Router, one Virtual Router have one subniet /64
> >>>>>>>>> 3) Virtual Router allocate single IPv6 (within the range of 
> >>>>>>>>> /64 allocated to VR)  to VM
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng 
> >>>>>>>>> <heanseng@gmail.com <ma...@gmail.com>> wrote:
> >>>>>>>>>
> >>>>>>>>>         Hi Wido,
> >>>>>>>>>
> >>>>>>>>>         I think the /48 is at physical router as gateway , 
> >>>>>>>>> and
> subnet of
> >>>>>> /64
> >>>>>>>>>         at VR of Cloudstack.   Cloudstack only keep which /48
> prefix and
> >>>>>>>>>         vlan information of this /48 to be later split the  /64.
> to VR.
> >>>>>>>>>
> >>>>>>>>>         And the instances is getting singe IPv6 of /64  IP.
>  The VR is
> >>>>>>>>>         getting /64.  The default gateway shall goes to /48 
> >>>>>>>>> of
> physical
> >>>>>>>>>         router ip .   In this case ,does not need any BGP router
> .
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         Similar concept as IPv4 :
> >>>>>>>>>
> >>>>>>>>>         /48 subnet of IPv6 is equivalent to current /24 
> >>>>>>>>> subnet
> of IPv4
> >>>>>> that
> >>>>>>>>>         created in Network.
> >>>>>>>>>         and /64  of IPv6 is equivalent to single IP of IPv4
> assign to VM.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander <
> >>>>>> wido@widodh.nl
> >>>>>>>>>         <ma...@widodh.nl>> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>             Op 14-07-2021 om 16:44 schreef Hean Seng:
> >>>>>>>>>              > Hi
> >>>>>>>>>              >
> >>>>>>>>>              > I replied in another thread, i think do not 
> >>>>>>>>> need
> implement
> >>>>>>>>>             BGP or OSPF,
> >>>>>>>>>              > that would be complicated .
> >>>>>>>>>              >
> >>>>>>>>>              > We only need assign  IPv6 's /64 prefix to
> Virtual Router
> >>>>>>>>>             (VR) in NAT
> >>>>>>>>>              > zone, and the VR responsible to deliver 
> >>>>>>>>> single
> IPv6 to VM
> >>>>>> via
> >>>>>>>>>             DHCP6.
> >>>>>>>>>              >
> >>>>>>>>>              > In VR, you need to have Default IPv6 route to
> Physical
> >>>>>>>>>             Router's /48. IP
> >>>>>>>>>              > as IPv6 Gateway.  Thens should be done .
> >>>>>>>>>              >
> >>>>>>>>>              > Example :
> >>>>>>>>>              > Physical Router Interface
> >>>>>>>>>              >   IPv6 IP : 2000:aaaa::1/48
> >>>>>>>>>              >
> >>>>>>>>>              > Cloudstack  virtual router :
> 2000:aaaa:200:201::1/64 with
> >>>>>>>>>             default ipv6
> >>>>>>>>>              > route to router ip 2000:aaaa::1
> >>>>>>>>>              > and Clodustack Virtual router dhcp allocate 
> >>>>>>>>> IP to
> VM , and
> >>>>>>>>>             VM will have
> >>>>>>>>>              > default route to VR. IPv6 2000:aaaa:200:201::1
> >>>>>>>>>              >
> >>>>>>>>>              > So in cloudstack need to allow  user to enter 
> >>>>>>>>> ,
> IPv6
> >>>>>>>>>             gwateway , and
> >>>>>>>>>              > the  /48 Ipv6 prefix , then it will self 
> >>>>>>>>> allocate
> the /64
> >>>>>> ip
> >>>>>>>>>             to the VR ,
> >>>>>>>>>              > and maintain make sure not ovelap allocation
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>
> >>>>>>>>>             But NAT is truly not the solution with IPv6. 
> >>>>>>>>> IPv6 is
> supposed
> >>>>>> to
> >>>>>>>> be
> >>>>>>>>>             routable. In addition you should avoid DHCPv6 as
> much as
> >>>>>>>>>             possible as
> >>>>>>>>>             that's not really the intended use-case for 
> >>>>>>>>> address
> allocation
> >>>>>>>>>             with IPv6.
> >>>>>>>>>
> >>>>>>>>>             In order to route an /48 IPv6 subnet to the VR 
> >>>>>>>>> you
> have a few
> >>>>>>>>>             possibilities:
> >>>>>>>>>
> >>>>>>>>>             - Static route from the upperlying routers which 
> >>>>>>>>> are
> outside
> >>>>>> of
> >>>>>>>>>             CloudStack
> >>>>>>>>>             - BGP
> >>>>>>>>>             - OSPFv3 (broken in most cases!)
> >>>>>>>>>             - DHCPv6 Prefix Delegation
> >>>>>>>>>
> >>>>>>>>>             BGP and/or Static routes are still the best bet here.
> >>>>>>>>>
> >>>>>>>>>             So what you do is that you tell CloudStack that 
> >>>>>>>>> you
> will route
> >>>>>>>>>             2001:db8::/48 to the VR, the VR can then use 
> >>>>>>>>> that to
> split it
> >>>>>> up
> >>>>>>>>>             into
> >>>>>>>>>             multiple /64 subnets going towards the instances:
> >>>>>>>>>
> >>>>>>>>>             - 2001:db8::/64
> >>>>>>>>>             - 2001:db8:1::/64
> >>>>>>>>>             - 2001:db8:2::/64
> >>>>>>>>>             ...
> >>>>>>>>>             - 2001:db8:f::/64
> >>>>>>>>>
> >>>>>>>>>             And go on.
> >>>>>>>>>
> >>>>>>>>>             In case of BGP you indeed have to tell the VR a 
> >>>>>>>>> few
> things:
> >>>>>>>>>
> >>>>>>>>>             - It's own AS number
> >>>>>>>>>             - The peer's address(es)
> >>>>>>>>>
> >>>>>>>>>             With FRR you can simply say:
> >>>>>>>>>
> >>>>>>>>>             neighbor 2001:db8:4fa::179 remote-as external
> >>>>>>>>>
> >>>>>>>>>             The /48 you need to have at the VR anyway in 
> >>>>>>>>> case of
> either a
> >>>>>>>>>             static
> >>>>>>>>>             route or BGP.
> >>>>>>>>>
> >>>>>>>>>             We just need to add a NullRoute on the VR for 
> >>>>>>>>> that
> /48 so that
> >>>>>>>>>             traffic
> >>>>>>>>>             will not be routed to the upper gateway in case 
> >>>>>>>>> of
> the VR
> >>>>>> can't
> >>>>>>>>>             find a
> >>>>>>>>>             route.
> >>>>>>>>>
> >>>>>>>>>             Wido
> >>>>>>>>>
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli
> >>>>>>>>>              > <Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>             <mailto:Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> wrote:
> >>>>>>>>>              >
> >>>>>>>>>              >     Hi Wido,
> >>>>>>>>>              >     That's pretty much in line with our thoughts,
> thanks
> >>>>>> for
> >>>>>>>>>             the input.
> >>>>>>>>>              >     I believe we agree on the following points
> then:
> >>>>>>>>>              >
> >>>>>>>>>              >     - FRR with BGP (no OSPF)
> >>>>>>>>>              >     - Route /48 (or/56) down to the VR
> >>>>>>>>>              >     - /64 per network
> >>>>>>>>>              >     - SLACC for IP addressing
> >>>>>>>>>              >
> >>>>>>>>>              >     I believe the next big question is then "on
> which level
> >>>>>>>>>             of ACS do we
> >>>>>>>>>              >     manage AS numbers?".  I see two options:
> >>>>>>>>>              >     1) Private AS number on a per-zone basis
> >>>>>>>>>              >     2) Root Admin assigned AS number on a
> domain/account
> >>>>>> basis
> >>>>>>>>>              >     3) End-user driven AS number on a per network
> basis
> >>>>>> (for
> >>>>>>>>>             bring your
> >>>>>>>>>              >     own AS and IP scenario)
> >>>>>>>>>              >
> >>>>>>>>>              >     Thoughts?
> >>>>>>>>>              >
> >>>>>>>>>              >     Cheers
> >>>>>>>>>              >     Alex
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >     -----Original Message-----
> >>>>>>>>>              >     From: Wido den Hollander <wido@widodh.nl
> >>>>>>>>>             <ma...@widodh.nl> <mailto:wido@widodh.nl
> >>>>>>>>>             <ma...@widodh.nl>>>
> >>>>>>>>>              >     Sent: 13 July 2021 15:08
> >>>>>>>>>              >     To: dev@cloudstack.apache.org
> >>>>>>>>>             <ma...@cloudstack.apache.org>
> >>>>>>>>>             <mailto:dev@cloudstack.apache.org
> >>>>>>>>>             <ma...@cloudstack.apache.org>>;
> >>>>>>>>>              >     Alex Mattioli <Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>
> >>>>>>>>>              >     Cc: Wei Zhou <Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>; Rohit Yadav
> >>>>>>>>>              >     <rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>             <mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>;
> >>>>>>>>>              >     Gabriel Beims Bräscher <gabriel@pcextreme.nl
> >>>>>>>>>             <ma...@pcextreme.nl>
> >>>>>>>>>              >     <mailto:gabriel@pcextreme.nl <mailto:
> >>>>>> gabriel@pcextreme.nl
> >>>>>>>>>>>
> >>>>>>>>>              >     Subject: Re: IPV6 in Isolated/VPC networks
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >     On 7/7/21 1:16 PM, Alex Mattioli wrote:
> >>>>>>>>>              >      > Hi all,
> >>>>>>>>>              >      > @Wei Zhou<mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> @Rohit
> >>>>>>>>>              >     Yadav<mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> and myself are
> >>>>>>>>>             investigating how
> >>>>>>>>>              >     to enable IPV6 support on Isolated and VPC
> networks and
> >>>>>>>>>             would like
> >>>>>>>>>              >     your input on it.
> >>>>>>>>>              >      > At the moment we are looking at
> implementing FRR
> >>>>>> with
> >>>>>>>>>             BGP (and
> >>>>>>>>>              >     possibly OSPF) on the ACS VR.
> >>>>>>>>>              >      >
> >>>>>>>>>              >      > We are looking for requirements,
> recommendations,
> >>>>>>>>>             ideas, rants,
> >>>>>>>>>              >     etc...etc...
> >>>>>>>>>              >      >
> >>>>>>>>>              >
> >>>>>>>>>              >     Ok! Here we go.
> >>>>>>>>>              >
> >>>>>>>>>              >     I think that you mean that the VR will
> actually route
> >>>>>> the
> >>>>>>>>>             IPv6
> >>>>>>>>>              >     traffic and for that you need to have a way
> of getting
> >>>>>> a
> >>>>>>>>>             subnet
> >>>>>>>>>              >     routed to the VR.
> >>>>>>>>>              >
> >>>>>>>>>              >     BGP is probably you best bet here. Although
> OSPFv3
> >>>>>>>>>             technically
> >>>>>>>>>              >     supports this it is very badly implemented in
> Frr for
> >>>>>>>>>             example.
> >>>>>>>>>              >
> >>>>>>>>>              >     Now FRR is a very good router and one of the
> fancy
> >>>>>>>>>             features it
> >>>>>>>>>              >     supports is BGP Unnumered. This allows for
> auto
> >>>>>>>>>             configuration of BGP
> >>>>>>>>>              >     over a L2 network when both sides are sending
> Router
> >>>>>>>>>             Advertisements.
> >>>>>>>>>              >     This is very easy for flexible BGP
> configurations where
> >>>>>>>>>             both sides
> >>>>>>>>>              >     have dynamic IPs.
> >>>>>>>>>              >
> >>>>>>>>>              >     What you want to do is that you get a /56,
> /48 or
> >>>>>>>>>             something which is
> >>>>>>>>>              >      >/64 bits routed to the VR.
> >>>>>>>>>              >
> >>>>>>>>>              >     Now you can sub-segment this into separate
> /64 subnets.
> >>>>>>>>>             You don't
> >>>>>>>>>              >     want to go smaller then a /64 is that
> prevents you from
> >>>>>>>>>             using SLAAC
> >>>>>>>>>              >     for IPv6 address configuration. This is how
> it works
> >>>>>> for
> >>>>>>>>>             Shared
> >>>>>>>>>              >     Networks now in Basic and Advanced Zones.
> >>>>>>>>>              >
> >>>>>>>>>              >     FRR can now also send out the Router
> Advertisements on
> >>>>>>>>>             the downlinks
> >>>>>>>>>              >     sending out:
> >>>>>>>>>              >
> >>>>>>>>>              >     - DNS servers
> >>>>>>>>>              >     - DNS domain
> >>>>>>>>>              >     - Prefix (/64) to be used
> >>>>>>>>>              >
> >>>>>>>>>              >     There is no need for DHCPv6. You can
> calculate the IPv6
> >>>>>>>>>             address the
> >>>>>>>>>              >     VM will obtain by using the MAC and the
> prefix.
> >>>>>>>>>              >
> >>>>>>>>>              >     So in short:
> >>>>>>>>>              >
> >>>>>>>>>              >     - Using BGP you routed a /48 to the VR
> >>>>>>>>>              >     - Now you split this into /64 subnets towards
> the
> >>>>>>>>>             isolated networks
> >>>>>>>>>              >
> >>>>>>>>>              >     Wido
> >>>>>>>>>              >
> >>>>>>>>>              >      > Alex Mattioli
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              > --
> >>>>>>>>>              > Regards,
> >>>>>>>>>              > Hean Seng
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         --
> >>>>>>>>>         Regards,
> >>>>>>>>>         Hean Seng
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Regards,
> >>>>>>>>> Hean Seng
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>

Re: IPV6 in Isolated/VPC networks

Posted by Kristaps Cudars <kr...@gmail.com>.
Hi Rohit,

I think its excellent and satisfies all involved sides.

We will be in front seats bringing this to production.


Suggestion:

This

*   At the zone level root-admin specifies a /64 public range that will be
used for VRs, then

they can add a /48, or /56 IPv6 range for guest networks (to be used by
isolated networks and VPC tiers)


To

*   At the zone level root-admin specifies a /64 public range that will be
used for VRs, then

they can add one or more ranges equal or larger than /64 IPv6 and ACS will
subnet it in /64s

for Isolated networks or VPC tiers.


Way:

RIPE NCC in case of PI (Provider Independent) range is granting /48 and its
important to have

possibility of assigning several subnets, for example /58 to be used for
internal /64 allocations.

If you want to extend your /48 you will need to have good explanation why
you want it.
RIPE NCC: Organisations requesting an IPv6 PI assignment larger than a /48
need to provide

additional documentation to justify the larger assignment size.

The RIPE NCC will evaluate the PI assignment request and assign an
independent IPv6 prefix

to the End User organization directly.

On Wed, 11 Aug 2021 at 15:26, Rohit Yadav <ro...@shapeblue.com> wrote:

> Hi all,
>
> Thanks for your feedback and ideas, I've gone ahead with discussing them
> with Alex and came up with a PoC/design which can be implemented in the
> following phases:
>
>   *   Phase1: implement ipv6 support in isolated networks and VPC with
> static routing
>   *   Phase2: discuss and implement support for dynamic routing (TBD)
>
> For Phase1 here's the high-level proposal:
>
>   *   IPv6 address management:
>      *   At the zone level root-admin specifies a /64 public range that
> will be used for VRs, then they can add a /48, or /56 IPv6 range for guest
> networks (to be used by isolated networks and VPC tiers)
>      *   On creation of any IPv6 enabled isolated network or VPC tier,
> from the /48 or /56 block a /64 network is allocated/used
>      *   We assume SLAAC and autoconfiguration, no DHCPv6 in the zone
> (discuss: is privacy a concern, can privacy extensions rfc4941 of slaac be
> explored?)
>   *   Network offerings: root-admin can create new network offerings (with
> VPC too) that specifies a network stack option:
>      *   ipv4 only (default, for backward compatibility all
> networks/offerings post-upgrade migrate to this option)
>      *   ipv4-and-ipv6
>      *   ipv6-only (this can be phase 1.b)
>      *   A new routing option: static (phase1), dynamic (phase2, with
> multiple sub-options such as ospf/bgp etc...)
>   *   VR changes:
>      *   VR gets its guest and public nics set to inet6 auto
>      *   For each /64 allocated to guest network and VPC tiers, radvd is
> configured to do RA
>      *   Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6
> firewall rules (ACLs, ingress, egress) are implemented; ACLs between VPC
> tiers are managed/implemented by ipv6 firewall on VR
>      *   It is assumed that static routes are created on the core/main
> router by the admin or automated using some scripts/tools; for this
> CloudStack will announce events with details of /64 networks and VR's
> public IPv6 address that can be consumed by a rabbitmq/message bus client
> (for example), or a custom cron job or script as part of orchestration.
> (this wouldn't be necessary for dynamic routing bgp with phase2)
>   *   Guest Networking: With SLAAC, it's easy for CloudStack to calculate
> allocate and use a /64 and determine the IPv6 address of VR nics and guest
> VM nics
>      *   A user create an isolated network/VPC with an offering that is
> ipv6 enabled
>      *   A user can manage firewall for the IPv6 address/guest nics;
> there'll be no port forward and LB feature though for IPv6
>      *   A users can run workloads in the guest VMs that listen on
> publically routable ipv6 addresses
>      *   Usage/billing etc continue to work, no change needed
>
> Network layout:
>
> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] ->
> [guest VMs/nics]
> *core/ISP router needs static routes to be added (manually or automated),
> assumes a /48 or /56 configured for the zone
>
> Thoughts, feedback?
>
> Proof-of-concept commentary: here's what I did to test the idea:
>
>   *   Created an isolated network and deployed a VM in my home lab
> The VR running on KVM has following nics
> eth0 - guest network
> eth1 - link local
> eth2 - public network
>
>   *   I setup a custom openwrt router on a RPi4 to serve as a toy-core
> router where I create a wan6 IPv6 tunnel using tunnel broker and I got a
> /48 allocated. My configuration looks like:
> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> /64 - 2001:470:36:3e2::/64 (default allocated by)
>
> I create a LAN ipv6 (public network for CloudStack VR): at subnet/prefix 0:
> LAN IPv6 address: 2001:470:ed36:0::1/64
> Address mode: SLAAC+stateless DHCP (no dhcpv6)
>   *
>   *
> In the isolated VR, I enabled ipv6 as:
> net.ipv6.conf.all.disable_ipv6 = 0
> net.ipv6.conf.all.forwarding = 1
> net.ipv6.conf.all.accept_ra = 1
> net.ipv6.conf.all.accept_redirects = 1
> net.ipv6.conf.all.autoconf = 1
>
> Set up a IPv6 nameserver/dns in /etc/resolve.conf
> And configured the nics:
> echo iface eth0 inet6 auto >> /etc/network/interfaces
> echo iface eth2 inet6 auto >> /etc/network/interfaces
> /etc/init.d/networking restart
> Next, restart ACS isolated network without cleanup to have it reconfigure
> IPv4 nics, firewall, NAT etc
>
>   *
> Next, I created a /64 network for the isolated guest network on eth0 of VR
> using radvd:
>
> # cat /etc/radvd.conf
> interface eth0
> {
>     AdvSendAdvert on;
>     MinRtrAdvInterval 5;
>     MaxRtrAdvInterval 15;
>     prefix 2001:470:ed36:1::/64
>     {
>         AdvOnLink on;
>         AdvAutonomous on;
>     };
> };
> systemctl restart radvd
> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this
> ...:1::/64 network
>   *   Finally I added a static route in toy core-router for the new /64
> IPv6 range in the isolated network
> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev
> <local lan nic>
>   *
> ... and I enabled firewall rules to allow any traffic to pass for the new
> /64 network
>
> And voila all done! I create a domain AAAA record that points to my guest
> VM IPv6 address a test webserver on
> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>
> (Note: I'll get rid of the tunnel and request a new /48 block after a few
> days, sharing this solely for testing purposes)
>
>
> Regards.
>
> ________________________________
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: Tuesday, July 20, 2021 12:46
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
>
>
> Op 19-07-2021 om 20:38 schreef Kristaps Cudars:
> > Hi Wido,
> >
> > I assume that flouting ip will not work grate with ingress/egress acl on
> VR.
> >
> >  From regular ACS user perspective:
> > I have Instance with dualstack its running web app on 443.
> > I want to swap instances for whatever reason.
> > In case of IPv4 change d-nat rule.
> > In case of IPv6 if flouting IP was not created upfront he will need to
> change dns entry that usually has 24h ttl. Inconvenience degradation in
> experience.
> >
>
> Yes, but, keep in mind that the IP you are using can also be terminated
> on the VR where HAProxy proxies request to the backend VM (could even be
> v4!)
>
> I'm not against DHCPv6, but I have seen many issues with implementing
> it. Therefor I always stick to SLAAC.
>
> >  From ACS admin perspective:
> > I don’t want to have these tickets in helpdesk.
> > You needed to create another flouting IP that it would be seamless- will
> not work as answer.
> >
>
> I understand that as well.
>
> Wido
>
> >
> > On 2021/07/19 09:05:54, Wido den Hollander <wi...@widodh.nl> wrote:
> >>
> >>
> >> Op 16-07-2021 om 21:46 schreef Kristaps Cudars:
> >>> Hi Wido,
> >>>
> >>> Your proposal is to sacrifice ability to reassign IPv6 to instance,
> have internal domain prefix, and list/db in ACS what IPv6 has been assigned
> to what instance and go with RA and SLAAC. For route signaling to switch
> use BGP/OSPFv3 or manual pre-creation.
> >>>
> >>
> >> You can still list the IPs which have been assigned. You'll know exactly
> >> what IPv6 address a VM has because of the prefix + MAC. Privacy
> >> Extensions need to be disabled in the VM.
> >>
> >> This already works in CloudStack in Shared Networks in this way.
> >>
> >> Using secondary IPs you can always have 'floating' IPv6 addressess.
> >>
> >> Wido
> >>
> >>> Option with RA and managed flag that DHCPv6 is in use to support
> preset information and ability to create route information from ACS is not
> an option as DHCPv6 its failing?
> >>>
> >>>
> >>> On 2021/07/16 15:17:42, Wido den Hollander <wi...@widodh.nl> wrote:
> >>>>
> >>>>
> >>>> Op 16-07-2021 om 16:42 schreef Hean Seng:
> >>>>> Hi Wido,
> >>>>>
> >>>>> In current setup,  each Cloudstack have own VR, so in this new  IPv6
> subnet
> >>>>> allocation , each VR (which have Frr) will need to have peering with
> ISP
> >>>>> router (and either BGP or Static Route) , and there is 1000
> Acocunts,  it
> >>>>> will 1000 BGP session with ISP router ,  Am I right for this ? or I
> >>>>> understand wrong .
> >>>>>
> >>>>
> >>>> Yes, that is correct. A /56 would also be sufficient or a /60 which is
> >>>> enough to allocate a few /64 subnets.
> >>>>
> >>>> 1000 BGP connections isn't really a problem for a proper router at the
> >>>> ISP. OSPF(v3) would be better, but as I said that's poorly supported.
> >>>>
> >>>> The ISP could also install 1000 static routes, but that means that the
> >>>> ISP's router needs to have those configured.
> >>>>
> >>>> http://docs.frrouting.org/en/latest/ospf6d.html
> >>>> (While looking up this URL I see that Frr recently put in a lot of
> work
> >>>> in OSPFv3, seems better now)
> >>>>
> >>>>> I understand IPv6 is different then IPv4, and in IPv6 it suppose each
> >>>>> devices have own IP. It just how to realize in easy way.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
>
>
> > On Fri, Jul 16, 2021 at 8:17 PM Wido den Hollander <wi...@widodh.nl>
> wrote:
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> Op 16-07-2021 om 05:54 schreef Hean Seng:
> >>>>>>> Hi Wido,
> >>>>>>>
> >>>>>>> My initial thought is not like this,  it is the /48 at ISP router,
> and
> >>>>>> /64
> >>>>>>> subnet assign to AdvanceZoneVR,   AdvanceZoneVR responsible is
> >>>>>>> distribule IPv6 ip (from the assigned /64 sunet) to VM,  and not
> routing
> >>>>>>> the traffic,   in the VM that get the IPv6 IP will default route
> to ISP
> >>>>>>> router as gw.   It can may be a bridge over via Advancezone-VR.
> >>>>>>>
> >>>>>>
> >>>>>> How would you bridge this? That sounds like NAT?
> >>>>>>
> >>>>>> IPv6 is meant to be routed. Not to be translated or bridged in any
> way.
> >>>>>>
> >>>>>> The way a made the drawing is exactly how IPv6 should work in a VPC
> >>>>>> environment.
> >>>>>>
> >>>>>> Traffic flows through the VR where it can do firewalling of the
> traffic.
> >>>>>>
> >>>>>>> However, If do as the way described in the drawing, then i suppose
> will
> >>>>>> be
> >>>>>>> another kind of virtual router going to introduce , to get hold
> the /48
> >>>>>> in
> >>>>>>> this virtual router right ?
> >>>>>>>
> >>>>>>
> >>>>>> It can be the same VR. But keep in mind that IPv6 != IPv4.
> >>>>>>
> >>>>>> The VR will get Frr as a new daemon which can talk BGP with the
> upper
> >>>>>> network to route traffic.
> >>>>>>
> >>>>>>> After this,  The Advance Zone, NAT's  VR will peer with this new
> IPv6 VR
> >>>>>>> for getting the IPv6 /64 prefix ?
> >>>>>>>
> >>>>>>
> >>>>>> IPv4 will be behind NAT, but IPv6 will not be behind NAT.
> >>>>>>
> >>>>>>> If do in this way, then I guess  you just only need Static route,
> with
> >>>>>>> peering ip both end  as one /48 can have a lot of /64 on it.  And
> >>>>>> hardware
> >>>>>>> budgeting for new IPv6-VR will become very important, as all
> traffic will
> >>>>>>> need to pass over it .
> >>>>>>>
> >>>>>>
> >>>>>> Routing or NAT is the same for the VR. You don't need a very beefy
> VR
> >>>>>> for this.
> >>>>>>
> >>>>>>> It will be like
> >>>>>>>
> >>>>>>> ISP Router  ------ >  (new IPV6-VR ) ---- > AdvanceZone-VR ----> VM
> >>>>>>>
> >>>>>>> Relationship of (new IPv6 VR) and AdvanceZone-VR , may be
> considering on
> >>>>>>> OSPF instead of  BGP , otherwise few thousand of AdvanceZone-VR
> wil have
> >>>>>>> few thousand of BGP session. on new-IPv6-VR
> >>>>>>>
> >>>>>>> Also, I suppose we cannot do ISP router. -->. Advancezone VR
> direct,   ,
> >>>>>>> otherwise ISP router will be full of /64 prefix route either on
> BGP( Many
> >>>>>>> BGP Session) , or  Many Static route .   If few thousand account,
> ti will
> >>>>>>> be few thousand of BGP session with ISP router or few thousand
> static
> >>>>>> route
> >>>>>>> which  is not possible .
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Jul 15, 2021 at 10:47 PM Wido den Hollander <
> wido@widodh.nl>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> But you still need routing. See the attached PNG (and draw.io
> XML).
> >>>>>>>>
> >>>>>>>> You need to route the /48 subnet TO the VR which can then route
> it to
> >>>>>>>> the Virtual Networks behind the VR.
> >>>>>>>>
> >>>>>>>> There is no other way then routing with either BGP or a Static
> route.
> >>>>>>>>
> >>>>>>>> Wido
> >>>>>>>>
> >>>>>>>> Op 15-07-2021 om 12:39 schreef Hean Seng:
> >>>>>>>>> Or explain like this :
> >>>>>>>>>
> >>>>>>>>> 1) Cloudstack generate list of /64 subnet from /48 that Network
> admin
> >>>>>>>>> assigned to Cloudstack
> >>>>>>>>> 2) Cloudsack allocated the subnet (that generated from step1) to
> >>>>>> Virtual
> >>>>>>>>> Router, one Virtual Router have one subniet /64
> >>>>>>>>> 3) Virtual Router allocate single IPv6 (within the range of /64
> >>>>>>>>> allocated to VR)  to VM
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng <heanseng@gmail.com
> >>>>>>>>> <ma...@gmail.com>> wrote:
> >>>>>>>>>
> >>>>>>>>>         Hi Wido,
> >>>>>>>>>
> >>>>>>>>>         I think the /48 is at physical router as gateway , and
> subnet of
> >>>>>> /64
> >>>>>>>>>         at VR of Cloudstack.   Cloudstack only keep which /48
> prefix and
> >>>>>>>>>         vlan information of this /48 to be later split the  /64.
> to VR.
> >>>>>>>>>
> >>>>>>>>>         And the instances is getting singe IPv6 of /64  IP.
>  The VR is
> >>>>>>>>>         getting /64.  The default gateway shall goes to /48 of
> physical
> >>>>>>>>>         router ip .   In this case ,does not need any BGP router
> .
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         Similar concept as IPv4 :
> >>>>>>>>>
> >>>>>>>>>         /48 subnet of IPv6 is equivalent to current /24 subnet
> of IPv4
> >>>>>> that
> >>>>>>>>>         created in Network.
> >>>>>>>>>         and /64  of IPv6 is equivalent to single IP of IPv4
> assign to VM.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander <
> >>>>>> wido@widodh.nl
> >>>>>>>>>         <ma...@widodh.nl>> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>             Op 14-07-2021 om 16:44 schreef Hean Seng:
> >>>>>>>>>              > Hi
> >>>>>>>>>              >
> >>>>>>>>>              > I replied in another thread, i think do not need
> implement
> >>>>>>>>>             BGP or OSPF,
> >>>>>>>>>              > that would be complicated .
> >>>>>>>>>              >
> >>>>>>>>>              > We only need assign  IPv6 's /64 prefix to
> Virtual Router
> >>>>>>>>>             (VR) in NAT
> >>>>>>>>>              > zone, and the VR responsible to deliver single
> IPv6 to VM
> >>>>>> via
> >>>>>>>>>             DHCP6.
> >>>>>>>>>              >
> >>>>>>>>>              > In VR, you need to have Default IPv6 route to
> Physical
> >>>>>>>>>             Router's /48. IP
> >>>>>>>>>              > as IPv6 Gateway.  Thens should be done .
> >>>>>>>>>              >
> >>>>>>>>>              > Example :
> >>>>>>>>>              > Physical Router Interface
> >>>>>>>>>              >   IPv6 IP : 2000:aaaa::1/48
> >>>>>>>>>              >
> >>>>>>>>>              > Cloudstack  virtual router :
> 2000:aaaa:200:201::1/64 with
> >>>>>>>>>             default ipv6
> >>>>>>>>>              > route to router ip 2000:aaaa::1
> >>>>>>>>>              > and Clodustack Virtual router dhcp allocate IP to
> VM , and
> >>>>>>>>>             VM will have
> >>>>>>>>>              > default route to VR. IPv6 2000:aaaa:200:201::1
> >>>>>>>>>              >
> >>>>>>>>>              > So in cloudstack need to allow  user to enter ,
> IPv6
> >>>>>>>>>             gwateway , and
> >>>>>>>>>              > the  /48 Ipv6 prefix , then it will self allocate
> the /64
> >>>>>> ip
> >>>>>>>>>             to the VR ,
> >>>>>>>>>              > and maintain make sure not ovelap allocation
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>
> >>>>>>>>>             But NAT is truly not the solution with IPv6. IPv6 is
> supposed
> >>>>>> to
> >>>>>>>> be
> >>>>>>>>>             routable. In addition you should avoid DHCPv6 as
> much as
> >>>>>>>>>             possible as
> >>>>>>>>>             that's not really the intended use-case for address
> allocation
> >>>>>>>>>             with IPv6.
> >>>>>>>>>
> >>>>>>>>>             In order to route an /48 IPv6 subnet to the VR you
> have a few
> >>>>>>>>>             possibilities:
> >>>>>>>>>
> >>>>>>>>>             - Static route from the upperlying routers which are
> outside
> >>>>>> of
> >>>>>>>>>             CloudStack
> >>>>>>>>>             - BGP
> >>>>>>>>>             - OSPFv3 (broken in most cases!)
> >>>>>>>>>             - DHCPv6 Prefix Delegation
> >>>>>>>>>
> >>>>>>>>>             BGP and/or Static routes are still the best bet here.
> >>>>>>>>>
> >>>>>>>>>             So what you do is that you tell CloudStack that you
> will route
> >>>>>>>>>             2001:db8::/48 to the VR, the VR can then use that to
> split it
> >>>>>> up
> >>>>>>>>>             into
> >>>>>>>>>             multiple /64 subnets going towards the instances:
> >>>>>>>>>
> >>>>>>>>>             - 2001:db8::/64
> >>>>>>>>>             - 2001:db8:1::/64
> >>>>>>>>>             - 2001:db8:2::/64
> >>>>>>>>>             ...
> >>>>>>>>>             - 2001:db8:f::/64
> >>>>>>>>>
> >>>>>>>>>             And go on.
> >>>>>>>>>
> >>>>>>>>>             In case of BGP you indeed have to tell the VR a few
> things:
> >>>>>>>>>
> >>>>>>>>>             - It's own AS number
> >>>>>>>>>             - The peer's address(es)
> >>>>>>>>>
> >>>>>>>>>             With FRR you can simply say:
> >>>>>>>>>
> >>>>>>>>>             neighbor 2001:db8:4fa::179 remote-as external
> >>>>>>>>>
> >>>>>>>>>             The /48 you need to have at the VR anyway in case of
> either a
> >>>>>>>>>             static
> >>>>>>>>>             route or BGP.
> >>>>>>>>>
> >>>>>>>>>             We just need to add a NullRoute on the VR for that
> /48 so that
> >>>>>>>>>             traffic
> >>>>>>>>>             will not be routed to the upper gateway in case of
> the VR
> >>>>>> can't
> >>>>>>>>>             find a
> >>>>>>>>>             route.
> >>>>>>>>>
> >>>>>>>>>             Wido
> >>>>>>>>>
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli
> >>>>>>>>>              > <Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>             <mailto:Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> wrote:
> >>>>>>>>>              >
> >>>>>>>>>              >     Hi Wido,
> >>>>>>>>>              >     That's pretty much in line with our thoughts,
> thanks
> >>>>>> for
> >>>>>>>>>             the input.
> >>>>>>>>>              >     I believe we agree on the following points
> then:
> >>>>>>>>>              >
> >>>>>>>>>              >     - FRR with BGP (no OSPF)
> >>>>>>>>>              >     - Route /48 (or/56) down to the VR
> >>>>>>>>>              >     - /64 per network
> >>>>>>>>>              >     - SLACC for IP addressing
> >>>>>>>>>              >
> >>>>>>>>>              >     I believe the next big question is then "on
> which level
> >>>>>>>>>             of ACS do we
> >>>>>>>>>              >     manage AS numbers?".  I see two options:
> >>>>>>>>>              >     1) Private AS number on a per-zone basis
> >>>>>>>>>              >     2) Root Admin assigned AS number on a
> domain/account
> >>>>>> basis
> >>>>>>>>>              >     3) End-user driven AS number on a per network
> basis
> >>>>>> (for
> >>>>>>>>>             bring your
> >>>>>>>>>              >     own AS and IP scenario)
> >>>>>>>>>              >
> >>>>>>>>>              >     Thoughts?
> >>>>>>>>>              >
> >>>>>>>>>              >     Cheers
> >>>>>>>>>              >     Alex
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >     -----Original Message-----
> >>>>>>>>>              >     From: Wido den Hollander <wido@widodh.nl
> >>>>>>>>>             <ma...@widodh.nl> <mailto:wido@widodh.nl
> >>>>>>>>>             <ma...@widodh.nl>>>
> >>>>>>>>>              >     Sent: 13 July 2021 15:08
> >>>>>>>>>              >     To: dev@cloudstack.apache.org
> >>>>>>>>>             <ma...@cloudstack.apache.org>
> >>>>>>>>>             <mailto:dev@cloudstack.apache.org
> >>>>>>>>>             <ma...@cloudstack.apache.org>>;
> >>>>>>>>>              >     Alex Mattioli <Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>
> >>>>>>>>>              >     Cc: Wei Zhou <Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>; Rohit Yadav
> >>>>>>>>>              >     <rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>             <mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>;
> >>>>>>>>>              >     Gabriel Beims Bräscher <gabriel@pcextreme.nl
> >>>>>>>>>             <ma...@pcextreme.nl>
> >>>>>>>>>              >     <mailto:gabriel@pcextreme.nl <mailto:
> >>>>>> gabriel@pcextreme.nl
> >>>>>>>>>>>
> >>>>>>>>>              >     Subject: Re: IPV6 in Isolated/VPC networks
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >     On 7/7/21 1:16 PM, Alex Mattioli wrote:
> >>>>>>>>>              >      > Hi all,
> >>>>>>>>>              >      > @Wei Zhou<mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> @Rohit
> >>>>>>>>>              >     Yadav<mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> and myself are
> >>>>>>>>>             investigating how
> >>>>>>>>>              >     to enable IPV6 support on Isolated and VPC
> networks and
> >>>>>>>>>             would like
> >>>>>>>>>              >     your input on it.
> >>>>>>>>>              >      > At the moment we are looking at
> implementing FRR
> >>>>>> with
> >>>>>>>>>             BGP (and
> >>>>>>>>>              >     possibly OSPF) on the ACS VR.
> >>>>>>>>>              >      >
> >>>>>>>>>              >      > We are looking for requirements,
> recommendations,
> >>>>>>>>>             ideas, rants,
> >>>>>>>>>              >     etc...etc...
> >>>>>>>>>              >      >
> >>>>>>>>>              >
> >>>>>>>>>              >     Ok! Here we go.
> >>>>>>>>>              >
> >>>>>>>>>              >     I think that you mean that the VR will
> actually route
> >>>>>> the
> >>>>>>>>>             IPv6
> >>>>>>>>>              >     traffic and for that you need to have a way
> of getting
> >>>>>> a
> >>>>>>>>>             subnet
> >>>>>>>>>              >     routed to the VR.
> >>>>>>>>>              >
> >>>>>>>>>              >     BGP is probably you best bet here. Although
> OSPFv3
> >>>>>>>>>             technically
> >>>>>>>>>              >     supports this it is very badly implemented in
> Frr for
> >>>>>>>>>             example.
> >>>>>>>>>              >
> >>>>>>>>>              >     Now FRR is a very good router and one of the
> fancy
> >>>>>>>>>             features it
> >>>>>>>>>              >     supports is BGP Unnumered. This allows for
> auto
> >>>>>>>>>             configuration of BGP
> >>>>>>>>>              >     over a L2 network when both sides are sending
> Router
> >>>>>>>>>             Advertisements.
> >>>>>>>>>              >     This is very easy for flexible BGP
> configurations where
> >>>>>>>>>             both sides
> >>>>>>>>>              >     have dynamic IPs.
> >>>>>>>>>              >
> >>>>>>>>>              >     What you want to do is that you get a /56,
> /48 or
> >>>>>>>>>             something which is
> >>>>>>>>>              >      >/64 bits routed to the VR.
> >>>>>>>>>              >
> >>>>>>>>>              >     Now you can sub-segment this into separate
> /64 subnets.
> >>>>>>>>>             You don't
> >>>>>>>>>              >     want to go smaller then a /64 is that
> prevents you from
> >>>>>>>>>             using SLAAC
> >>>>>>>>>              >     for IPv6 address configuration. This is how
> it works
> >>>>>> for
> >>>>>>>>>             Shared
> >>>>>>>>>              >     Networks now in Basic and Advanced Zones.
> >>>>>>>>>              >
> >>>>>>>>>              >     FRR can now also send out the Router
> Advertisements on
> >>>>>>>>>             the downlinks
> >>>>>>>>>              >     sending out:
> >>>>>>>>>              >
> >>>>>>>>>              >     - DNS servers
> >>>>>>>>>              >     - DNS domain
> >>>>>>>>>              >     - Prefix (/64) to be used
> >>>>>>>>>              >
> >>>>>>>>>              >     There is no need for DHCPv6. You can
> calculate the IPv6
> >>>>>>>>>             address the
> >>>>>>>>>              >     VM will obtain by using the MAC and the
> prefix.
> >>>>>>>>>              >
> >>>>>>>>>              >     So in short:
> >>>>>>>>>              >
> >>>>>>>>>              >     - Using BGP you routed a /48 to the VR
> >>>>>>>>>              >     - Now you split this into /64 subnets towards
> the
> >>>>>>>>>             isolated networks
> >>>>>>>>>              >
> >>>>>>>>>              >     Wido
> >>>>>>>>>              >
> >>>>>>>>>              >      > Alex Mattioli
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              > --
> >>>>>>>>>              > Regards,
> >>>>>>>>>              > Hean Seng
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         --
> >>>>>>>>>         Regards,
> >>>>>>>>>         Hean Seng
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Regards,
> >>>>>>>>> Hean Seng
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>

Re: IPV6 in Isolated/VPC networks

Posted by Kristaps Cudars <kr...@gmail.com>.
Hi Rohit,

I think its excellent and satisfies all involved sides.

We will be in front seats bringing this to production.


Suggestion:

This

*   At the zone level root-admin specifies a /64 public range that will be
used for VRs, then

they can add a /48, or /56 IPv6 range for guest networks (to be used by
isolated networks and VPC tiers)


To

*   At the zone level root-admin specifies a /64 public range that will be
used for VRs, then

they can add one or more ranges equal or larger than /64 IPv6 and ACS will
subnet it in /64s

for Isolated networks or VPC tiers.


Way:

RIPE NCC in case of PI (Provider Independent) range is granting /48 and its
important to have

possibility of assigning several subnets, for example /58 to be used for
internal /64 allocations.

If you want to extend your /48 you will need to have good explanation why
you want it.
RIPE NCC: Organisations requesting an IPv6 PI assignment larger than a /48
need to provide

additional documentation to justify the larger assignment size.

The RIPE NCC will evaluate the PI assignment request and assign an
independent IPv6 prefix

to the End User organization directly.

On Wed, 11 Aug 2021 at 15:26, Rohit Yadav <ro...@shapeblue.com> wrote:

> Hi all,
>
> Thanks for your feedback and ideas, I've gone ahead with discussing them
> with Alex and came up with a PoC/design which can be implemented in the
> following phases:
>
>   *   Phase1: implement ipv6 support in isolated networks and VPC with
> static routing
>   *   Phase2: discuss and implement support for dynamic routing (TBD)
>
> For Phase1 here's the high-level proposal:
>
>   *   IPv6 address management:
>      *   At the zone level root-admin specifies a /64 public range that
> will be used for VRs, then they can add a /48, or /56 IPv6 range for guest
> networks (to be used by isolated networks and VPC tiers)
>      *   On creation of any IPv6 enabled isolated network or VPC tier,
> from the /48 or /56 block a /64 network is allocated/used
>      *   We assume SLAAC and autoconfiguration, no DHCPv6 in the zone
> (discuss: is privacy a concern, can privacy extensions rfc4941 of slaac be
> explored?)
>   *   Network offerings: root-admin can create new network offerings (with
> VPC too) that specifies a network stack option:
>      *   ipv4 only (default, for backward compatibility all
> networks/offerings post-upgrade migrate to this option)
>      *   ipv4-and-ipv6
>      *   ipv6-only (this can be phase 1.b)
>      *   A new routing option: static (phase1), dynamic (phase2, with
> multiple sub-options such as ospf/bgp etc...)
>   *   VR changes:
>      *   VR gets its guest and public nics set to inet6 auto
>      *   For each /64 allocated to guest network and VPC tiers, radvd is
> configured to do RA
>      *   Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6
> firewall rules (ACLs, ingress, egress) are implemented; ACLs between VPC
> tiers are managed/implemented by ipv6 firewall on VR
>      *   It is assumed that static routes are created on the core/main
> router by the admin or automated using some scripts/tools; for this
> CloudStack will announce events with details of /64 networks and VR's
> public IPv6 address that can be consumed by a rabbitmq/message bus client
> (for example), or a custom cron job or script as part of orchestration.
> (this wouldn't be necessary for dynamic routing bgp with phase2)
>   *   Guest Networking: With SLAAC, it's easy for CloudStack to calculate
> allocate and use a /64 and determine the IPv6 address of VR nics and guest
> VM nics
>      *   A user create an isolated network/VPC with an offering that is
> ipv6 enabled
>      *   A user can manage firewall for the IPv6 address/guest nics;
> there'll be no port forward and LB feature though for IPv6
>      *   A users can run workloads in the guest VMs that listen on
> publically routable ipv6 addresses
>      *   Usage/billing etc continue to work, no change needed
>
> Network layout:
>
> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] ->
> [guest VMs/nics]
> *core/ISP router needs static routes to be added (manually or automated),
> assumes a /48 or /56 configured for the zone
>
> Thoughts, feedback?
>
> Proof-of-concept commentary: here's what I did to test the idea:
>
>   *   Created an isolated network and deployed a VM in my home lab
> The VR running on KVM has following nics
> eth0 - guest network
> eth1 - link local
> eth2 - public network
>
>   *   I setup a custom openwrt router on a RPi4 to serve as a toy-core
> router where I create a wan6 IPv6 tunnel using tunnel broker and I got a
> /48 allocated. My configuration looks like:
> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> /64 - 2001:470:36:3e2::/64 (default allocated by)
>
> I create a LAN ipv6 (public network for CloudStack VR): at subnet/prefix 0:
> LAN IPv6 address: 2001:470:ed36:0::1/64
> Address mode: SLAAC+stateless DHCP (no dhcpv6)
>   *
>   *
> In the isolated VR, I enabled ipv6 as:
> net.ipv6.conf.all.disable_ipv6 = 0
> net.ipv6.conf.all.forwarding = 1
> net.ipv6.conf.all.accept_ra = 1
> net.ipv6.conf.all.accept_redirects = 1
> net.ipv6.conf.all.autoconf = 1
>
> Set up a IPv6 nameserver/dns in /etc/resolve.conf
> And configured the nics:
> echo iface eth0 inet6 auto >> /etc/network/interfaces
> echo iface eth2 inet6 auto >> /etc/network/interfaces
> /etc/init.d/networking restart
> Next, restart ACS isolated network without cleanup to have it reconfigure
> IPv4 nics, firewall, NAT etc
>
>   *
> Next, I created a /64 network for the isolated guest network on eth0 of VR
> using radvd:
>
> # cat /etc/radvd.conf
> interface eth0
> {
>     AdvSendAdvert on;
>     MinRtrAdvInterval 5;
>     MaxRtrAdvInterval 15;
>     prefix 2001:470:ed36:1::/64
>     {
>         AdvOnLink on;
>         AdvAutonomous on;
>     };
> };
> systemctl restart radvd
> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this
> ...:1::/64 network
>   *   Finally I added a static route in toy core-router for the new /64
> IPv6 range in the isolated network
> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev
> <local lan nic>
>   *
> ... and I enabled firewall rules to allow any traffic to pass for the new
> /64 network
>
> And voila all done! I create a domain AAAA record that points to my guest
> VM IPv6 address a test webserver on
> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>
> (Note: I'll get rid of the tunnel and request a new /48 block after a few
> days, sharing this solely for testing purposes)
>
>
> Regards.
>
> ________________________________
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: Tuesday, July 20, 2021 12:46
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
>
>
> Op 19-07-2021 om 20:38 schreef Kristaps Cudars:
> > Hi Wido,
> >
> > I assume that flouting ip will not work grate with ingress/egress acl on
> VR.
> >
> >  From regular ACS user perspective:
> > I have Instance with dualstack its running web app on 443.
> > I want to swap instances for whatever reason.
> > In case of IPv4 change d-nat rule.
> > In case of IPv6 if flouting IP was not created upfront he will need to
> change dns entry that usually has 24h ttl. Inconvenience degradation in
> experience.
> >
>
> Yes, but, keep in mind that the IP you are using can also be terminated
> on the VR where HAProxy proxies request to the backend VM (could even be
> v4!)
>
> I'm not against DHCPv6, but I have seen many issues with implementing
> it. Therefor I always stick to SLAAC.
>
> >  From ACS admin perspective:
> > I don’t want to have these tickets in helpdesk.
> > You needed to create another flouting IP that it would be seamless- will
> not work as answer.
> >
>
> I understand that as well.
>
> Wido
>
> >
> > On 2021/07/19 09:05:54, Wido den Hollander <wi...@widodh.nl> wrote:
> >>
> >>
> >> Op 16-07-2021 om 21:46 schreef Kristaps Cudars:
> >>> Hi Wido,
> >>>
> >>> Your proposal is to sacrifice ability to reassign IPv6 to instance,
> have internal domain prefix, and list/db in ACS what IPv6 has been assigned
> to what instance and go with RA and SLAAC. For route signaling to switch
> use BGP/OSPFv3 or manual pre-creation.
> >>>
> >>
> >> You can still list the IPs which have been assigned. You'll know exactly
> >> what IPv6 address a VM has because of the prefix + MAC. Privacy
> >> Extensions need to be disabled in the VM.
> >>
> >> This already works in CloudStack in Shared Networks in this way.
> >>
> >> Using secondary IPs you can always have 'floating' IPv6 addressess.
> >>
> >> Wido
> >>
> >>> Option with RA and managed flag that DHCPv6 is in use to support
> preset information and ability to create route information from ACS is not
> an option as DHCPv6 its failing?
> >>>
> >>>
> >>> On 2021/07/16 15:17:42, Wido den Hollander <wi...@widodh.nl> wrote:
> >>>>
> >>>>
> >>>> Op 16-07-2021 om 16:42 schreef Hean Seng:
> >>>>> Hi Wido,
> >>>>>
> >>>>> In current setup,  each Cloudstack have own VR, so in this new  IPv6
> subnet
> >>>>> allocation , each VR (which have Frr) will need to have peering with
> ISP
> >>>>> router (and either BGP or Static Route) , and there is 1000
> Acocunts,  it
> >>>>> will 1000 BGP session with ISP router ,  Am I right for this ? or I
> >>>>> understand wrong .
> >>>>>
> >>>>
> >>>> Yes, that is correct. A /56 would also be sufficient or a /60 which is
> >>>> enough to allocate a few /64 subnets.
> >>>>
> >>>> 1000 BGP connections isn't really a problem for a proper router at the
> >>>> ISP. OSPF(v3) would be better, but as I said that's poorly supported.
> >>>>
> >>>> The ISP could also install 1000 static routes, but that means that the
> >>>> ISP's router needs to have those configured.
> >>>>
> >>>> http://docs.frrouting.org/en/latest/ospf6d.html
> >>>> (While looking up this URL I see that Frr recently put in a lot of
> work
> >>>> in OSPFv3, seems better now)
> >>>>
> >>>>> I understand IPv6 is different then IPv4, and in IPv6 it suppose each
> >>>>> devices have own IP. It just how to realize in easy way.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
>
>
> > On Fri, Jul 16, 2021 at 8:17 PM Wido den Hollander <wi...@widodh.nl>
> wrote:
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> Op 16-07-2021 om 05:54 schreef Hean Seng:
> >>>>>>> Hi Wido,
> >>>>>>>
> >>>>>>> My initial thought is not like this,  it is the /48 at ISP router,
> and
> >>>>>> /64
> >>>>>>> subnet assign to AdvanceZoneVR,   AdvanceZoneVR responsible is
> >>>>>>> distribule IPv6 ip (from the assigned /64 sunet) to VM,  and not
> routing
> >>>>>>> the traffic,   in the VM that get the IPv6 IP will default route
> to ISP
> >>>>>>> router as gw.   It can may be a bridge over via Advancezone-VR.
> >>>>>>>
> >>>>>>
> >>>>>> How would you bridge this? That sounds like NAT?
> >>>>>>
> >>>>>> IPv6 is meant to be routed. Not to be translated or bridged in any
> way.
> >>>>>>
> >>>>>> The way a made the drawing is exactly how IPv6 should work in a VPC
> >>>>>> environment.
> >>>>>>
> >>>>>> Traffic flows through the VR where it can do firewalling of the
> traffic.
> >>>>>>
> >>>>>>> However, If do as the way described in the drawing, then i suppose
> will
> >>>>>> be
> >>>>>>> another kind of virtual router going to introduce , to get hold
> the /48
> >>>>>> in
> >>>>>>> this virtual router right ?
> >>>>>>>
> >>>>>>
> >>>>>> It can be the same VR. But keep in mind that IPv6 != IPv4.
> >>>>>>
> >>>>>> The VR will get Frr as a new daemon which can talk BGP with the
> upper
> >>>>>> network to route traffic.
> >>>>>>
> >>>>>>> After this,  The Advance Zone, NAT's  VR will peer with this new
> IPv6 VR
> >>>>>>> for getting the IPv6 /64 prefix ?
> >>>>>>>
> >>>>>>
> >>>>>> IPv4 will be behind NAT, but IPv6 will not be behind NAT.
> >>>>>>
> >>>>>>> If do in this way, then I guess  you just only need Static route,
> with
> >>>>>>> peering ip both end  as one /48 can have a lot of /64 on it.  And
> >>>>>> hardware
> >>>>>>> budgeting for new IPv6-VR will become very important, as all
> traffic will
> >>>>>>> need to pass over it .
> >>>>>>>
> >>>>>>
> >>>>>> Routing or NAT is the same for the VR. You don't need a very beefy
> VR
> >>>>>> for this.
> >>>>>>
> >>>>>>> It will be like
> >>>>>>>
> >>>>>>> ISP Router  ------ >  (new IPV6-VR ) ---- > AdvanceZone-VR ----> VM
> >>>>>>>
> >>>>>>> Relationship of (new IPv6 VR) and AdvanceZone-VR , may be
> considering on
> >>>>>>> OSPF instead of  BGP , otherwise few thousand of AdvanceZone-VR
> wil have
> >>>>>>> few thousand of BGP session. on new-IPv6-VR
> >>>>>>>
> >>>>>>> Also, I suppose we cannot do ISP router. -->. Advancezone VR
> direct,   ,
> >>>>>>> otherwise ISP router will be full of /64 prefix route either on
> BGP( Many
> >>>>>>> BGP Session) , or  Many Static route .   If few thousand account,
> ti will
> >>>>>>> be few thousand of BGP session with ISP router or few thousand
> static
> >>>>>> route
> >>>>>>> which  is not possible .
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Jul 15, 2021 at 10:47 PM Wido den Hollander <
> wido@widodh.nl>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> But you still need routing. See the attached PNG (and draw.io
> XML).
> >>>>>>>>
> >>>>>>>> You need to route the /48 subnet TO the VR which can then route
> it to
> >>>>>>>> the Virtual Networks behind the VR.
> >>>>>>>>
> >>>>>>>> There is no other way then routing with either BGP or a Static
> route.
> >>>>>>>>
> >>>>>>>> Wido
> >>>>>>>>
> >>>>>>>> Op 15-07-2021 om 12:39 schreef Hean Seng:
> >>>>>>>>> Or explain like this :
> >>>>>>>>>
> >>>>>>>>> 1) Cloudstack generate list of /64 subnet from /48 that Network
> admin
> >>>>>>>>> assigned to Cloudstack
> >>>>>>>>> 2) Cloudsack allocated the subnet (that generated from step1) to
> >>>>>> Virtual
> >>>>>>>>> Router, one Virtual Router have one subniet /64
> >>>>>>>>> 3) Virtual Router allocate single IPv6 (within the range of /64
> >>>>>>>>> allocated to VR)  to VM
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng <heanseng@gmail.com
> >>>>>>>>> <ma...@gmail.com>> wrote:
> >>>>>>>>>
> >>>>>>>>>         Hi Wido,
> >>>>>>>>>
> >>>>>>>>>         I think the /48 is at physical router as gateway , and
> subnet of
> >>>>>> /64
> >>>>>>>>>         at VR of Cloudstack.   Cloudstack only keep which /48
> prefix and
> >>>>>>>>>         vlan information of this /48 to be later split the  /64.
> to VR.
> >>>>>>>>>
> >>>>>>>>>         And the instances is getting singe IPv6 of /64  IP.
>  The VR is
> >>>>>>>>>         getting /64.  The default gateway shall goes to /48 of
> physical
> >>>>>>>>>         router ip .   In this case ,does not need any BGP router
> .
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         Similar concept as IPv4 :
> >>>>>>>>>
> >>>>>>>>>         /48 subnet of IPv6 is equivalent to current /24 subnet
> of IPv4
> >>>>>> that
> >>>>>>>>>         created in Network.
> >>>>>>>>>         and /64  of IPv6 is equivalent to single IP of IPv4
> assign to VM.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander <
> >>>>>> wido@widodh.nl
> >>>>>>>>>         <ma...@widodh.nl>> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>             Op 14-07-2021 om 16:44 schreef Hean Seng:
> >>>>>>>>>              > Hi
> >>>>>>>>>              >
> >>>>>>>>>              > I replied in another thread, i think do not need
> implement
> >>>>>>>>>             BGP or OSPF,
> >>>>>>>>>              > that would be complicated .
> >>>>>>>>>              >
> >>>>>>>>>              > We only need assign  IPv6 's /64 prefix to
> Virtual Router
> >>>>>>>>>             (VR) in NAT
> >>>>>>>>>              > zone, and the VR responsible to deliver single
> IPv6 to VM
> >>>>>> via
> >>>>>>>>>             DHCP6.
> >>>>>>>>>              >
> >>>>>>>>>              > In VR, you need to have Default IPv6 route to
> Physical
> >>>>>>>>>             Router's /48. IP
> >>>>>>>>>              > as IPv6 Gateway.  Thens should be done .
> >>>>>>>>>              >
> >>>>>>>>>              > Example :
> >>>>>>>>>              > Physical Router Interface
> >>>>>>>>>              >   IPv6 IP : 2000:aaaa::1/48
> >>>>>>>>>              >
> >>>>>>>>>              > Cloudstack  virtual router :
> 2000:aaaa:200:201::1/64 with
> >>>>>>>>>             default ipv6
> >>>>>>>>>              > route to router ip 2000:aaaa::1
> >>>>>>>>>              > and Clodustack Virtual router dhcp allocate IP to
> VM , and
> >>>>>>>>>             VM will have
> >>>>>>>>>              > default route to VR. IPv6 2000:aaaa:200:201::1
> >>>>>>>>>              >
> >>>>>>>>>              > So in cloudstack need to allow  user to enter ,
> IPv6
> >>>>>>>>>             gwateway , and
> >>>>>>>>>              > the  /48 Ipv6 prefix , then it will self allocate
> the /64
> >>>>>> ip
> >>>>>>>>>             to the VR ,
> >>>>>>>>>              > and maintain make sure not ovelap allocation
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>
> >>>>>>>>>             But NAT is truly not the solution with IPv6. IPv6 is
> supposed
> >>>>>> to
> >>>>>>>> be
> >>>>>>>>>             routable. In addition you should avoid DHCPv6 as
> much as
> >>>>>>>>>             possible as
> >>>>>>>>>             that's not really the intended use-case for address
> allocation
> >>>>>>>>>             with IPv6.
> >>>>>>>>>
> >>>>>>>>>             In order to route an /48 IPv6 subnet to the VR you
> have a few
> >>>>>>>>>             possibilities:
> >>>>>>>>>
> >>>>>>>>>             - Static route from the upperlying routers which are
> outside
> >>>>>> of
> >>>>>>>>>             CloudStack
> >>>>>>>>>             - BGP
> >>>>>>>>>             - OSPFv3 (broken in most cases!)
> >>>>>>>>>             - DHCPv6 Prefix Delegation
> >>>>>>>>>
> >>>>>>>>>             BGP and/or Static routes are still the best bet here.
> >>>>>>>>>
> >>>>>>>>>             So what you do is that you tell CloudStack that you
> will route
> >>>>>>>>>             2001:db8::/48 to the VR, the VR can then use that to
> split it
> >>>>>> up
> >>>>>>>>>             into
> >>>>>>>>>             multiple /64 subnets going towards the instances:
> >>>>>>>>>
> >>>>>>>>>             - 2001:db8::/64
> >>>>>>>>>             - 2001:db8:1::/64
> >>>>>>>>>             - 2001:db8:2::/64
> >>>>>>>>>             ...
> >>>>>>>>>             - 2001:db8:f::/64
> >>>>>>>>>
> >>>>>>>>>             And go on.
> >>>>>>>>>
> >>>>>>>>>             In case of BGP you indeed have to tell the VR a few
> things:
> >>>>>>>>>
> >>>>>>>>>             - It's own AS number
> >>>>>>>>>             - The peer's address(es)
> >>>>>>>>>
> >>>>>>>>>             With FRR you can simply say:
> >>>>>>>>>
> >>>>>>>>>             neighbor 2001:db8:4fa::179 remote-as external
> >>>>>>>>>
> >>>>>>>>>             The /48 you need to have at the VR anyway in case of
> either a
> >>>>>>>>>             static
> >>>>>>>>>             route or BGP.
> >>>>>>>>>
> >>>>>>>>>             We just need to add a NullRoute on the VR for that
> /48 so that
> >>>>>>>>>             traffic
> >>>>>>>>>             will not be routed to the upper gateway in case of
> the VR
> >>>>>> can't
> >>>>>>>>>             find a
> >>>>>>>>>             route.
> >>>>>>>>>
> >>>>>>>>>             Wido
> >>>>>>>>>
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli
> >>>>>>>>>              > <Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>             <mailto:Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> wrote:
> >>>>>>>>>              >
> >>>>>>>>>              >     Hi Wido,
> >>>>>>>>>              >     That's pretty much in line with our thoughts,
> thanks
> >>>>>> for
> >>>>>>>>>             the input.
> >>>>>>>>>              >     I believe we agree on the following points
> then:
> >>>>>>>>>              >
> >>>>>>>>>              >     - FRR with BGP (no OSPF)
> >>>>>>>>>              >     - Route /48 (or/56) down to the VR
> >>>>>>>>>              >     - /64 per network
> >>>>>>>>>              >     - SLACC for IP addressing
> >>>>>>>>>              >
> >>>>>>>>>              >     I believe the next big question is then "on
> which level
> >>>>>>>>>             of ACS do we
> >>>>>>>>>              >     manage AS numbers?".  I see two options:
> >>>>>>>>>              >     1) Private AS number on a per-zone basis
> >>>>>>>>>              >     2) Root Admin assigned AS number on a
> domain/account
> >>>>>> basis
> >>>>>>>>>              >     3) End-user driven AS number on a per network
> basis
> >>>>>> (for
> >>>>>>>>>             bring your
> >>>>>>>>>              >     own AS and IP scenario)
> >>>>>>>>>              >
> >>>>>>>>>              >     Thoughts?
> >>>>>>>>>              >
> >>>>>>>>>              >     Cheers
> >>>>>>>>>              >     Alex
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >     -----Original Message-----
> >>>>>>>>>              >     From: Wido den Hollander <wido@widodh.nl
> >>>>>>>>>             <ma...@widodh.nl> <mailto:wido@widodh.nl
> >>>>>>>>>             <ma...@widodh.nl>>>
> >>>>>>>>>              >     Sent: 13 July 2021 15:08
> >>>>>>>>>              >     To: dev@cloudstack.apache.org
> >>>>>>>>>             <ma...@cloudstack.apache.org>
> >>>>>>>>>             <mailto:dev@cloudstack.apache.org
> >>>>>>>>>             <ma...@cloudstack.apache.org>>;
> >>>>>>>>>              >     Alex Mattioli <Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>
> >>>>>>>>>              >     Cc: Wei Zhou <Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>; Rohit Yadav
> >>>>>>>>>              >     <rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>             <mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>;
> >>>>>>>>>              >     Gabriel Beims Bräscher <gabriel@pcextreme.nl
> >>>>>>>>>             <ma...@pcextreme.nl>
> >>>>>>>>>              >     <mailto:gabriel@pcextreme.nl <mailto:
> >>>>>> gabriel@pcextreme.nl
> >>>>>>>>>>>
> >>>>>>>>>              >     Subject: Re: IPV6 in Isolated/VPC networks
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >     On 7/7/21 1:16 PM, Alex Mattioli wrote:
> >>>>>>>>>              >      > Hi all,
> >>>>>>>>>              >      > @Wei Zhou<mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> @Rohit
> >>>>>>>>>              >     Yadav<mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> and myself are
> >>>>>>>>>             investigating how
> >>>>>>>>>              >     to enable IPV6 support on Isolated and VPC
> networks and
> >>>>>>>>>             would like
> >>>>>>>>>              >     your input on it.
> >>>>>>>>>              >      > At the moment we are looking at
> implementing FRR
> >>>>>> with
> >>>>>>>>>             BGP (and
> >>>>>>>>>              >     possibly OSPF) on the ACS VR.
> >>>>>>>>>              >      >
> >>>>>>>>>              >      > We are looking for requirements,
> recommendations,
> >>>>>>>>>             ideas, rants,
> >>>>>>>>>              >     etc...etc...
> >>>>>>>>>              >      >
> >>>>>>>>>              >
> >>>>>>>>>              >     Ok! Here we go.
> >>>>>>>>>              >
> >>>>>>>>>              >     I think that you mean that the VR will
> actually route
> >>>>>> the
> >>>>>>>>>             IPv6
> >>>>>>>>>              >     traffic and for that you need to have a way
> of getting
> >>>>>> a
> >>>>>>>>>             subnet
> >>>>>>>>>              >     routed to the VR.
> >>>>>>>>>              >
> >>>>>>>>>              >     BGP is probably you best bet here. Although
> OSPFv3
> >>>>>>>>>             technically
> >>>>>>>>>              >     supports this it is very badly implemented in
> Frr for
> >>>>>>>>>             example.
> >>>>>>>>>              >
> >>>>>>>>>              >     Now FRR is a very good router and one of the
> fancy
> >>>>>>>>>             features it
> >>>>>>>>>              >     supports is BGP Unnumered. This allows for
> auto
> >>>>>>>>>             configuration of BGP
> >>>>>>>>>              >     over a L2 network when both sides are sending
> Router
> >>>>>>>>>             Advertisements.
> >>>>>>>>>              >     This is very easy for flexible BGP
> configurations where
> >>>>>>>>>             both sides
> >>>>>>>>>              >     have dynamic IPs.
> >>>>>>>>>              >
> >>>>>>>>>              >     What you want to do is that you get a /56,
> /48 or
> >>>>>>>>>             something which is
> >>>>>>>>>              >      >/64 bits routed to the VR.
> >>>>>>>>>              >
> >>>>>>>>>              >     Now you can sub-segment this into separate
> /64 subnets.
> >>>>>>>>>             You don't
> >>>>>>>>>              >     want to go smaller then a /64 is that
> prevents you from
> >>>>>>>>>             using SLAAC
> >>>>>>>>>              >     for IPv6 address configuration. This is how
> it works
> >>>>>> for
> >>>>>>>>>             Shared
> >>>>>>>>>              >     Networks now in Basic and Advanced Zones.
> >>>>>>>>>              >
> >>>>>>>>>              >     FRR can now also send out the Router
> Advertisements on
> >>>>>>>>>             the downlinks
> >>>>>>>>>              >     sending out:
> >>>>>>>>>              >
> >>>>>>>>>              >     - DNS servers
> >>>>>>>>>              >     - DNS domain
> >>>>>>>>>              >     - Prefix (/64) to be used
> >>>>>>>>>              >
> >>>>>>>>>              >     There is no need for DHCPv6. You can
> calculate the IPv6
> >>>>>>>>>             address the
> >>>>>>>>>              >     VM will obtain by using the MAC and the
> prefix.
> >>>>>>>>>              >
> >>>>>>>>>              >     So in short:
> >>>>>>>>>              >
> >>>>>>>>>              >     - Using BGP you routed a /48 to the VR
> >>>>>>>>>              >     - Now you split this into /64 subnets towards
> the
> >>>>>>>>>             isolated networks
> >>>>>>>>>              >
> >>>>>>>>>              >     Wido
> >>>>>>>>>              >
> >>>>>>>>>              >      > Alex Mattioli
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              > --
> >>>>>>>>>              > Regards,
> >>>>>>>>>              > Hean Seng
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         --
> >>>>>>>>>         Regards,
> >>>>>>>>>         Hean Seng
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Regards,
> >>>>>>>>> Hean Seng
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>

Re: IPV6 in Isolated/VPC networks

Posted by Hean Seng <he...@gmail.com>.
I am very welcome on this ,  the near future may have almost all out of
IPv4, and will going to dependent on IPv6:

    *   ipv4 only (default, for backward compatibility all
networks/offerings post-upgrade migrate to this option)
     *   ipv4-and-ipv6
     *   ipv6-only (this can be phase 1.b)

Also allow change of IPv6  IP , and  Secondary IPv6 as well, especially for
current Shared Network IPv6.




On Wed, Aug 11, 2021 at 8:27 PM Rohit Yadav <ro...@shapeblue.com>
wrote:

> Hi all,
>
> Thanks for your feedback and ideas, I've gone ahead with discussing them
> with Alex and came up with a PoC/design which can be implemented in the
> following phases:
>
>   *   Phase1: implement ipv6 support in isolated networks and VPC with
> static routing
>   *   Phase2: discuss and implement support for dynamic routing (TBD)
>
> For Phase1 here's the high-level proposal:
>
>   *   IPv6 address management:
>      *   At the zone level root-admin specifies a /64 public range that
> will be used for VRs, then they can add a /48, or /56 IPv6 range for guest
> networks (to be used by isolated networks and VPC tiers)
>      *   On creation of any IPv6 enabled isolated network or VPC tier,
> from the /48 or /56 block a /64 network is allocated/used
>      *   We assume SLAAC and autoconfiguration, no DHCPv6 in the zone
> (discuss: is privacy a concern, can privacy extensions rfc4941 of slaac be
> explored?)
>   *   Network offerings: root-admin can create new network offerings (with
> VPC too) that specifies a network stack option:
>      *   ipv4 only (default, for backward compatibility all
> networks/offerings post-upgrade migrate to this option)
>      *   ipv4-and-ipv6
>      *   ipv6-only (this can be phase 1.b)
>      *   A new routing option: static (phase1), dynamic (phase2, with
> multiple sub-options such as ospf/bgp etc...)
>   *   VR changes:
>      *   VR gets its guest and public nics set to inet6 auto
>      *   For each /64 allocated to guest network and VPC tiers, radvd is
> configured to do RA
>      *   Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6
> firewall rules (ACLs, ingress, egress) are implemented; ACLs between VPC
> tiers are managed/implemented by ipv6 firewall on VR
>      *   It is assumed that static routes are created on the core/main
> router by the admin or automated using some scripts/tools; for this
> CloudStack will announce events with details of /64 networks and VR's
> public IPv6 address that can be consumed by a rabbitmq/message bus client
> (for example), or a custom cron job or script as part of orchestration.
> (this wouldn't be necessary for dynamic routing bgp with phase2)
>   *   Guest Networking: With SLAAC, it's easy for CloudStack to calculate
> allocate and use a /64 and determine the IPv6 address of VR nics and guest
> VM nics
>      *   A user create an isolated network/VPC with an offering that is
> ipv6 enabled
>      *   A user can manage firewall for the IPv6 address/guest nics;
> there'll be no port forward and LB feature though for IPv6
>      *   A users can run workloads in the guest VMs that listen on
> publically routable ipv6 addresses
>      *   Usage/billing etc continue to work, no change needed
>
> Network layout:
>
> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] ->
> [guest VMs/nics]
> *core/ISP router needs static routes to be added (manually or automated),
> assumes a /48 or /56 configured for the zone
>
> Thoughts, feedback?
>
> Proof-of-concept commentary: here's what I did to test the idea:
>
>   *   Created an isolated network and deployed a VM in my home lab
> The VR running on KVM has following nics
> eth0 - guest network
> eth1 - link local
> eth2 - public network
>
>   *   I setup a custom openwrt router on a RPi4 to serve as a toy-core
> router where I create a wan6 IPv6 tunnel using tunnel broker and I got a
> /48 allocated. My configuration looks like:
> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> /64 - 2001:470:36:3e2::/64 (default allocated by)
>
> I create a LAN ipv6 (public network for CloudStack VR): at subnet/prefix 0:
> LAN IPv6 address: 2001:470:ed36:0::1/64
> Address mode: SLAAC+stateless DHCP (no dhcpv6)
>   *
>   *
> In the isolated VR, I enabled ipv6 as:
> net.ipv6.conf.all.disable_ipv6 = 0
> net.ipv6.conf.all.forwarding = 1
> net.ipv6.conf.all.accept_ra = 1
> net.ipv6.conf.all.accept_redirects = 1
> net.ipv6.conf.all.autoconf = 1
>
> Set up a IPv6 nameserver/dns in /etc/resolve.conf
> And configured the nics:
> echo iface eth0 inet6 auto >> /etc/network/interfaces
> echo iface eth2 inet6 auto >> /etc/network/interfaces
> /etc/init.d/networking restart
> Next, restart ACS isolated network without cleanup to have it reconfigure
> IPv4 nics, firewall, NAT etc
>
>   *
> Next, I created a /64 network for the isolated guest network on eth0 of VR
> using radvd:
>
> # cat /etc/radvd.conf
> interface eth0
> {
>     AdvSendAdvert on;
>     MinRtrAdvInterval 5;
>     MaxRtrAdvInterval 15;
>     prefix 2001:470:ed36:1::/64
>     {
>         AdvOnLink on;
>         AdvAutonomous on;
>     };
> };
> systemctl restart radvd
> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this
> ...:1::/64 network
>   *   Finally I added a static route in toy core-router for the new /64
> IPv6 range in the isolated network
> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev
> <local lan nic>
>   *
> ... and I enabled firewall rules to allow any traffic to pass for the new
> /64 network
>
> And voila all done! I create a domain AAAA record that points to my guest
> VM IPv6 address a test webserver on
> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>
> (Note: I'll get rid of the tunnel and request a new /48 block after a few
> days, sharing this solely for testing purposes)
>
>
> Regards.
>
> ________________________________
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: Tuesday, July 20, 2021 12:46
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
>
>
> Op 19-07-2021 om 20:38 schreef Kristaps Cudars:
> > Hi Wido,
> >
> > I assume that flouting ip will not work grate with ingress/egress acl on
> VR.
> >
> >  From regular ACS user perspective:
> > I have Instance with dualstack its running web app on 443.
> > I want to swap instances for whatever reason.
> > In case of IPv4 change d-nat rule.
> > In case of IPv6 if flouting IP was not created upfront he will need to
> change dns entry that usually has 24h ttl. Inconvenience degradation in
> experience.
> >
>
> Yes, but, keep in mind that the IP you are using can also be terminated
> on the VR where HAProxy proxies request to the backend VM (could even be
> v4!)
>
> I'm not against DHCPv6, but I have seen many issues with implementing
> it. Therefor I always stick to SLAAC.
>
> >  From ACS admin perspective:
> > I don’t want to have these tickets in helpdesk.
> > You needed to create another flouting IP that it would be seamless- will
> not work as answer.
> >
>
> I understand that as well.
>
> Wido
>
> >
> > On 2021/07/19 09:05:54, Wido den Hollander <wi...@widodh.nl> wrote:
> >>
> >>
> >> Op 16-07-2021 om 21:46 schreef Kristaps Cudars:
> >>> Hi Wido,
> >>>
> >>> Your proposal is to sacrifice ability to reassign IPv6 to instance,
> have internal domain prefix, and list/db in ACS what IPv6 has been assigned
> to what instance and go with RA and SLAAC. For route signaling to switch
> use BGP/OSPFv3 or manual pre-creation.
> >>>
> >>
> >> You can still list the IPs which have been assigned. You'll know exactly
> >> what IPv6 address a VM has because of the prefix + MAC. Privacy
> >> Extensions need to be disabled in the VM.
> >>
> >> This already works in CloudStack in Shared Networks in this way.
> >>
> >> Using secondary IPs you can always have 'floating' IPv6 addressess.
> >>
> >> Wido
> >>
> >>> Option with RA and managed flag that DHCPv6 is in use to support
> preset information and ability to create route information from ACS is not
> an option as DHCPv6 its failing?
> >>>
> >>>
> >>> On 2021/07/16 15:17:42, Wido den Hollander <wi...@widodh.nl> wrote:
> >>>>
> >>>>
> >>>> Op 16-07-2021 om 16:42 schreef Hean Seng:
> >>>>> Hi Wido,
> >>>>>
> >>>>> In current setup,  each Cloudstack have own VR, so in this new  IPv6
> subnet
> >>>>> allocation , each VR (which have Frr) will need to have peering with
> ISP
> >>>>> router (and either BGP or Static Route) , and there is 1000
> Acocunts,  it
> >>>>> will 1000 BGP session with ISP router ,  Am I right for this ? or I
> >>>>> understand wrong .
> >>>>>
> >>>>
> >>>> Yes, that is correct. A /56 would also be sufficient or a /60 which is
> >>>> enough to allocate a few /64 subnets.
> >>>>
> >>>> 1000 BGP connections isn't really a problem for a proper router at the
> >>>> ISP. OSPF(v3) would be better, but as I said that's poorly supported.
> >>>>
> >>>> The ISP could also install 1000 static routes, but that means that the
> >>>> ISP's router needs to have those configured.
> >>>>
> >>>> http://docs.frrouting.org/en/latest/ospf6d.html
> >>>> (While looking up this URL I see that Frr recently put in a lot of
> work
> >>>> in OSPFv3, seems better now)
> >>>>
> >>>>> I understand IPv6 is different then IPv4, and in IPv6 it suppose each
> >>>>> devices have own IP. It just how to realize in easy way.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
>
>
> > On Fri, Jul 16, 2021 at 8:17 PM Wido den Hollander <wi...@widodh.nl>
> wrote:
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> Op 16-07-2021 om 05:54 schreef Hean Seng:
> >>>>>>> Hi Wido,
> >>>>>>>
> >>>>>>> My initial thought is not like this,  it is the /48 at ISP router,
> and
> >>>>>> /64
> >>>>>>> subnet assign to AdvanceZoneVR,   AdvanceZoneVR responsible is
> >>>>>>> distribule IPv6 ip (from the assigned /64 sunet) to VM,  and not
> routing
> >>>>>>> the traffic,   in the VM that get the IPv6 IP will default route
> to ISP
> >>>>>>> router as gw.   It can may be a bridge over via Advancezone-VR.
> >>>>>>>
> >>>>>>
> >>>>>> How would you bridge this? That sounds like NAT?
> >>>>>>
> >>>>>> IPv6 is meant to be routed. Not to be translated or bridged in any
> way.
> >>>>>>
> >>>>>> The way a made the drawing is exactly how IPv6 should work in a VPC
> >>>>>> environment.
> >>>>>>
> >>>>>> Traffic flows through the VR where it can do firewalling of the
> traffic.
> >>>>>>
> >>>>>>> However, If do as the way described in the drawing, then i suppose
> will
> >>>>>> be
> >>>>>>> another kind of virtual router going to introduce , to get hold
> the /48
> >>>>>> in
> >>>>>>> this virtual router right ?
> >>>>>>>
> >>>>>>
> >>>>>> It can be the same VR. But keep in mind that IPv6 != IPv4.
> >>>>>>
> >>>>>> The VR will get Frr as a new daemon which can talk BGP with the
> upper
> >>>>>> network to route traffic.
> >>>>>>
> >>>>>>> After this,  The Advance Zone, NAT's  VR will peer with this new
> IPv6 VR
> >>>>>>> for getting the IPv6 /64 prefix ?
> >>>>>>>
> >>>>>>
> >>>>>> IPv4 will be behind NAT, but IPv6 will not be behind NAT.
> >>>>>>
> >>>>>>> If do in this way, then I guess  you just only need Static route,
> with
> >>>>>>> peering ip both end  as one /48 can have a lot of /64 on it.  And
> >>>>>> hardware
> >>>>>>> budgeting for new IPv6-VR will become very important, as all
> traffic will
> >>>>>>> need to pass over it .
> >>>>>>>
> >>>>>>
> >>>>>> Routing or NAT is the same for the VR. You don't need a very beefy
> VR
> >>>>>> for this.
> >>>>>>
> >>>>>>> It will be like
> >>>>>>>
> >>>>>>> ISP Router  ------ >  (new IPV6-VR ) ---- > AdvanceZone-VR ----> VM
> >>>>>>>
> >>>>>>> Relationship of (new IPv6 VR) and AdvanceZone-VR , may be
> considering on
> >>>>>>> OSPF instead of  BGP , otherwise few thousand of AdvanceZone-VR
> wil have
> >>>>>>> few thousand of BGP session. on new-IPv6-VR
> >>>>>>>
> >>>>>>> Also, I suppose we cannot do ISP router. -->. Advancezone VR
> direct,   ,
> >>>>>>> otherwise ISP router will be full of /64 prefix route either on
> BGP( Many
> >>>>>>> BGP Session) , or  Many Static route .   If few thousand account,
> ti will
> >>>>>>> be few thousand of BGP session with ISP router or few thousand
> static
> >>>>>> route
> >>>>>>> which  is not possible .
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Jul 15, 2021 at 10:47 PM Wido den Hollander <
> wido@widodh.nl>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> But you still need routing. See the attached PNG (and draw.io
> XML).
> >>>>>>>>
> >>>>>>>> You need to route the /48 subnet TO the VR which can then route
> it to
> >>>>>>>> the Virtual Networks behind the VR.
> >>>>>>>>
> >>>>>>>> There is no other way then routing with either BGP or a Static
> route.
> >>>>>>>>
> >>>>>>>> Wido
> >>>>>>>>
> >>>>>>>> Op 15-07-2021 om 12:39 schreef Hean Seng:
> >>>>>>>>> Or explain like this :
> >>>>>>>>>
> >>>>>>>>> 1) Cloudstack generate list of /64 subnet from /48 that Network
> admin
> >>>>>>>>> assigned to Cloudstack
> >>>>>>>>> 2) Cloudsack allocated the subnet (that generated from step1) to
> >>>>>> Virtual
> >>>>>>>>> Router, one Virtual Router have one subniet /64
> >>>>>>>>> 3) Virtual Router allocate single IPv6 (within the range of /64
> >>>>>>>>> allocated to VR)  to VM
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng <heanseng@gmail.com
> >>>>>>>>> <ma...@gmail.com>> wrote:
> >>>>>>>>>
> >>>>>>>>>         Hi Wido,
> >>>>>>>>>
> >>>>>>>>>         I think the /48 is at physical router as gateway , and
> subnet of
> >>>>>> /64
> >>>>>>>>>         at VR of Cloudstack.   Cloudstack only keep which /48
> prefix and
> >>>>>>>>>         vlan information of this /48 to be later split the  /64.
> to VR.
> >>>>>>>>>
> >>>>>>>>>         And the instances is getting singe IPv6 of /64  IP.
>  The VR is
> >>>>>>>>>         getting /64.  The default gateway shall goes to /48 of
> physical
> >>>>>>>>>         router ip .   In this case ,does not need any BGP router
> .
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         Similar concept as IPv4 :
> >>>>>>>>>
> >>>>>>>>>         /48 subnet of IPv6 is equivalent to current /24 subnet
> of IPv4
> >>>>>> that
> >>>>>>>>>         created in Network.
> >>>>>>>>>         and /64  of IPv6 is equivalent to single IP of IPv4
> assign to VM.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander <
> >>>>>> wido@widodh.nl
> >>>>>>>>>         <ma...@widodh.nl>> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>             Op 14-07-2021 om 16:44 schreef Hean Seng:
> >>>>>>>>>              > Hi
> >>>>>>>>>              >
> >>>>>>>>>              > I replied in another thread, i think do not need
> implement
> >>>>>>>>>             BGP or OSPF,
> >>>>>>>>>              > that would be complicated .
> >>>>>>>>>              >
> >>>>>>>>>              > We only need assign  IPv6 's /64 prefix to
> Virtual Router
> >>>>>>>>>             (VR) in NAT
> >>>>>>>>>              > zone, and the VR responsible to deliver single
> IPv6 to VM
> >>>>>> via
> >>>>>>>>>             DHCP6.
> >>>>>>>>>              >
> >>>>>>>>>              > In VR, you need to have Default IPv6 route to
> Physical
> >>>>>>>>>             Router's /48. IP
> >>>>>>>>>              > as IPv6 Gateway.  Thens should be done .
> >>>>>>>>>              >
> >>>>>>>>>              > Example :
> >>>>>>>>>              > Physical Router Interface
> >>>>>>>>>              >   IPv6 IP : 2000:aaaa::1/48
> >>>>>>>>>              >
> >>>>>>>>>              > Cloudstack  virtual router :
> 2000:aaaa:200:201::1/64 with
> >>>>>>>>>             default ipv6
> >>>>>>>>>              > route to router ip 2000:aaaa::1
> >>>>>>>>>              > and Clodustack Virtual router dhcp allocate IP to
> VM , and
> >>>>>>>>>             VM will have
> >>>>>>>>>              > default route to VR. IPv6 2000:aaaa:200:201::1
> >>>>>>>>>              >
> >>>>>>>>>              > So in cloudstack need to allow  user to enter ,
> IPv6
> >>>>>>>>>             gwateway , and
> >>>>>>>>>              > the  /48 Ipv6 prefix , then it will self allocate
> the /64
> >>>>>> ip
> >>>>>>>>>             to the VR ,
> >>>>>>>>>              > and maintain make sure not ovelap allocation
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>
> >>>>>>>>>             But NAT is truly not the solution with IPv6. IPv6 is
> supposed
> >>>>>> to
> >>>>>>>> be
> >>>>>>>>>             routable. In addition you should avoid DHCPv6 as
> much as
> >>>>>>>>>             possible as
> >>>>>>>>>             that's not really the intended use-case for address
> allocation
> >>>>>>>>>             with IPv6.
> >>>>>>>>>
> >>>>>>>>>             In order to route an /48 IPv6 subnet to the VR you
> have a few
> >>>>>>>>>             possibilities:
> >>>>>>>>>
> >>>>>>>>>             - Static route from the upperlying routers which are
> outside
> >>>>>> of
> >>>>>>>>>             CloudStack
> >>>>>>>>>             - BGP
> >>>>>>>>>             - OSPFv3 (broken in most cases!)
> >>>>>>>>>             - DHCPv6 Prefix Delegation
> >>>>>>>>>
> >>>>>>>>>             BGP and/or Static routes are still the best bet here.
> >>>>>>>>>
> >>>>>>>>>             So what you do is that you tell CloudStack that you
> will route
> >>>>>>>>>             2001:db8::/48 to the VR, the VR can then use that to
> split it
> >>>>>> up
> >>>>>>>>>             into
> >>>>>>>>>             multiple /64 subnets going towards the instances:
> >>>>>>>>>
> >>>>>>>>>             - 2001:db8::/64
> >>>>>>>>>             - 2001:db8:1::/64
> >>>>>>>>>             - 2001:db8:2::/64
> >>>>>>>>>             ...
> >>>>>>>>>             - 2001:db8:f::/64
> >>>>>>>>>
> >>>>>>>>>             And go on.
> >>>>>>>>>
> >>>>>>>>>             In case of BGP you indeed have to tell the VR a few
> things:
> >>>>>>>>>
> >>>>>>>>>             - It's own AS number
> >>>>>>>>>             - The peer's address(es)
> >>>>>>>>>
> >>>>>>>>>             With FRR you can simply say:
> >>>>>>>>>
> >>>>>>>>>             neighbor 2001:db8:4fa::179 remote-as external
> >>>>>>>>>
> >>>>>>>>>             The /48 you need to have at the VR anyway in case of
> either a
> >>>>>>>>>             static
> >>>>>>>>>             route or BGP.
> >>>>>>>>>
> >>>>>>>>>             We just need to add a NullRoute on the VR for that
> /48 so that
> >>>>>>>>>             traffic
> >>>>>>>>>             will not be routed to the upper gateway in case of
> the VR
> >>>>>> can't
> >>>>>>>>>             find a
> >>>>>>>>>             route.
> >>>>>>>>>
> >>>>>>>>>             Wido
> >>>>>>>>>
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli
> >>>>>>>>>              > <Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>             <mailto:Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> wrote:
> >>>>>>>>>              >
> >>>>>>>>>              >     Hi Wido,
> >>>>>>>>>              >     That's pretty much in line with our thoughts,
> thanks
> >>>>>> for
> >>>>>>>>>             the input.
> >>>>>>>>>              >     I believe we agree on the following points
> then:
> >>>>>>>>>              >
> >>>>>>>>>              >     - FRR with BGP (no OSPF)
> >>>>>>>>>              >     - Route /48 (or/56) down to the VR
> >>>>>>>>>              >     - /64 per network
> >>>>>>>>>              >     - SLACC for IP addressing
> >>>>>>>>>              >
> >>>>>>>>>              >     I believe the next big question is then "on
> which level
> >>>>>>>>>             of ACS do we
> >>>>>>>>>              >     manage AS numbers?".  I see two options:
> >>>>>>>>>              >     1) Private AS number on a per-zone basis
> >>>>>>>>>              >     2) Root Admin assigned AS number on a
> domain/account
> >>>>>> basis
> >>>>>>>>>              >     3) End-user driven AS number on a per network
> basis
> >>>>>> (for
> >>>>>>>>>             bring your
> >>>>>>>>>              >     own AS and IP scenario)
> >>>>>>>>>              >
> >>>>>>>>>              >     Thoughts?
> >>>>>>>>>              >
> >>>>>>>>>              >     Cheers
> >>>>>>>>>              >     Alex
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >     -----Original Message-----
> >>>>>>>>>              >     From: Wido den Hollander <wido@widodh.nl
> >>>>>>>>>             <ma...@widodh.nl> <mailto:wido@widodh.nl
> >>>>>>>>>             <ma...@widodh.nl>>>
> >>>>>>>>>              >     Sent: 13 July 2021 15:08
> >>>>>>>>>              >     To: dev@cloudstack.apache.org
> >>>>>>>>>             <ma...@cloudstack.apache.org>
> >>>>>>>>>             <mailto:dev@cloudstack.apache.org
> >>>>>>>>>             <ma...@cloudstack.apache.org>>;
> >>>>>>>>>              >     Alex Mattioli <Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>
> >>>>>>>>>              >     Cc: Wei Zhou <Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>; Rohit Yadav
> >>>>>>>>>              >     <rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>             <mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>;
> >>>>>>>>>              >     Gabriel Beims Bräscher <gabriel@pcextreme.nl
> >>>>>>>>>             <ma...@pcextreme.nl>
> >>>>>>>>>              >     <mailto:gabriel@pcextreme.nl <mailto:
> >>>>>> gabriel@pcextreme.nl
> >>>>>>>>>>>
> >>>>>>>>>              >     Subject: Re: IPV6 in Isolated/VPC networks
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >     On 7/7/21 1:16 PM, Alex Mattioli wrote:
> >>>>>>>>>              >      > Hi all,
> >>>>>>>>>              >      > @Wei Zhou<mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> @Rohit
> >>>>>>>>>              >     Yadav<mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> and myself are
> >>>>>>>>>             investigating how
> >>>>>>>>>              >     to enable IPV6 support on Isolated and VPC
> networks and
> >>>>>>>>>             would like
> >>>>>>>>>              >     your input on it.
> >>>>>>>>>              >      > At the moment we are looking at
> implementing FRR
> >>>>>> with
> >>>>>>>>>             BGP (and
> >>>>>>>>>              >     possibly OSPF) on the ACS VR.
> >>>>>>>>>              >      >
> >>>>>>>>>              >      > We are looking for requirements,
> recommendations,
> >>>>>>>>>             ideas, rants,
> >>>>>>>>>              >     etc...etc...
> >>>>>>>>>              >      >
> >>>>>>>>>              >
> >>>>>>>>>              >     Ok! Here we go.
> >>>>>>>>>              >
> >>>>>>>>>              >     I think that you mean that the VR will
> actually route
> >>>>>> the
> >>>>>>>>>             IPv6
> >>>>>>>>>              >     traffic and for that you need to have a way
> of getting
> >>>>>> a
> >>>>>>>>>             subnet
> >>>>>>>>>              >     routed to the VR.
> >>>>>>>>>              >
> >>>>>>>>>              >     BGP is probably you best bet here. Although
> OSPFv3
> >>>>>>>>>             technically
> >>>>>>>>>              >     supports this it is very badly implemented in
> Frr for
> >>>>>>>>>             example.
> >>>>>>>>>              >
> >>>>>>>>>              >     Now FRR is a very good router and one of the
> fancy
> >>>>>>>>>             features it
> >>>>>>>>>              >     supports is BGP Unnumered. This allows for
> auto
> >>>>>>>>>             configuration of BGP
> >>>>>>>>>              >     over a L2 network when both sides are sending
> Router
> >>>>>>>>>             Advertisements.
> >>>>>>>>>              >     This is very easy for flexible BGP
> configurations where
> >>>>>>>>>             both sides
> >>>>>>>>>              >     have dynamic IPs.
> >>>>>>>>>              >
> >>>>>>>>>              >     What you want to do is that you get a /56,
> /48 or
> >>>>>>>>>             something which is
> >>>>>>>>>              >      >/64 bits routed to the VR.
> >>>>>>>>>              >
> >>>>>>>>>              >     Now you can sub-segment this into separate
> /64 subnets.
> >>>>>>>>>             You don't
> >>>>>>>>>              >     want to go smaller then a /64 is that
> prevents you from
> >>>>>>>>>             using SLAAC
> >>>>>>>>>              >     for IPv6 address configuration. This is how
> it works
> >>>>>> for
> >>>>>>>>>             Shared
> >>>>>>>>>              >     Networks now in Basic and Advanced Zones.
> >>>>>>>>>              >
> >>>>>>>>>              >     FRR can now also send out the Router
> Advertisements on
> >>>>>>>>>             the downlinks
> >>>>>>>>>              >     sending out:
> >>>>>>>>>              >
> >>>>>>>>>              >     - DNS servers
> >>>>>>>>>              >     - DNS domain
> >>>>>>>>>              >     - Prefix (/64) to be used
> >>>>>>>>>              >
> >>>>>>>>>              >     There is no need for DHCPv6. You can
> calculate the IPv6
> >>>>>>>>>             address the
> >>>>>>>>>              >     VM will obtain by using the MAC and the
> prefix.
> >>>>>>>>>              >
> >>>>>>>>>              >     So in short:
> >>>>>>>>>              >
> >>>>>>>>>              >     - Using BGP you routed a /48 to the VR
> >>>>>>>>>              >     - Now you split this into /64 subnets towards
> the
> >>>>>>>>>             isolated networks
> >>>>>>>>>              >
> >>>>>>>>>              >     Wido
> >>>>>>>>>              >
> >>>>>>>>>              >      > Alex Mattioli
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              > --
> >>>>>>>>>              > Regards,
> >>>>>>>>>              > Hean Seng
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         --
> >>>>>>>>>         Regards,
> >>>>>>>>>         Hean Seng
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Regards,
> >>>>>>>>> Hean Seng
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>
>


-- 
Regards,
Hean Seng

Re: IPV6 in Isolated/VPC networks

Posted by Hean Seng <he...@gmail.com>.
I am very welcome on this ,  the near future may have almost all out of
IPv4, and will going to dependent on IPv6:

    *   ipv4 only (default, for backward compatibility all
networks/offerings post-upgrade migrate to this option)
     *   ipv4-and-ipv6
     *   ipv6-only (this can be phase 1.b)

Also allow change of IPv6  IP , and  Secondary IPv6 as well, especially for
current Shared Network IPv6.




On Wed, Aug 11, 2021 at 8:27 PM Rohit Yadav <ro...@shapeblue.com>
wrote:

> Hi all,
>
> Thanks for your feedback and ideas, I've gone ahead with discussing them
> with Alex and came up with a PoC/design which can be implemented in the
> following phases:
>
>   *   Phase1: implement ipv6 support in isolated networks and VPC with
> static routing
>   *   Phase2: discuss and implement support for dynamic routing (TBD)
>
> For Phase1 here's the high-level proposal:
>
>   *   IPv6 address management:
>      *   At the zone level root-admin specifies a /64 public range that
> will be used for VRs, then they can add a /48, or /56 IPv6 range for guest
> networks (to be used by isolated networks and VPC tiers)
>      *   On creation of any IPv6 enabled isolated network or VPC tier,
> from the /48 or /56 block a /64 network is allocated/used
>      *   We assume SLAAC and autoconfiguration, no DHCPv6 in the zone
> (discuss: is privacy a concern, can privacy extensions rfc4941 of slaac be
> explored?)
>   *   Network offerings: root-admin can create new network offerings (with
> VPC too) that specifies a network stack option:
>      *   ipv4 only (default, for backward compatibility all
> networks/offerings post-upgrade migrate to this option)
>      *   ipv4-and-ipv6
>      *   ipv6-only (this can be phase 1.b)
>      *   A new routing option: static (phase1), dynamic (phase2, with
> multiple sub-options such as ospf/bgp etc...)
>   *   VR changes:
>      *   VR gets its guest and public nics set to inet6 auto
>      *   For each /64 allocated to guest network and VPC tiers, radvd is
> configured to do RA
>      *   Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6
> firewall rules (ACLs, ingress, egress) are implemented; ACLs between VPC
> tiers are managed/implemented by ipv6 firewall on VR
>      *   It is assumed that static routes are created on the core/main
> router by the admin or automated using some scripts/tools; for this
> CloudStack will announce events with details of /64 networks and VR's
> public IPv6 address that can be consumed by a rabbitmq/message bus client
> (for example), or a custom cron job or script as part of orchestration.
> (this wouldn't be necessary for dynamic routing bgp with phase2)
>   *   Guest Networking: With SLAAC, it's easy for CloudStack to calculate
> allocate and use a /64 and determine the IPv6 address of VR nics and guest
> VM nics
>      *   A user create an isolated network/VPC with an offering that is
> ipv6 enabled
>      *   A user can manage firewall for the IPv6 address/guest nics;
> there'll be no port forward and LB feature though for IPv6
>      *   A users can run workloads in the guest VMs that listen on
> publically routable ipv6 addresses
>      *   Usage/billing etc continue to work, no change needed
>
> Network layout:
>
> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] ->
> [guest VMs/nics]
> *core/ISP router needs static routes to be added (manually or automated),
> assumes a /48 or /56 configured for the zone
>
> Thoughts, feedback?
>
> Proof-of-concept commentary: here's what I did to test the idea:
>
>   *   Created an isolated network and deployed a VM in my home lab
> The VR running on KVM has following nics
> eth0 - guest network
> eth1 - link local
> eth2 - public network
>
>   *   I setup a custom openwrt router on a RPi4 to serve as a toy-core
> router where I create a wan6 IPv6 tunnel using tunnel broker and I got a
> /48 allocated. My configuration looks like:
> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> /64 - 2001:470:36:3e2::/64 (default allocated by)
>
> I create a LAN ipv6 (public network for CloudStack VR): at subnet/prefix 0:
> LAN IPv6 address: 2001:470:ed36:0::1/64
> Address mode: SLAAC+stateless DHCP (no dhcpv6)
>   *
>   *
> In the isolated VR, I enabled ipv6 as:
> net.ipv6.conf.all.disable_ipv6 = 0
> net.ipv6.conf.all.forwarding = 1
> net.ipv6.conf.all.accept_ra = 1
> net.ipv6.conf.all.accept_redirects = 1
> net.ipv6.conf.all.autoconf = 1
>
> Set up a IPv6 nameserver/dns in /etc/resolve.conf
> And configured the nics:
> echo iface eth0 inet6 auto >> /etc/network/interfaces
> echo iface eth2 inet6 auto >> /etc/network/interfaces
> /etc/init.d/networking restart
> Next, restart ACS isolated network without cleanup to have it reconfigure
> IPv4 nics, firewall, NAT etc
>
>   *
> Next, I created a /64 network for the isolated guest network on eth0 of VR
> using radvd:
>
> # cat /etc/radvd.conf
> interface eth0
> {
>     AdvSendAdvert on;
>     MinRtrAdvInterval 5;
>     MaxRtrAdvInterval 15;
>     prefix 2001:470:ed36:1::/64
>     {
>         AdvOnLink on;
>         AdvAutonomous on;
>     };
> };
> systemctl restart radvd
> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this
> ...:1::/64 network
>   *   Finally I added a static route in toy core-router for the new /64
> IPv6 range in the isolated network
> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev
> <local lan nic>
>   *
> ... and I enabled firewall rules to allow any traffic to pass for the new
> /64 network
>
> And voila all done! I create a domain AAAA record that points to my guest
> VM IPv6 address a test webserver on
> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>
> (Note: I'll get rid of the tunnel and request a new /48 block after a few
> days, sharing this solely for testing purposes)
>
>
> Regards.
>
> ________________________________
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: Tuesday, July 20, 2021 12:46
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
>
>
> Op 19-07-2021 om 20:38 schreef Kristaps Cudars:
> > Hi Wido,
> >
> > I assume that flouting ip will not work grate with ingress/egress acl on
> VR.
> >
> >  From regular ACS user perspective:
> > I have Instance with dualstack its running web app on 443.
> > I want to swap instances for whatever reason.
> > In case of IPv4 change d-nat rule.
> > In case of IPv6 if flouting IP was not created upfront he will need to
> change dns entry that usually has 24h ttl. Inconvenience degradation in
> experience.
> >
>
> Yes, but, keep in mind that the IP you are using can also be terminated
> on the VR where HAProxy proxies request to the backend VM (could even be
> v4!)
>
> I'm not against DHCPv6, but I have seen many issues with implementing
> it. Therefor I always stick to SLAAC.
>
> >  From ACS admin perspective:
> > I don’t want to have these tickets in helpdesk.
> > You needed to create another flouting IP that it would be seamless- will
> not work as answer.
> >
>
> I understand that as well.
>
> Wido
>
> >
> > On 2021/07/19 09:05:54, Wido den Hollander <wi...@widodh.nl> wrote:
> >>
> >>
> >> Op 16-07-2021 om 21:46 schreef Kristaps Cudars:
> >>> Hi Wido,
> >>>
> >>> Your proposal is to sacrifice ability to reassign IPv6 to instance,
> have internal domain prefix, and list/db in ACS what IPv6 has been assigned
> to what instance and go with RA and SLAAC. For route signaling to switch
> use BGP/OSPFv3 or manual pre-creation.
> >>>
> >>
> >> You can still list the IPs which have been assigned. You'll know exactly
> >> what IPv6 address a VM has because of the prefix + MAC. Privacy
> >> Extensions need to be disabled in the VM.
> >>
> >> This already works in CloudStack in Shared Networks in this way.
> >>
> >> Using secondary IPs you can always have 'floating' IPv6 addressess.
> >>
> >> Wido
> >>
> >>> Option with RA and managed flag that DHCPv6 is in use to support
> preset information and ability to create route information from ACS is not
> an option as DHCPv6 its failing?
> >>>
> >>>
> >>> On 2021/07/16 15:17:42, Wido den Hollander <wi...@widodh.nl> wrote:
> >>>>
> >>>>
> >>>> Op 16-07-2021 om 16:42 schreef Hean Seng:
> >>>>> Hi Wido,
> >>>>>
> >>>>> In current setup,  each Cloudstack have own VR, so in this new  IPv6
> subnet
> >>>>> allocation , each VR (which have Frr) will need to have peering with
> ISP
> >>>>> router (and either BGP or Static Route) , and there is 1000
> Acocunts,  it
> >>>>> will 1000 BGP session with ISP router ,  Am I right for this ? or I
> >>>>> understand wrong .
> >>>>>
> >>>>
> >>>> Yes, that is correct. A /56 would also be sufficient or a /60 which is
> >>>> enough to allocate a few /64 subnets.
> >>>>
> >>>> 1000 BGP connections isn't really a problem for a proper router at the
> >>>> ISP. OSPF(v3) would be better, but as I said that's poorly supported.
> >>>>
> >>>> The ISP could also install 1000 static routes, but that means that the
> >>>> ISP's router needs to have those configured.
> >>>>
> >>>> http://docs.frrouting.org/en/latest/ospf6d.html
> >>>> (While looking up this URL I see that Frr recently put in a lot of
> work
> >>>> in OSPFv3, seems better now)
> >>>>
> >>>>> I understand IPv6 is different then IPv4, and in IPv6 it suppose each
> >>>>> devices have own IP. It just how to realize in easy way.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
>
>
> > On Fri, Jul 16, 2021 at 8:17 PM Wido den Hollander <wi...@widodh.nl>
> wrote:
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> Op 16-07-2021 om 05:54 schreef Hean Seng:
> >>>>>>> Hi Wido,
> >>>>>>>
> >>>>>>> My initial thought is not like this,  it is the /48 at ISP router,
> and
> >>>>>> /64
> >>>>>>> subnet assign to AdvanceZoneVR,   AdvanceZoneVR responsible is
> >>>>>>> distribule IPv6 ip (from the assigned /64 sunet) to VM,  and not
> routing
> >>>>>>> the traffic,   in the VM that get the IPv6 IP will default route
> to ISP
> >>>>>>> router as gw.   It can may be a bridge over via Advancezone-VR.
> >>>>>>>
> >>>>>>
> >>>>>> How would you bridge this? That sounds like NAT?
> >>>>>>
> >>>>>> IPv6 is meant to be routed. Not to be translated or bridged in any
> way.
> >>>>>>
> >>>>>> The way a made the drawing is exactly how IPv6 should work in a VPC
> >>>>>> environment.
> >>>>>>
> >>>>>> Traffic flows through the VR where it can do firewalling of the
> traffic.
> >>>>>>
> >>>>>>> However, If do as the way described in the drawing, then i suppose
> will
> >>>>>> be
> >>>>>>> another kind of virtual router going to introduce , to get hold
> the /48
> >>>>>> in
> >>>>>>> this virtual router right ?
> >>>>>>>
> >>>>>>
> >>>>>> It can be the same VR. But keep in mind that IPv6 != IPv4.
> >>>>>>
> >>>>>> The VR will get Frr as a new daemon which can talk BGP with the
> upper
> >>>>>> network to route traffic.
> >>>>>>
> >>>>>>> After this,  The Advance Zone, NAT's  VR will peer with this new
> IPv6 VR
> >>>>>>> for getting the IPv6 /64 prefix ?
> >>>>>>>
> >>>>>>
> >>>>>> IPv4 will be behind NAT, but IPv6 will not be behind NAT.
> >>>>>>
> >>>>>>> If do in this way, then I guess  you just only need Static route,
> with
> >>>>>>> peering ip both end  as one /48 can have a lot of /64 on it.  And
> >>>>>> hardware
> >>>>>>> budgeting for new IPv6-VR will become very important, as all
> traffic will
> >>>>>>> need to pass over it .
> >>>>>>>
> >>>>>>
> >>>>>> Routing or NAT is the same for the VR. You don't need a very beefy
> VR
> >>>>>> for this.
> >>>>>>
> >>>>>>> It will be like
> >>>>>>>
> >>>>>>> ISP Router  ------ >  (new IPV6-VR ) ---- > AdvanceZone-VR ----> VM
> >>>>>>>
> >>>>>>> Relationship of (new IPv6 VR) and AdvanceZone-VR , may be
> considering on
> >>>>>>> OSPF instead of  BGP , otherwise few thousand of AdvanceZone-VR
> wil have
> >>>>>>> few thousand of BGP session. on new-IPv6-VR
> >>>>>>>
> >>>>>>> Also, I suppose we cannot do ISP router. -->. Advancezone VR
> direct,   ,
> >>>>>>> otherwise ISP router will be full of /64 prefix route either on
> BGP( Many
> >>>>>>> BGP Session) , or  Many Static route .   If few thousand account,
> ti will
> >>>>>>> be few thousand of BGP session with ISP router or few thousand
> static
> >>>>>> route
> >>>>>>> which  is not possible .
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Jul 15, 2021 at 10:47 PM Wido den Hollander <
> wido@widodh.nl>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> But you still need routing. See the attached PNG (and draw.io
> XML).
> >>>>>>>>
> >>>>>>>> You need to route the /48 subnet TO the VR which can then route
> it to
> >>>>>>>> the Virtual Networks behind the VR.
> >>>>>>>>
> >>>>>>>> There is no other way then routing with either BGP or a Static
> route.
> >>>>>>>>
> >>>>>>>> Wido
> >>>>>>>>
> >>>>>>>> Op 15-07-2021 om 12:39 schreef Hean Seng:
> >>>>>>>>> Or explain like this :
> >>>>>>>>>
> >>>>>>>>> 1) Cloudstack generate list of /64 subnet from /48 that Network
> admin
> >>>>>>>>> assigned to Cloudstack
> >>>>>>>>> 2) Cloudsack allocated the subnet (that generated from step1) to
> >>>>>> Virtual
> >>>>>>>>> Router, one Virtual Router have one subniet /64
> >>>>>>>>> 3) Virtual Router allocate single IPv6 (within the range of /64
> >>>>>>>>> allocated to VR)  to VM
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng <heanseng@gmail.com
> >>>>>>>>> <ma...@gmail.com>> wrote:
> >>>>>>>>>
> >>>>>>>>>         Hi Wido,
> >>>>>>>>>
> >>>>>>>>>         I think the /48 is at physical router as gateway , and
> subnet of
> >>>>>> /64
> >>>>>>>>>         at VR of Cloudstack.   Cloudstack only keep which /48
> prefix and
> >>>>>>>>>         vlan information of this /48 to be later split the  /64.
> to VR.
> >>>>>>>>>
> >>>>>>>>>         And the instances is getting singe IPv6 of /64  IP.
>  The VR is
> >>>>>>>>>         getting /64.  The default gateway shall goes to /48 of
> physical
> >>>>>>>>>         router ip .   In this case ,does not need any BGP router
> .
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         Similar concept as IPv4 :
> >>>>>>>>>
> >>>>>>>>>         /48 subnet of IPv6 is equivalent to current /24 subnet
> of IPv4
> >>>>>> that
> >>>>>>>>>         created in Network.
> >>>>>>>>>         and /64  of IPv6 is equivalent to single IP of IPv4
> assign to VM.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander <
> >>>>>> wido@widodh.nl
> >>>>>>>>>         <ma...@widodh.nl>> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>             Op 14-07-2021 om 16:44 schreef Hean Seng:
> >>>>>>>>>              > Hi
> >>>>>>>>>              >
> >>>>>>>>>              > I replied in another thread, i think do not need
> implement
> >>>>>>>>>             BGP or OSPF,
> >>>>>>>>>              > that would be complicated .
> >>>>>>>>>              >
> >>>>>>>>>              > We only need assign  IPv6 's /64 prefix to
> Virtual Router
> >>>>>>>>>             (VR) in NAT
> >>>>>>>>>              > zone, and the VR responsible to deliver single
> IPv6 to VM
> >>>>>> via
> >>>>>>>>>             DHCP6.
> >>>>>>>>>              >
> >>>>>>>>>              > In VR, you need to have Default IPv6 route to
> Physical
> >>>>>>>>>             Router's /48. IP
> >>>>>>>>>              > as IPv6 Gateway.  Thens should be done .
> >>>>>>>>>              >
> >>>>>>>>>              > Example :
> >>>>>>>>>              > Physical Router Interface
> >>>>>>>>>              >   IPv6 IP : 2000:aaaa::1/48
> >>>>>>>>>              >
> >>>>>>>>>              > Cloudstack  virtual router :
> 2000:aaaa:200:201::1/64 with
> >>>>>>>>>             default ipv6
> >>>>>>>>>              > route to router ip 2000:aaaa::1
> >>>>>>>>>              > and Clodustack Virtual router dhcp allocate IP to
> VM , and
> >>>>>>>>>             VM will have
> >>>>>>>>>              > default route to VR. IPv6 2000:aaaa:200:201::1
> >>>>>>>>>              >
> >>>>>>>>>              > So in cloudstack need to allow  user to enter ,
> IPv6
> >>>>>>>>>             gwateway , and
> >>>>>>>>>              > the  /48 Ipv6 prefix , then it will self allocate
> the /64
> >>>>>> ip
> >>>>>>>>>             to the VR ,
> >>>>>>>>>              > and maintain make sure not ovelap allocation
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>
> >>>>>>>>>             But NAT is truly not the solution with IPv6. IPv6 is
> supposed
> >>>>>> to
> >>>>>>>> be
> >>>>>>>>>             routable. In addition you should avoid DHCPv6 as
> much as
> >>>>>>>>>             possible as
> >>>>>>>>>             that's not really the intended use-case for address
> allocation
> >>>>>>>>>             with IPv6.
> >>>>>>>>>
> >>>>>>>>>             In order to route an /48 IPv6 subnet to the VR you
> have a few
> >>>>>>>>>             possibilities:
> >>>>>>>>>
> >>>>>>>>>             - Static route from the upperlying routers which are
> outside
> >>>>>> of
> >>>>>>>>>             CloudStack
> >>>>>>>>>             - BGP
> >>>>>>>>>             - OSPFv3 (broken in most cases!)
> >>>>>>>>>             - DHCPv6 Prefix Delegation
> >>>>>>>>>
> >>>>>>>>>             BGP and/or Static routes are still the best bet here.
> >>>>>>>>>
> >>>>>>>>>             So what you do is that you tell CloudStack that you
> will route
> >>>>>>>>>             2001:db8::/48 to the VR, the VR can then use that to
> split it
> >>>>>> up
> >>>>>>>>>             into
> >>>>>>>>>             multiple /64 subnets going towards the instances:
> >>>>>>>>>
> >>>>>>>>>             - 2001:db8::/64
> >>>>>>>>>             - 2001:db8:1::/64
> >>>>>>>>>             - 2001:db8:2::/64
> >>>>>>>>>             ...
> >>>>>>>>>             - 2001:db8:f::/64
> >>>>>>>>>
> >>>>>>>>>             And go on.
> >>>>>>>>>
> >>>>>>>>>             In case of BGP you indeed have to tell the VR a few
> things:
> >>>>>>>>>
> >>>>>>>>>             - It's own AS number
> >>>>>>>>>             - The peer's address(es)
> >>>>>>>>>
> >>>>>>>>>             With FRR you can simply say:
> >>>>>>>>>
> >>>>>>>>>             neighbor 2001:db8:4fa::179 remote-as external
> >>>>>>>>>
> >>>>>>>>>             The /48 you need to have at the VR anyway in case of
> either a
> >>>>>>>>>             static
> >>>>>>>>>             route or BGP.
> >>>>>>>>>
> >>>>>>>>>             We just need to add a NullRoute on the VR for that
> /48 so that
> >>>>>>>>>             traffic
> >>>>>>>>>             will not be routed to the upper gateway in case of
> the VR
> >>>>>> can't
> >>>>>>>>>             find a
> >>>>>>>>>             route.
> >>>>>>>>>
> >>>>>>>>>             Wido
> >>>>>>>>>
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli
> >>>>>>>>>              > <Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>             <mailto:Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> wrote:
> >>>>>>>>>              >
> >>>>>>>>>              >     Hi Wido,
> >>>>>>>>>              >     That's pretty much in line with our thoughts,
> thanks
> >>>>>> for
> >>>>>>>>>             the input.
> >>>>>>>>>              >     I believe we agree on the following points
> then:
> >>>>>>>>>              >
> >>>>>>>>>              >     - FRR with BGP (no OSPF)
> >>>>>>>>>              >     - Route /48 (or/56) down to the VR
> >>>>>>>>>              >     - /64 per network
> >>>>>>>>>              >     - SLACC for IP addressing
> >>>>>>>>>              >
> >>>>>>>>>              >     I believe the next big question is then "on
> which level
> >>>>>>>>>             of ACS do we
> >>>>>>>>>              >     manage AS numbers?".  I see two options:
> >>>>>>>>>              >     1) Private AS number on a per-zone basis
> >>>>>>>>>              >     2) Root Admin assigned AS number on a
> domain/account
> >>>>>> basis
> >>>>>>>>>              >     3) End-user driven AS number on a per network
> basis
> >>>>>> (for
> >>>>>>>>>             bring your
> >>>>>>>>>              >     own AS and IP scenario)
> >>>>>>>>>              >
> >>>>>>>>>              >     Thoughts?
> >>>>>>>>>              >
> >>>>>>>>>              >     Cheers
> >>>>>>>>>              >     Alex
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >     -----Original Message-----
> >>>>>>>>>              >     From: Wido den Hollander <wido@widodh.nl
> >>>>>>>>>             <ma...@widodh.nl> <mailto:wido@widodh.nl
> >>>>>>>>>             <ma...@widodh.nl>>>
> >>>>>>>>>              >     Sent: 13 July 2021 15:08
> >>>>>>>>>              >     To: dev@cloudstack.apache.org
> >>>>>>>>>             <ma...@cloudstack.apache.org>
> >>>>>>>>>             <mailto:dev@cloudstack.apache.org
> >>>>>>>>>             <ma...@cloudstack.apache.org>>;
> >>>>>>>>>              >     Alex Mattioli <Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Alex.Mattioli@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>
> >>>>>>>>>              >     Cc: Wei Zhou <Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>; Rohit Yadav
> >>>>>>>>>              >     <rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>             <mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>>;
> >>>>>>>>>              >     Gabriel Beims Bräscher <gabriel@pcextreme.nl
> >>>>>>>>>             <ma...@pcextreme.nl>
> >>>>>>>>>              >     <mailto:gabriel@pcextreme.nl <mailto:
> >>>>>> gabriel@pcextreme.nl
> >>>>>>>>>>>
> >>>>>>>>>              >     Subject: Re: IPV6 in Isolated/VPC networks
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >     On 7/7/21 1:16 PM, Alex Mattioli wrote:
> >>>>>>>>>              >      > Hi all,
> >>>>>>>>>              >      > @Wei Zhou<mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:Wei.Zhou@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> @Rohit
> >>>>>>>>>              >     Yadav<mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>
> >>>>>>>>>              >     <mailto:rohit.yadav@shapeblue.com
> >>>>>>>>>             <ma...@shapeblue.com>>> and myself are
> >>>>>>>>>             investigating how
> >>>>>>>>>              >     to enable IPV6 support on Isolated and VPC
> networks and
> >>>>>>>>>             would like
> >>>>>>>>>              >     your input on it.
> >>>>>>>>>              >      > At the moment we are looking at
> implementing FRR
> >>>>>> with
> >>>>>>>>>             BGP (and
> >>>>>>>>>              >     possibly OSPF) on the ACS VR.
> >>>>>>>>>              >      >
> >>>>>>>>>              >      > We are looking for requirements,
> recommendations,
> >>>>>>>>>             ideas, rants,
> >>>>>>>>>              >     etc...etc...
> >>>>>>>>>              >      >
> >>>>>>>>>              >
> >>>>>>>>>              >     Ok! Here we go.
> >>>>>>>>>              >
> >>>>>>>>>              >     I think that you mean that the VR will
> actually route
> >>>>>> the
> >>>>>>>>>             IPv6
> >>>>>>>>>              >     traffic and for that you need to have a way
> of getting
> >>>>>> a
> >>>>>>>>>             subnet
> >>>>>>>>>              >     routed to the VR.
> >>>>>>>>>              >
> >>>>>>>>>              >     BGP is probably you best bet here. Although
> OSPFv3
> >>>>>>>>>             technically
> >>>>>>>>>              >     supports this it is very badly implemented in
> Frr for
> >>>>>>>>>             example.
> >>>>>>>>>              >
> >>>>>>>>>              >     Now FRR is a very good router and one of the
> fancy
> >>>>>>>>>             features it
> >>>>>>>>>              >     supports is BGP Unnumered. This allows for
> auto
> >>>>>>>>>             configuration of BGP
> >>>>>>>>>              >     over a L2 network when both sides are sending
> Router
> >>>>>>>>>             Advertisements.
> >>>>>>>>>              >     This is very easy for flexible BGP
> configurations where
> >>>>>>>>>             both sides
> >>>>>>>>>              >     have dynamic IPs.
> >>>>>>>>>              >
> >>>>>>>>>              >     What you want to do is that you get a /56,
> /48 or
> >>>>>>>>>             something which is
> >>>>>>>>>              >      >/64 bits routed to the VR.
> >>>>>>>>>              >
> >>>>>>>>>              >     Now you can sub-segment this into separate
> /64 subnets.
> >>>>>>>>>             You don't
> >>>>>>>>>              >     want to go smaller then a /64 is that
> prevents you from
> >>>>>>>>>             using SLAAC
> >>>>>>>>>              >     for IPv6 address configuration. This is how
> it works
> >>>>>> for
> >>>>>>>>>             Shared
> >>>>>>>>>              >     Networks now in Basic and Advanced Zones.
> >>>>>>>>>              >
> >>>>>>>>>              >     FRR can now also send out the Router
> Advertisements on
> >>>>>>>>>             the downlinks
> >>>>>>>>>              >     sending out:
> >>>>>>>>>              >
> >>>>>>>>>              >     - DNS servers
> >>>>>>>>>              >     - DNS domain
> >>>>>>>>>              >     - Prefix (/64) to be used
> >>>>>>>>>              >
> >>>>>>>>>              >     There is no need for DHCPv6. You can
> calculate the IPv6
> >>>>>>>>>             address the
> >>>>>>>>>              >     VM will obtain by using the MAC and the
> prefix.
> >>>>>>>>>              >
> >>>>>>>>>              >     So in short:
> >>>>>>>>>              >
> >>>>>>>>>              >     - Using BGP you routed a /48 to the VR
> >>>>>>>>>              >     - Now you split this into /64 subnets towards
> the
> >>>>>>>>>             isolated networks
> >>>>>>>>>              >
> >>>>>>>>>              >     Wido
> >>>>>>>>>              >
> >>>>>>>>>              >      > Alex Mattioli
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >      >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              >
> >>>>>>>>>              > --
> >>>>>>>>>              > Regards,
> >>>>>>>>>              > Hean Seng
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>         --
> >>>>>>>>>         Regards,
> >>>>>>>>>         Hean Seng
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Regards,
> >>>>>>>>> Hean Seng
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>
>


-- 
Regards,
Hean Seng

RE: IPV6 in Isolated/VPC networks

Posted by Alex Mattioli <Al...@shapeblue.com>.
Hi Wido,
Thanks for the idea, this sounds extremely interesting. We'll investigate.

Cheers
Alex

 


-----Original Message-----
From: Wido den Hollander <wi...@widodh.nl> 
Sent: 17 August 2021 12:46
To: dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks



Op 16-08-2021 om 11:29 schreef Rohit Yadav:
> Thanks Hean, Kristaps, Wido for your feedback.
> 
> I think we've some quorum and consensus on how we should proceed with IPv6 support with static routing (phase1). Based on my proof-of-concept and discussions, I believe we may target this feature as early as 4.17 and I welcome offer by Kristaps and others who may want to be involved in testing the feature as and when we'll develop it.
> 
> As the next step I'll write a short design doc on cwiki including some of the new ideas/suggestions and share it on this thread for another iteration.
> 

I was thinking about this and I have nother idea for phase 2.

As mentioned earlier it can be quite difficult when you have a lot of VRs running to configure OSPFv3 or BGP in the upstream routers to have this all working.

Instead we could think about using ExaBGP. With ExaBGP you would have some additional tooling which picks up the subnets assigned to VRs from the messagebus.

ExaBGP: https://github.com/Thomas-Mangin/exabgp

This tooling then dynamically injects this route into ExaBGP where the destination of thus /48 (example) points to the VR.

Inside the VR there is no need to configure OSPF or BGP. The admin doesn't need to configure static routers either.

Their routers only peer via BGP with ExaBGP which injects the routes.

See this blogpost to get an idea: 
https://vincent.bernat.ch/en/blog/2013-exabgp-highavailability

"Redundancy with ExaBGP"
"ExaBGP is a convenient tool to plug scripts into BGP. They can then receive and advertise routes. ExaBGP does the hard work of speaking BGP with your routers. The scripts just have to read routes from standard input or advertise them on standard output."

So we would just need to feed ExaBGP all the routes which then advertises them again towards the upstream routers.

ExaBGP could be running anywhere as long as it receives the messages from CloudStack and has a BGP connection with the routers.

Might be worth looking into in the future for phase 2.

And again: Also in shared networks it would be very nice to be able to route a subnet towards a single Instance. Native IPv6 with Docker, VPN services with IPv6 inside a VM, etc, etc.

Wido

> 
> Regards.
> 
> ________________________________
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: Friday, August 13, 2021 15:48
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
> 
> Hi,
> 
> See my inline responses:
> 
> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>> Hi all,
>>
>> Thanks for your feedback and ideas, I've gone ahead with discussing them with Alex and came up with a PoC/design which can be implemented in the following phases:
>>
>>     *   Phase1: implement ipv6 support in isolated networks and VPC with static routing
>>     *   Phase2: discuss and implement support for dynamic routing (TBD)
>>
>> For Phase1 here's the high-level proposal:
>>
>>     *   IPv6 address management:
>>        *   At the zone level root-admin specifies a /64 public range that will be used for VRs, then they can add a /48, or /56 IPv6 range for guest networks (to be used by isolated networks and VPC tiers)
>>        *   On creation of any IPv6 enabled isolated network or VPC tier, from the /48 or /56 block a /64 network is allocated/used
>>        *   We assume SLAAC and autoconfiguration, no DHCPv6 in the zone (discuss: is privacy a concern, can privacy extensions rfc4941 of slaac be explored?)
> 
> Privacy Extensions are only a concern for client devices which roam 
> between different IPv6 networks.
> 
> If you IPv6 address of a client keeps the same suffix (MAC based) and 
> switches network then only the prefix (/64) will change.
> 
> This way a network like Google, Facebook, etc could track your device 
> moving from network to network if they only look at the last 64-bits 
> of the IPv6 address.
> 
> For servers this is not a problem as you already know in which network 
> they are.
> 
>>     *   Network offerings: root-admin can create new network offerings (with VPC too) that specifies a network stack option:
>>        *   ipv4 only (default, for backward compatibility all networks/offerings post-upgrade migrate to this option)
>>        *   ipv4-and-ipv6
>>        *   ipv6-only (this can be phase 1.b)
>>        *   A new routing option: static (phase1), dynamic (phase2, with multiple sub-options such as ospf/bgp etc...)
> 
> This means that the network admin will need to statically route the 
> IPv6 subnet to the VR's outside IPv6 address, for example, on a JunOS router:
> 
> set routing-options rib inet6.0 static route 2001:db8:500::/48 
> next-hop
> 2001:db8:100::50
> 
> I'm assuming that 2001:db8:100::50 is the address of the VR on the 
> outside (/64) network. In reality this will probably be a longer 
> address, but this is for just the example.
> 
>>     *   VR changes:
>>        *   VR gets its guest and public nics set to inet6 auto
>>        *   For each /64 allocated to guest network and VPC tiers, radvd is configured to do RA
> 
> radvd is fine, but looking at phase 2 with dynamic routing you might 
> already want to look into FRRouting. FRR can also advertise RAs while 
> not doing any routing.
> 
> interface ens4
>     no ipv6 nd suppress-ra
>     ipv6 nd prefix 2001:db8:500::/64
>     ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> 
> See: http://docs.frrouting.org/en/latest/ipv6.html
> 
>>        *   Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6 firewall rules (ACLs, ingress, egress) are implemented; ACLs between VPC tiers are managed/implemented by ipv6 firewall on VR
> 
> Please take a look at the existing security_group.py script which 
> implements RFC4890
> 
> https://datatracker.ietf.org/doc/html/rfc4890
> 
> ICMPv6 is a vital part of IPv6 and certain packets should always be allowed.
> 
>>        *   It is assumed that static routes are created on the core/main router by the admin or automated using some scripts/tools; for this CloudStack will announce events with details of /64 networks and VR's public IPv6 address that can be consumed by a rabbitmq/message bus client (for example), or a custom cron job or script as part of orchestration. (this wouldn't be necessary for dynamic routing bgp with phase2)\\
> 
> You would only need to announce the /48 or /56 allocated to the VR, 
> that's all. You don't need to inform the upstream router about the /64 
> subnets created within that larger subnet.
> 
>>     *   Guest Networking: With SLAAC, it's easy for CloudStack to calculate allocate and use a /64 and determine the IPv6 address of VR nics and guest VM nics
>>        *   A user create an isolated network/VPC with an offering that is ipv6 enabled
>>        *   A user can manage firewall for the IPv6 address/guest nics; there'll be no port forward and LB feature though for IPv6
>>        *   A users can run workloads in the guest VMs that listen on publically routable ipv6 addresses
>>        *   Usage/billing etc continue to work, no change needed
>>
>> Network layout:
>>
>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] -> 
>> [guest VMs/nics] *core/ISP router needs static routes to be added 
>> (manually or automated), assumes a /48 or /56 configured for the zone
>>
>> Thoughts, feedback?
> 
> Looks doable!
> 
> Side-note: It would be very cool if you could use parts of this 
> implementation to also route /48, /56, or /60 subnets to individual 
> VMs in Shared networks.
> 
> Why? This allows for running Docker containers with native IPv6 inside 
> the VM or running a (Open)VPN server inside a VM which then assigns 
> public IPv6 addresses to clients connected.
> 
> Instead of routing the subnet to a VR we route the subnet to a single 
> instance in a shared network.
> 
> If we could then also move these subnets between Instances easily one 
> can quickly migrate to a different instance while keeping the same 
> IPv6 subnet.
> 
> Wido
> 
>>
>> Proof-of-concept commentary: here's what I did to test the idea:
>>
>>     *   Created an isolated network and deployed a VM in my home lab
>> The VR running on KVM has following nics
>> eth0 - guest network
>> eth1 - link local
>> eth2 - public network
>>
>>     *   I setup a custom openwrt router on a RPi4 to serve as a toy-core router where I create a wan6 IPv6 tunnel using tunnel broker and I got a /48 allocated. My configuration looks like:
>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>
>> I create a LAN ipv6 (public network for CloudStack VR): at subnet/prefix 0:
>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode: SLAAC+stateless 
>> DHCP (no dhcpv6)
>>     *
>>     *
>> In the isolated VR, I enabled ipv6 as:
>> net.ipv6.conf.all.disable_ipv6 = 0
>> net.ipv6.conf.all.forwarding = 1
>> net.ipv6.conf.all.accept_ra = 1
>> net.ipv6.conf.all.accept_redirects = 1 net.ipv6.conf.all.autoconf = 1
>>
>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And configured the 
>> nics:
>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo iface eth2 
>> inet6 auto >> /etc/network/interfaces /etc/init.d/networking restart 
>> Next, restart ACS isolated network without cleanup to have it 
>> reconfigure IPv4 nics, firewall, NAT etc
>>
>>     *
>> Next, I created a /64 network for the isolated guest network on eth0 of VR using radvd:
>>
>> # cat /etc/radvd.conf
>> interface eth0
>> {
>>       AdvSendAdvert on;
>>       MinRtrAdvInterval 5;
>>       MaxRtrAdvInterval 15;
>>       prefix 2001:470:ed36:1::/64
>>       {
>>           AdvOnLink on;
>>           AdvAutonomous on;
>>       };
>> };
>> systemctl restart radvd
>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this ...:1::/64 network
>>     *   Finally I added a static route in toy core-router for the new /64 IPv6 range in the isolated network
>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev <local lan nic>
>>     *
>> ... and I enabled firewall rules to allow any traffic to pass for the 
>> new /64 network
>>
>> And voila all done! I create a domain AAAA record that points to my 
>> guest VM IPv6 address a test webserver on 
>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>>
>> (Note: I'll get rid of the tunnel and request a new /48 block after a 
>> few days, sharing this solely for testing purposes)
>>
>>
>> Regards.
>>
>> ________________________________
>> From: Wido den Hollander <wi...@widodh.nl>
>> Sent: Tuesday, July 20, 2021 12:46
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>>
>>
>> Op 19-07-2021 om 20:38 schreef Kristaps Cudars:
>>> Hi Wido,
>>>
>>> I assume that flouting ip will not work grate with ingress/egress acl on VR.
>>>
>>>    From regular ACS user perspective:
>>> I have Instance with dualstack its running web app on 443.
>>> I want to swap instances for whatever reason.
>>> In case of IPv4 change d-nat rule.
>>> In case of IPv6 if flouting IP was not created upfront he will need to change dns entry that usually has 24h ttl. Inconvenience degradation in experience.
>>>
>>
>> Yes, but, keep in mind that the IP you are using can also be 
>> terminated on the VR where HAProxy proxies request to the backend VM 
>> (could even be
>> v4!)
>>
>> I'm not against DHCPv6, but I have seen many issues with implementing 
>> it. Therefor I always stick to SLAAC.
>>
>>>    From ACS admin perspective:
>>> I don’t want to have these tickets in helpdesk.
>>> You needed to create another flouting IP that it would be seamless- will not work as answer.
>>>
>>
>> I understand that as well.
>>
>> Wido
>>
>>>
>>> On 2021/07/19 09:05:54, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>
>>>>
>>>> Op 16-07-2021 om 21:46 schreef Kristaps Cudars:
>>>>> Hi Wido,
>>>>>
>>>>> Your proposal is to sacrifice ability to reassign IPv6 to instance, have internal domain prefix, and list/db in ACS what IPv6 has been assigned to what instance and go with RA and SLAAC. For route signaling to switch use BGP/OSPFv3 or manual pre-creation.
>>>>>
>>>>
>>>> You can still list the IPs which have been assigned. You'll know 
>>>> exactly what IPv6 address a VM has because of the prefix + MAC. 
>>>> Privacy Extensions need to be disabled in the VM.
>>>>
>>>> This already works in CloudStack in Shared Networks in this way.
>>>>
>>>> Using secondary IPs you can always have 'floating' IPv6 addressess.
>>>>
>>>> Wido
>>>>
>>>>> Option with RA and managed flag that DHCPv6 is in use to support preset information and ability to create route information from ACS is not an option as DHCPv6 its failing?
>>>>>
>>>>>
>>>>> On 2021/07/16 15:17:42, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>
>>>>>>
>>>>>> Op 16-07-2021 om 16:42 schreef Hean Seng:
>>>>>>> Hi Wido,
>>>>>>>
>>>>>>> In current setup,  each Cloudstack have own VR, so in this new  
>>>>>>> IPv6 subnet allocation , each VR (which have Frr) will need to 
>>>>>>> have peering with ISP router (and either BGP or Static Route) , 
>>>>>>> and there is 1000 Acocunts,  it will 1000 BGP session with ISP 
>>>>>>> router ,  Am I right for this ? or I understand wrong .
>>>>>>>
>>>>>>
>>>>>> Yes, that is correct. A /56 would also be sufficient or a /60 
>>>>>> which is enough to allocate a few /64 subnets.
>>>>>>
>>>>>> 1000 BGP connections isn't really a problem for a proper router 
>>>>>> at the ISP. OSPF(v3) would be better, but as I said that's poorly supported.
>>>>>>
>>>>>> The ISP could also install 1000 static routes, but that means 
>>>>>> that the ISP's router needs to have those configured.
>>>>>>
>>>>>> http://docs.frrouting.org/en/latest/ospf6d.html
>>>>>> (While looking up this URL I see that Frr recently put in a lot 
>>>>>> of work in OSPFv3, seems better now)
>>>>>>
>>>>>>> I understand IPv6 is different then IPv4, and in IPv6 it suppose 
>>>>>>> each devices have own IP. It just how to realize in easy way.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>
>>
>>
>   
> 
>> On Fri, Jul 16, 2021 at 8:17 PM Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Op 16-07-2021 om 05:54 schreef Hean Seng:
>>>>>>>>> Hi Wido,
>>>>>>>>>
>>>>>>>>> My initial thought is not like this,  it is the /48 at ISP 
>>>>>>>>> router, and
>>>>>>>> /64
>>>>>>>>> subnet assign to AdvanceZoneVR,   AdvanceZoneVR responsible is
>>>>>>>>> distribule IPv6 ip (from the assigned /64 sunet) to VM,  and not routing
>>>>>>>>> the traffic,   in the VM that get the IPv6 IP will default route to ISP
>>>>>>>>> router as gw.   It can may be a bridge over via Advancezone-VR.
>>>>>>>>>
>>>>>>>>
>>>>>>>> How would you bridge this? That sounds like NAT?
>>>>>>>>
>>>>>>>> IPv6 is meant to be routed. Not to be translated or bridged in any way.
>>>>>>>>
>>>>>>>> The way a made the drawing is exactly how IPv6 should work in a 
>>>>>>>> VPC environment.
>>>>>>>>
>>>>>>>> Traffic flows through the VR where it can do firewalling of the traffic.
>>>>>>>>
>>>>>>>>> However, If do as the way described in the drawing, then i 
>>>>>>>>> suppose will
>>>>>>>> be
>>>>>>>>> another kind of virtual router going to introduce , to get 
>>>>>>>>> hold the /48
>>>>>>>> in
>>>>>>>>> this virtual router right ?
>>>>>>>>>
>>>>>>>>
>>>>>>>> It can be the same VR. But keep in mind that IPv6 != IPv4.
>>>>>>>>
>>>>>>>> The VR will get Frr as a new daemon which can talk BGP with the 
>>>>>>>> upper network to route traffic.
>>>>>>>>
>>>>>>>>> After this,  The Advance Zone, NAT's  VR will peer with this 
>>>>>>>>> new IPv6 VR for getting the IPv6 /64 prefix ?
>>>>>>>>>
>>>>>>>>
>>>>>>>> IPv4 will be behind NAT, but IPv6 will not be behind NAT.
>>>>>>>>
>>>>>>>>> If do in this way, then I guess  you just only need Static 
>>>>>>>>> route, with peering ip both end  as one /48 can have a lot of 
>>>>>>>>> /64 on it.  And
>>>>>>>> hardware
>>>>>>>>> budgeting for new IPv6-VR will become very important, as all 
>>>>>>>>> traffic will need to pass over it .
>>>>>>>>>
>>>>>>>>
>>>>>>>> Routing or NAT is the same for the VR. You don't need a very 
>>>>>>>> beefy VR for this.
>>>>>>>>
>>>>>>>>> It will be like
>>>>>>>>>
>>>>>>>>> ISP Router  ------ >  (new IPV6-VR ) ---- > AdvanceZone-VR 
>>>>>>>>> ----> VM
>>>>>>>>>
>>>>>>>>> Relationship of (new IPv6 VR) and AdvanceZone-VR , may be 
>>>>>>>>> considering on OSPF instead of  BGP , otherwise few thousand 
>>>>>>>>> of AdvanceZone-VR wil have few thousand of BGP session. on 
>>>>>>>>> new-IPv6-VR
>>>>>>>>>
>>>>>>>>> Also, I suppose we cannot do ISP router. -->. Advancezone VR direct,   ,
>>>>>>>>> otherwise ISP router will be full of /64 prefix route either on BGP( Many
>>>>>>>>> BGP Session) , or  Many Static route .   If few thousand account, ti will
>>>>>>>>> be few thousand of BGP session with ISP router or few thousand 
>>>>>>>>> static
>>>>>>>> route
>>>>>>>>> which  is not possible .
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jul 15, 2021 at 10:47 PM Wido den Hollander 
>>>>>>>>> <wi...@widodh.nl>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> But you still need routing. See the attached PNG (and draw.io XML).
>>>>>>>>>>
>>>>>>>>>> You need to route the /48 subnet TO the VR which can then 
>>>>>>>>>> route it to the Virtual Networks behind the VR.
>>>>>>>>>>
>>>>>>>>>> There is no other way then routing with either BGP or a Static route.
>>>>>>>>>>
>>>>>>>>>> Wido
>>>>>>>>>>
>>>>>>>>>> Op 15-07-2021 om 12:39 schreef Hean Seng:
>>>>>>>>>>> Or explain like this :
>>>>>>>>>>>
>>>>>>>>>>> 1) Cloudstack generate list of /64 subnet from /48 that 
>>>>>>>>>>> Network admin assigned to Cloudstack
>>>>>>>>>>> 2) Cloudsack allocated the subnet (that generated from 
>>>>>>>>>>> step1) to
>>>>>>>> Virtual
>>>>>>>>>>> Router, one Virtual Router have one subniet /64
>>>>>>>>>>> 3) Virtual Router allocate single IPv6 (within the range of 
>>>>>>>>>>> /64 allocated to VR)  to VM
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng 
>>>>>>>>>>> <heanseng@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>           Hi Wido,
>>>>>>>>>>>
>>>>>>>>>>>           I think the /48 is at physical router as gateway , 
>>>>>>>>>>> and subnet of
>>>>>>>> /64
>>>>>>>>>>>           at VR of Cloudstack.   Cloudstack only keep which /48 prefix and
>>>>>>>>>>>           vlan information of this /48 to be later split the  /64. to VR.
>>>>>>>>>>>
>>>>>>>>>>>           And the instances is getting singe IPv6 of /64  IP.   The VR is
>>>>>>>>>>>           getting /64.  The default gateway shall goes to /48 of physical
>>>>>>>>>>>           router ip .   In this case ,does not need any BGP router .
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>           Similar concept as IPv4 :
>>>>>>>>>>>
>>>>>>>>>>>           /48 subnet of IPv6 is equivalent to current /24 
>>>>>>>>>>> subnet of IPv4
>>>>>>>> that
>>>>>>>>>>>           created in Network.
>>>>>>>>>>>           and /64  of IPv6 is equivalent to single IP of IPv4 assign to VM.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>           On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander 
>>>>>>>>>>> <
>>>>>>>> wido@widodh.nl
>>>>>>>>>>>           <ma...@widodh.nl>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>               Op 14-07-2021 om 16:44 schreef Hean Seng:
>>>>>>>>>>>                > Hi
>>>>>>>>>>>                >
>>>>>>>>>>>                > I replied in another thread, i think do not need implement
>>>>>>>>>>>               BGP or OSPF,
>>>>>>>>>>>                > that would be complicated .
>>>>>>>>>>>                >
>>>>>>>>>>>                > We only need assign  IPv6 's /64 prefix to Virtual Router
>>>>>>>>>>>               (VR) in NAT
>>>>>>>>>>>                > zone, and the VR responsible to deliver 
>>>>>>>>>>> single IPv6 to VM
>>>>>>>> via
>>>>>>>>>>>               DHCP6.
>>>>>>>>>>>                >
>>>>>>>>>>>                > In VR, you need to have Default IPv6 route to  Physical
>>>>>>>>>>>               Router's /48. IP
>>>>>>>>>>>                > as IPv6 Gateway.  Thens should be done .
>>>>>>>>>>>                >
>>>>>>>>>>>                > Example :
>>>>>>>>>>>                > Physical Router Interface
>>>>>>>>>>>                >   IPv6 IP : 2000:aaaa::1/48
>>>>>>>>>>>                >
>>>>>>>>>>>                > Cloudstack  virtual router : 2000:aaaa:200:201::1/64 with
>>>>>>>>>>>               default ipv6
>>>>>>>>>>>                > route to router ip 2000:aaaa::1
>>>>>>>>>>>                > and Clodustack Virtual router dhcp allocate IP to VM , and
>>>>>>>>>>>               VM will have
>>>>>>>>>>>                > default route to VR. IPv6 2000:aaaa:200:201::1
>>>>>>>>>>>                >
>>>>>>>>>>>                > So in cloudstack need to allow  user to enter ,  IPv6
>>>>>>>>>>>               gwateway , and
>>>>>>>>>>>                > the  /48 Ipv6 prefix , then it will self 
>>>>>>>>>>> allocate the /64
>>>>>>>> ip
>>>>>>>>>>>               to the VR ,
>>>>>>>>>>>                > and maintain make sure not ovelap allocation
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>
>>>>>>>>>>>               But NAT is truly not the solution with IPv6. 
>>>>>>>>>>> IPv6 is supposed
>>>>>>>> to
>>>>>>>>>> be
>>>>>>>>>>>               routable. In addition you should avoid DHCPv6 as much as
>>>>>>>>>>>               possible as
>>>>>>>>>>>               that's not really the intended use-case for address allocation
>>>>>>>>>>>               with IPv6.
>>>>>>>>>>>
>>>>>>>>>>>               In order to route an /48 IPv6 subnet to the VR you have a few
>>>>>>>>>>>               possibilities:
>>>>>>>>>>>
>>>>>>>>>>>               - Static route from the upperlying routers 
>>>>>>>>>>> which are outside
>>>>>>>> of
>>>>>>>>>>>               CloudStack
>>>>>>>>>>>               - BGP
>>>>>>>>>>>               - OSPFv3 (broken in most cases!)
>>>>>>>>>>>               - DHCPv6 Prefix Delegation
>>>>>>>>>>>
>>>>>>>>>>>               BGP and/or Static routes are still the best bet here.
>>>>>>>>>>>
>>>>>>>>>>>               So what you do is that you tell CloudStack that you will route
>>>>>>>>>>>               2001:db8::/48 to the VR, the VR can then use 
>>>>>>>>>>> that to split it
>>>>>>>> up
>>>>>>>>>>>               into
>>>>>>>>>>>               multiple /64 subnets going towards the instances:
>>>>>>>>>>>
>>>>>>>>>>>               - 2001:db8::/64
>>>>>>>>>>>               - 2001:db8:1::/64
>>>>>>>>>>>               - 2001:db8:2::/64
>>>>>>>>>>>               ...
>>>>>>>>>>>               - 2001:db8:f::/64
>>>>>>>>>>>
>>>>>>>>>>>               And go on.
>>>>>>>>>>>
>>>>>>>>>>>               In case of BGP you indeed have to tell the VR a few things:
>>>>>>>>>>>
>>>>>>>>>>>               - It's own AS number
>>>>>>>>>>>               - The peer's address(es)
>>>>>>>>>>>
>>>>>>>>>>>               With FRR you can simply say:
>>>>>>>>>>>
>>>>>>>>>>>               neighbor 2001:db8:4fa::179 remote-as external
>>>>>>>>>>>
>>>>>>>>>>>               The /48 you need to have at the VR anyway in case of either a
>>>>>>>>>>>               static
>>>>>>>>>>>               route or BGP.
>>>>>>>>>>>
>>>>>>>>>>>               We just need to add a NullRoute on the VR for that /48 so that
>>>>>>>>>>>               traffic
>>>>>>>>>>>               will not be routed to the upper gateway in 
>>>>>>>>>>> case of the VR
>>>>>>>> can't
>>>>>>>>>>>               find a
>>>>>>>>>>>               route.
>>>>>>>>>>>
>>>>>>>>>>>               Wido
>>>>>>>>>>>
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli
>>>>>>>>>>>                > <Alex.Mattioli@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>
>>>>>>>>>>>               <mailto:Alex.Mattioli@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>>> wrote:
>>>>>>>>>>>                >
>>>>>>>>>>>                >     Hi Wido,
>>>>>>>>>>>                >     That's pretty much in line with our thoughts, thanks
>>>>>>>> for
>>>>>>>>>>>               the input.
>>>>>>>>>>>                >     I believe we agree on the following points then:
>>>>>>>>>>>                >
>>>>>>>>>>>                >     - FRR with BGP (no OSPF)
>>>>>>>>>>>                >     - Route /48 (or/56) down to the VR
>>>>>>>>>>>                >     - /64 per network
>>>>>>>>>>>                >     - SLACC for IP addressing
>>>>>>>>>>>                >
>>>>>>>>>>>                >     I believe the next big question is then "on which level
>>>>>>>>>>>               of ACS do we
>>>>>>>>>>>                >     manage AS numbers?".  I see two options:
>>>>>>>>>>>                >     1) Private AS number on a per-zone basis
>>>>>>>>>>>                >     2) Root Admin assigned AS number on a domain/account
>>>>>>>> basis
>>>>>>>>>>>                >     3) End-user driven AS number on a per network basis
>>>>>>>> (for
>>>>>>>>>>>               bring your
>>>>>>>>>>>                >     own AS and IP scenario)
>>>>>>>>>>>                >
>>>>>>>>>>>                >     Thoughts?
>>>>>>>>>>>                >
>>>>>>>>>>>                >     Cheers
>>>>>>>>>>>                >     Alex
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >     -----Original Message-----
>>>>>>>>>>>                >     From: Wido den Hollander <wido@widodh.nl
>>>>>>>>>>>               <ma...@widodh.nl> <mailto:wido@widodh.nl
>>>>>>>>>>>               <ma...@widodh.nl>>>
>>>>>>>>>>>                >     Sent: 13 July 2021 15:08
>>>>>>>>>>>                >     To: dev@cloudstack.apache.org
>>>>>>>>>>>               <ma...@cloudstack.apache.org>
>>>>>>>>>>>               <mailto:dev@cloudstack.apache.org
>>>>>>>>>>>               <ma...@cloudstack.apache.org>>;
>>>>>>>>>>>                >     Alex Mattioli <Alex.Mattioli@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>
>>>>>>>>>>>                >     <mailto:Alex.Mattioli@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>>>
>>>>>>>>>>>                >     Cc: Wei Zhou <Wei.Zhou@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>
>>>>>>>>>>>                >     <mailto:Wei.Zhou@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>>>; Rohit Yadav
>>>>>>>>>>>                >     <rohit.yadav@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>
>>>>>>>>>>>               <mailto:rohit.yadav@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>>>;
>>>>>>>>>>>                >     Gabriel Beims Bräscher <gabriel@pcextreme.nl
>>>>>>>>>>>               <ma...@pcextreme.nl>
>>>>>>>>>>>                >     <mailto:gabriel@pcextreme.nl <mailto:
>>>>>>>> gabriel@pcextreme.nl
>>>>>>>>>>>>>
>>>>>>>>>>>                >     Subject: Re: IPV6 in Isolated/VPC networks
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >     On 7/7/21 1:16 PM, Alex Mattioli wrote:
>>>>>>>>>>>                >      > Hi all,
>>>>>>>>>>>                >      > @Wei Zhou<mailto:Wei.Zhou@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>
>>>>>>>>>>>                >     <mailto:Wei.Zhou@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>>> @Rohit
>>>>>>>>>>>                >     Yadav<mailto:rohit.yadav@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>
>>>>>>>>>>>                >     <mailto:rohit.yadav@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>>> and myself are
>>>>>>>>>>>               investigating how
>>>>>>>>>>>                >     to enable IPV6 support on Isolated and VPC networks and
>>>>>>>>>>>               would like
>>>>>>>>>>>                >     your input on it.
>>>>>>>>>>>                >      > At the moment we are looking at implementing FRR
>>>>>>>> with
>>>>>>>>>>>               BGP (and
>>>>>>>>>>>                >     possibly OSPF) on the ACS VR.
>>>>>>>>>>>                >      >
>>>>>>>>>>>                >      > We are looking for requirements, recommendations,
>>>>>>>>>>>               ideas, rants,
>>>>>>>>>>>                >     etc...etc...
>>>>>>>>>>>                >      >
>>>>>>>>>>>                >
>>>>>>>>>>>                >     Ok! Here we go.
>>>>>>>>>>>                >
>>>>>>>>>>>                >     I think that you mean that the VR will actually route
>>>>>>>> the
>>>>>>>>>>>               IPv6
>>>>>>>>>>>                >     traffic and for that you need to have a way of getting
>>>>>>>> a
>>>>>>>>>>>               subnet
>>>>>>>>>>>                >     routed to the VR.
>>>>>>>>>>>                >
>>>>>>>>>>>                >     BGP is probably you best bet here. Although OSPFv3
>>>>>>>>>>>               technically
>>>>>>>>>>>                >     supports this it is very badly implemented in Frr for
>>>>>>>>>>>               example.
>>>>>>>>>>>                >
>>>>>>>>>>>                >     Now FRR is a very good router and one of the fancy
>>>>>>>>>>>               features it
>>>>>>>>>>>                >     supports is BGP Unnumered. This allows for auto
>>>>>>>>>>>               configuration of BGP
>>>>>>>>>>>                >     over a L2 network when both sides are sending Router
>>>>>>>>>>>               Advertisements.
>>>>>>>>>>>                >     This is very easy for flexible BGP configurations where
>>>>>>>>>>>               both sides
>>>>>>>>>>>                >     have dynamic IPs.
>>>>>>>>>>>                >
>>>>>>>>>>>                >     What you want to do is that you get a /56, /48 or
>>>>>>>>>>>               something which is
>>>>>>>>>>>                >      >/64 bits routed to the VR.
>>>>>>>>>>>                >
>>>>>>>>>>>                >     Now you can sub-segment this into separate /64 subnets.
>>>>>>>>>>>               You don't
>>>>>>>>>>>                >     want to go smaller then a /64 is that prevents you from
>>>>>>>>>>>               using SLAAC
>>>>>>>>>>>                >     for IPv6 address configuration. This is how it works
>>>>>>>> for
>>>>>>>>>>>               Shared
>>>>>>>>>>>                >     Networks now in Basic and Advanced Zones.
>>>>>>>>>>>                >
>>>>>>>>>>>                >     FRR can now also send out the Router Advertisements on
>>>>>>>>>>>               the downlinks
>>>>>>>>>>>                >     sending out:
>>>>>>>>>>>                >
>>>>>>>>>>>                >     - DNS servers
>>>>>>>>>>>                >     - DNS domain
>>>>>>>>>>>                >     - Prefix (/64) to be used
>>>>>>>>>>>                >
>>>>>>>>>>>                >     There is no need for DHCPv6. You can calculate the IPv6
>>>>>>>>>>>               address the
>>>>>>>>>>>                >     VM will obtain by using the MAC and the prefix.
>>>>>>>>>>>                >
>>>>>>>>>>>                >     So in short:
>>>>>>>>>>>                >
>>>>>>>>>>>                >     - Using BGP you routed a /48 to the VR
>>>>>>>>>>>                >     - Now you split this into /64 subnets towards the
>>>>>>>>>>>               isolated networks
>>>>>>>>>>>                >
>>>>>>>>>>>                >     Wido
>>>>>>>>>>>                >
>>>>>>>>>>>                >      > Alex Mattioli
>>>>>>>>>>>                >      >
>>>>>>>>>>>                >      >
>>>>>>>>>>>                >      >
>>>>>>>>>>>                >      >
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                > --
>>>>>>>>>>>                > Regards,
>>>>>>>>>>>                > Hean Seng
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>           --
>>>>>>>>>>>           Regards,
>>>>>>>>>>>           Hean Seng
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Hean Seng
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>
> 

Re: IPV6 in Isolated/VPC networks

Posted by Wido den Hollander <wi...@widodh.nl>.

Op 16-08-2021 om 11:29 schreef Rohit Yadav:
> Thanks Hean, Kristaps, Wido for your feedback.
> 
> I think we've some quorum and consensus on how we should proceed with IPv6 support with static routing (phase1). Based on my proof-of-concept and discussions, I believe we may target this feature as early as 4.17 and I welcome offer by Kristaps and others who may want to be involved in testing the feature as and when we'll develop it.
> 
> As the next step I'll write a short design doc on cwiki including some of the new ideas/suggestions and share it on this thread for another iteration.
> 

I was thinking about this and I have nother idea for phase 2.

As mentioned earlier it can be quite difficult when you have a lot of 
VRs running to configure OSPFv3 or BGP in the upstream routers to have 
this all working.

Instead we could think about using ExaBGP. With ExaBGP you would have 
some additional tooling which picks up the subnets assigned to VRs from 
the messagebus.

ExaBGP: https://github.com/Thomas-Mangin/exabgp

This tooling then dynamically injects this route into ExaBGP where the 
destination of thus /48 (example) points to the VR.

Inside the VR there is no need to configure OSPF or BGP. The admin 
doesn't need to configure static routers either.

Their routers only peer via BGP with ExaBGP which injects the routes.

See this blogpost to get an idea: 
https://vincent.bernat.ch/en/blog/2013-exabgp-highavailability

"Redundancy with ExaBGP"
"ExaBGP is a convenient tool to plug scripts into BGP. They can then 
receive and advertise routes. ExaBGP does the hard work of speaking BGP 
with your routers. The scripts just have to read routes from standard 
input or advertise them on standard output."

So we would just need to feed ExaBGP all the routes which then 
advertises them again towards the upstream routers.

ExaBGP could be running anywhere as long as it receives the messages 
from CloudStack and has a BGP connection with the routers.

Might be worth looking into in the future for phase 2.

And again: Also in shared networks it would be very nice to be able to 
route a subnet towards a single Instance. Native IPv6 with Docker, VPN 
services with IPv6 inside a VM, etc, etc.

Wido

> 
> Regards.
> 
> ________________________________
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: Friday, August 13, 2021 15:48
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
> 
> Hi,
> 
> See my inline responses:
> 
> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>> Hi all,
>>
>> Thanks for your feedback and ideas, I've gone ahead with discussing them with Alex and came up with a PoC/design which can be implemented in the following phases:
>>
>>     *   Phase1: implement ipv6 support in isolated networks and VPC with static routing
>>     *   Phase2: discuss and implement support for dynamic routing (TBD)
>>
>> For Phase1 here's the high-level proposal:
>>
>>     *   IPv6 address management:
>>        *   At the zone level root-admin specifies a /64 public range that will be used for VRs, then they can add a /48, or /56 IPv6 range for guest networks (to be used by isolated networks and VPC tiers)
>>        *   On creation of any IPv6 enabled isolated network or VPC tier, from the /48 or /56 block a /64 network is allocated/used
>>        *   We assume SLAAC and autoconfiguration, no DHCPv6 in the zone (discuss: is privacy a concern, can privacy extensions rfc4941 of slaac be explored?)
> 
> Privacy Extensions are only a concern for client devices which roam
> between different IPv6 networks.
> 
> If you IPv6 address of a client keeps the same suffix (MAC based) and
> switches network then only the prefix (/64) will change.
> 
> This way a network like Google, Facebook, etc could track your device
> moving from network to network if they only look at the last 64-bits of
> the IPv6 address.
> 
> For servers this is not a problem as you already know in which network
> they are.
> 
>>     *   Network offerings: root-admin can create new network offerings (with VPC too) that specifies a network stack option:
>>        *   ipv4 only (default, for backward compatibility all networks/offerings post-upgrade migrate to this option)
>>        *   ipv4-and-ipv6
>>        *   ipv6-only (this can be phase 1.b)
>>        *   A new routing option: static (phase1), dynamic (phase2, with multiple sub-options such as ospf/bgp etc...)
> 
> This means that the network admin will need to statically route the IPv6
> subnet to the VR's outside IPv6 address, for example, on a JunOS router:
> 
> set routing-options rib inet6.0 static route 2001:db8:500::/48 next-hop
> 2001:db8:100::50
> 
> I'm assuming that 2001:db8:100::50 is the address of the VR on the
> outside (/64) network. In reality this will probably be a longer
> address, but this is for just the example.
> 
>>     *   VR changes:
>>        *   VR gets its guest and public nics set to inet6 auto
>>        *   For each /64 allocated to guest network and VPC tiers, radvd is configured to do RA
> 
> radvd is fine, but looking at phase 2 with dynamic routing you might
> already want to look into FRRouting. FRR can also advertise RAs while
> not doing any routing.
> 
> interface ens4
>     no ipv6 nd suppress-ra
>     ipv6 nd prefix 2001:db8:500::/64
>     ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> 
> See: http://docs.frrouting.org/en/latest/ipv6.html
> 
>>        *   Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6 firewall rules (ACLs, ingress, egress) are implemented; ACLs between VPC tiers are managed/implemented by ipv6 firewall on VR
> 
> Please take a look at the existing security_group.py script which
> implements RFC4890
> 
> https://datatracker.ietf.org/doc/html/rfc4890
> 
> ICMPv6 is a vital part of IPv6 and certain packets should always be allowed.
> 
>>        *   It is assumed that static routes are created on the core/main router by the admin or automated using some scripts/tools; for this CloudStack will announce events with details of /64 networks and VR's public IPv6 address that can be consumed by a rabbitmq/message bus client (for example), or a custom cron job or script as part of orchestration. (this wouldn't be necessary for dynamic routing bgp with phase2)\\
> 
> You would only need to announce the /48 or /56 allocated to the VR,
> that's all. You don't need to inform the upstream router about the /64
> subnets created within that larger subnet.
> 
>>     *   Guest Networking: With SLAAC, it's easy for CloudStack to calculate allocate and use a /64 and determine the IPv6 address of VR nics and guest VM nics
>>        *   A user create an isolated network/VPC with an offering that is ipv6 enabled
>>        *   A user can manage firewall for the IPv6 address/guest nics; there'll be no port forward and LB feature though for IPv6
>>        *   A users can run workloads in the guest VMs that listen on publically routable ipv6 addresses
>>        *   Usage/billing etc continue to work, no change needed
>>
>> Network layout:
>>
>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] -> [guest VMs/nics]
>> *core/ISP router needs static routes to be added (manually or automated), assumes a /48 or /56 configured for the zone
>>
>> Thoughts, feedback?
> 
> Looks doable!
> 
> Side-note: It would be very cool if you could use parts of this
> implementation to also route /48, /56, or /60 subnets to individual VMs
> in Shared networks.
> 
> Why? This allows for running Docker containers with native IPv6 inside
> the VM or running a (Open)VPN server inside a VM which then assigns
> public IPv6 addresses to clients connected.
> 
> Instead of routing the subnet to a VR we route the subnet to a single
> instance in a shared network.
> 
> If we could then also move these subnets between Instances easily one
> can quickly migrate to a different instance while keeping the same IPv6
> subnet.
> 
> Wido
> 
>>
>> Proof-of-concept commentary: here's what I did to test the idea:
>>
>>     *   Created an isolated network and deployed a VM in my home lab
>> The VR running on KVM has following nics
>> eth0 - guest network
>> eth1 - link local
>> eth2 - public network
>>
>>     *   I setup a custom openwrt router on a RPi4 to serve as a toy-core router where I create a wan6 IPv6 tunnel using tunnel broker and I got a /48 allocated. My configuration looks like:
>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>
>> I create a LAN ipv6 (public network for CloudStack VR): at subnet/prefix 0:
>> LAN IPv6 address: 2001:470:ed36:0::1/64
>> Address mode: SLAAC+stateless DHCP (no dhcpv6)
>>     *
>>     *
>> In the isolated VR, I enabled ipv6 as:
>> net.ipv6.conf.all.disable_ipv6 = 0
>> net.ipv6.conf.all.forwarding = 1
>> net.ipv6.conf.all.accept_ra = 1
>> net.ipv6.conf.all.accept_redirects = 1
>> net.ipv6.conf.all.autoconf = 1
>>
>> Set up a IPv6 nameserver/dns in /etc/resolve.conf
>> And configured the nics:
>> echo iface eth0 inet6 auto >> /etc/network/interfaces
>> echo iface eth2 inet6 auto >> /etc/network/interfaces
>> /etc/init.d/networking restart
>> Next, restart ACS isolated network without cleanup to have it reconfigure IPv4 nics, firewall, NAT etc
>>
>>     *
>> Next, I created a /64 network for the isolated guest network on eth0 of VR using radvd:
>>
>> # cat /etc/radvd.conf
>> interface eth0
>> {
>>       AdvSendAdvert on;
>>       MinRtrAdvInterval 5;
>>       MaxRtrAdvInterval 15;
>>       prefix 2001:470:ed36:1::/64
>>       {
>>           AdvOnLink on;
>>           AdvAutonomous on;
>>       };
>> };
>> systemctl restart radvd
>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this ...:1::/64 network
>>     *   Finally I added a static route in toy core-router for the new /64 IPv6 range in the isolated network
>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev <local lan nic>
>>     *
>> ... and I enabled firewall rules to allow any traffic to pass for the new /64 network
>>
>> And voila all done! I create a domain AAAA record that points to my guest VM IPv6 address a test webserver on
>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>>
>> (Note: I'll get rid of the tunnel and request a new /48 block after a few days, sharing this solely for testing purposes)
>>
>>
>> Regards.
>>
>> ________________________________
>> From: Wido den Hollander <wi...@widodh.nl>
>> Sent: Tuesday, July 20, 2021 12:46
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>>
>>
>> Op 19-07-2021 om 20:38 schreef Kristaps Cudars:
>>> Hi Wido,
>>>
>>> I assume that flouting ip will not work grate with ingress/egress acl on VR.
>>>
>>>    From regular ACS user perspective:
>>> I have Instance with dualstack its running web app on 443.
>>> I want to swap instances for whatever reason.
>>> In case of IPv4 change d-nat rule.
>>> In case of IPv6 if flouting IP was not created upfront he will need to change dns entry that usually has 24h ttl. Inconvenience degradation in experience.
>>>
>>
>> Yes, but, keep in mind that the IP you are using can also be terminated
>> on the VR where HAProxy proxies request to the backend VM (could even be
>> v4!)
>>
>> I'm not against DHCPv6, but I have seen many issues with implementing
>> it. Therefor I always stick to SLAAC.
>>
>>>    From ACS admin perspective:
>>> I don’t want to have these tickets in helpdesk.
>>> You needed to create another flouting IP that it would be seamless- will not work as answer.
>>>
>>
>> I understand that as well.
>>
>> Wido
>>
>>>
>>> On 2021/07/19 09:05:54, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>
>>>>
>>>> Op 16-07-2021 om 21:46 schreef Kristaps Cudars:
>>>>> Hi Wido,
>>>>>
>>>>> Your proposal is to sacrifice ability to reassign IPv6 to instance, have internal domain prefix, and list/db in ACS what IPv6 has been assigned to what instance and go with RA and SLAAC. For route signaling to switch use BGP/OSPFv3 or manual pre-creation.
>>>>>
>>>>
>>>> You can still list the IPs which have been assigned. You'll know exactly
>>>> what IPv6 address a VM has because of the prefix + MAC. Privacy
>>>> Extensions need to be disabled in the VM.
>>>>
>>>> This already works in CloudStack in Shared Networks in this way.
>>>>
>>>> Using secondary IPs you can always have 'floating' IPv6 addressess.
>>>>
>>>> Wido
>>>>
>>>>> Option with RA and managed flag that DHCPv6 is in use to support preset information and ability to create route information from ACS is not an option as DHCPv6 its failing?
>>>>>
>>>>>
>>>>> On 2021/07/16 15:17:42, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>
>>>>>>
>>>>>> Op 16-07-2021 om 16:42 schreef Hean Seng:
>>>>>>> Hi Wido,
>>>>>>>
>>>>>>> In current setup,  each Cloudstack have own VR, so in this new  IPv6 subnet
>>>>>>> allocation , each VR (which have Frr) will need to have peering with ISP
>>>>>>> router (and either BGP or Static Route) , and there is 1000 Acocunts,  it
>>>>>>> will 1000 BGP session with ISP router ,  Am I right for this ? or I
>>>>>>> understand wrong .
>>>>>>>
>>>>>>
>>>>>> Yes, that is correct. A /56 would also be sufficient or a /60 which is
>>>>>> enough to allocate a few /64 subnets.
>>>>>>
>>>>>> 1000 BGP connections isn't really a problem for a proper router at the
>>>>>> ISP. OSPF(v3) would be better, but as I said that's poorly supported.
>>>>>>
>>>>>> The ISP could also install 1000 static routes, but that means that the
>>>>>> ISP's router needs to have those configured.
>>>>>>
>>>>>> http://docs.frrouting.org/en/latest/ospf6d.html
>>>>>> (While looking up this URL I see that Frr recently put in a lot of work
>>>>>> in OSPFv3, seems better now)
>>>>>>
>>>>>>> I understand IPv6 is different then IPv4, and in IPv6 it suppose each
>>>>>>> devices have own IP. It just how to realize in easy way.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>
>>
>>
>   
> 
>> On Fri, Jul 16, 2021 at 8:17 PM Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Op 16-07-2021 om 05:54 schreef Hean Seng:
>>>>>>>>> Hi Wido,
>>>>>>>>>
>>>>>>>>> My initial thought is not like this,  it is the /48 at ISP router, and
>>>>>>>> /64
>>>>>>>>> subnet assign to AdvanceZoneVR,   AdvanceZoneVR responsible is
>>>>>>>>> distribule IPv6 ip (from the assigned /64 sunet) to VM,  and not routing
>>>>>>>>> the traffic,   in the VM that get the IPv6 IP will default route to ISP
>>>>>>>>> router as gw.   It can may be a bridge over via Advancezone-VR.
>>>>>>>>>
>>>>>>>>
>>>>>>>> How would you bridge this? That sounds like NAT?
>>>>>>>>
>>>>>>>> IPv6 is meant to be routed. Not to be translated or bridged in any way.
>>>>>>>>
>>>>>>>> The way a made the drawing is exactly how IPv6 should work in a VPC
>>>>>>>> environment.
>>>>>>>>
>>>>>>>> Traffic flows through the VR where it can do firewalling of the traffic.
>>>>>>>>
>>>>>>>>> However, If do as the way described in the drawing, then i suppose will
>>>>>>>> be
>>>>>>>>> another kind of virtual router going to introduce , to get hold the /48
>>>>>>>> in
>>>>>>>>> this virtual router right ?
>>>>>>>>>
>>>>>>>>
>>>>>>>> It can be the same VR. But keep in mind that IPv6 != IPv4.
>>>>>>>>
>>>>>>>> The VR will get Frr as a new daemon which can talk BGP with the upper
>>>>>>>> network to route traffic.
>>>>>>>>
>>>>>>>>> After this,  The Advance Zone, NAT's  VR will peer with this new IPv6 VR
>>>>>>>>> for getting the IPv6 /64 prefix ?
>>>>>>>>>
>>>>>>>>
>>>>>>>> IPv4 will be behind NAT, but IPv6 will not be behind NAT.
>>>>>>>>
>>>>>>>>> If do in this way, then I guess  you just only need Static route, with
>>>>>>>>> peering ip both end  as one /48 can have a lot of /64 on it.  And
>>>>>>>> hardware
>>>>>>>>> budgeting for new IPv6-VR will become very important, as all traffic will
>>>>>>>>> need to pass over it .
>>>>>>>>>
>>>>>>>>
>>>>>>>> Routing or NAT is the same for the VR. You don't need a very beefy VR
>>>>>>>> for this.
>>>>>>>>
>>>>>>>>> It will be like
>>>>>>>>>
>>>>>>>>> ISP Router  ------ >  (new IPV6-VR ) ---- > AdvanceZone-VR ----> VM
>>>>>>>>>
>>>>>>>>> Relationship of (new IPv6 VR) and AdvanceZone-VR , may be considering on
>>>>>>>>> OSPF instead of  BGP , otherwise few thousand of AdvanceZone-VR wil have
>>>>>>>>> few thousand of BGP session. on new-IPv6-VR
>>>>>>>>>
>>>>>>>>> Also, I suppose we cannot do ISP router. -->. Advancezone VR direct,   ,
>>>>>>>>> otherwise ISP router will be full of /64 prefix route either on BGP( Many
>>>>>>>>> BGP Session) , or  Many Static route .   If few thousand account, ti will
>>>>>>>>> be few thousand of BGP session with ISP router or few thousand static
>>>>>>>> route
>>>>>>>>> which  is not possible .
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jul 15, 2021 at 10:47 PM Wido den Hollander <wi...@widodh.nl>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> But you still need routing. See the attached PNG (and draw.io XML).
>>>>>>>>>>
>>>>>>>>>> You need to route the /48 subnet TO the VR which can then route it to
>>>>>>>>>> the Virtual Networks behind the VR.
>>>>>>>>>>
>>>>>>>>>> There is no other way then routing with either BGP or a Static route.
>>>>>>>>>>
>>>>>>>>>> Wido
>>>>>>>>>>
>>>>>>>>>> Op 15-07-2021 om 12:39 schreef Hean Seng:
>>>>>>>>>>> Or explain like this :
>>>>>>>>>>>
>>>>>>>>>>> 1) Cloudstack generate list of /64 subnet from /48 that Network admin
>>>>>>>>>>> assigned to Cloudstack
>>>>>>>>>>> 2) Cloudsack allocated the subnet (that generated from step1) to
>>>>>>>> Virtual
>>>>>>>>>>> Router, one Virtual Router have one subniet /64
>>>>>>>>>>> 3) Virtual Router allocate single IPv6 (within the range of /64
>>>>>>>>>>> allocated to VR)  to VM
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng <heanseng@gmail.com
>>>>>>>>>>> <ma...@gmail.com>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>           Hi Wido,
>>>>>>>>>>>
>>>>>>>>>>>           I think the /48 is at physical router as gateway , and subnet of
>>>>>>>> /64
>>>>>>>>>>>           at VR of Cloudstack.   Cloudstack only keep which /48 prefix and
>>>>>>>>>>>           vlan information of this /48 to be later split the  /64. to VR.
>>>>>>>>>>>
>>>>>>>>>>>           And the instances is getting singe IPv6 of /64  IP.   The VR is
>>>>>>>>>>>           getting /64.  The default gateway shall goes to /48 of physical
>>>>>>>>>>>           router ip .   In this case ,does not need any BGP router .
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>           Similar concept as IPv4 :
>>>>>>>>>>>
>>>>>>>>>>>           /48 subnet of IPv6 is equivalent to current /24 subnet of IPv4
>>>>>>>> that
>>>>>>>>>>>           created in Network.
>>>>>>>>>>>           and /64  of IPv6 is equivalent to single IP of IPv4 assign to VM.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>           On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander <
>>>>>>>> wido@widodh.nl
>>>>>>>>>>>           <ma...@widodh.nl>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>               Op 14-07-2021 om 16:44 schreef Hean Seng:
>>>>>>>>>>>                > Hi
>>>>>>>>>>>                >
>>>>>>>>>>>                > I replied in another thread, i think do not need implement
>>>>>>>>>>>               BGP or OSPF,
>>>>>>>>>>>                > that would be complicated .
>>>>>>>>>>>                >
>>>>>>>>>>>                > We only need assign  IPv6 's /64 prefix to Virtual Router
>>>>>>>>>>>               (VR) in NAT
>>>>>>>>>>>                > zone, and the VR responsible to deliver single IPv6 to VM
>>>>>>>> via
>>>>>>>>>>>               DHCP6.
>>>>>>>>>>>                >
>>>>>>>>>>>                > In VR, you need to have Default IPv6 route to  Physical
>>>>>>>>>>>               Router's /48. IP
>>>>>>>>>>>                > as IPv6 Gateway.  Thens should be done .
>>>>>>>>>>>                >
>>>>>>>>>>>                > Example :
>>>>>>>>>>>                > Physical Router Interface
>>>>>>>>>>>                >   IPv6 IP : 2000:aaaa::1/48
>>>>>>>>>>>                >
>>>>>>>>>>>                > Cloudstack  virtual router : 2000:aaaa:200:201::1/64 with
>>>>>>>>>>>               default ipv6
>>>>>>>>>>>                > route to router ip 2000:aaaa::1
>>>>>>>>>>>                > and Clodustack Virtual router dhcp allocate IP to VM , and
>>>>>>>>>>>               VM will have
>>>>>>>>>>>                > default route to VR. IPv6 2000:aaaa:200:201::1
>>>>>>>>>>>                >
>>>>>>>>>>>                > So in cloudstack need to allow  user to enter ,  IPv6
>>>>>>>>>>>               gwateway , and
>>>>>>>>>>>                > the  /48 Ipv6 prefix , then it will self allocate the /64
>>>>>>>> ip
>>>>>>>>>>>               to the VR ,
>>>>>>>>>>>                > and maintain make sure not ovelap allocation
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>
>>>>>>>>>>>               But NAT is truly not the solution with IPv6. IPv6 is supposed
>>>>>>>> to
>>>>>>>>>> be
>>>>>>>>>>>               routable. In addition you should avoid DHCPv6 as much as
>>>>>>>>>>>               possible as
>>>>>>>>>>>               that's not really the intended use-case for address allocation
>>>>>>>>>>>               with IPv6.
>>>>>>>>>>>
>>>>>>>>>>>               In order to route an /48 IPv6 subnet to the VR you have a few
>>>>>>>>>>>               possibilities:
>>>>>>>>>>>
>>>>>>>>>>>               - Static route from the upperlying routers which are outside
>>>>>>>> of
>>>>>>>>>>>               CloudStack
>>>>>>>>>>>               - BGP
>>>>>>>>>>>               - OSPFv3 (broken in most cases!)
>>>>>>>>>>>               - DHCPv6 Prefix Delegation
>>>>>>>>>>>
>>>>>>>>>>>               BGP and/or Static routes are still the best bet here.
>>>>>>>>>>>
>>>>>>>>>>>               So what you do is that you tell CloudStack that you will route
>>>>>>>>>>>               2001:db8::/48 to the VR, the VR can then use that to split it
>>>>>>>> up
>>>>>>>>>>>               into
>>>>>>>>>>>               multiple /64 subnets going towards the instances:
>>>>>>>>>>>
>>>>>>>>>>>               - 2001:db8::/64
>>>>>>>>>>>               - 2001:db8:1::/64
>>>>>>>>>>>               - 2001:db8:2::/64
>>>>>>>>>>>               ...
>>>>>>>>>>>               - 2001:db8:f::/64
>>>>>>>>>>>
>>>>>>>>>>>               And go on.
>>>>>>>>>>>
>>>>>>>>>>>               In case of BGP you indeed have to tell the VR a few things:
>>>>>>>>>>>
>>>>>>>>>>>               - It's own AS number
>>>>>>>>>>>               - The peer's address(es)
>>>>>>>>>>>
>>>>>>>>>>>               With FRR you can simply say:
>>>>>>>>>>>
>>>>>>>>>>>               neighbor 2001:db8:4fa::179 remote-as external
>>>>>>>>>>>
>>>>>>>>>>>               The /48 you need to have at the VR anyway in case of either a
>>>>>>>>>>>               static
>>>>>>>>>>>               route or BGP.
>>>>>>>>>>>
>>>>>>>>>>>               We just need to add a NullRoute on the VR for that /48 so that
>>>>>>>>>>>               traffic
>>>>>>>>>>>               will not be routed to the upper gateway in case of the VR
>>>>>>>> can't
>>>>>>>>>>>               find a
>>>>>>>>>>>               route.
>>>>>>>>>>>
>>>>>>>>>>>               Wido
>>>>>>>>>>>
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli
>>>>>>>>>>>                > <Alex.Mattioli@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>
>>>>>>>>>>>               <mailto:Alex.Mattioli@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>>> wrote:
>>>>>>>>>>>                >
>>>>>>>>>>>                >     Hi Wido,
>>>>>>>>>>>                >     That's pretty much in line with our thoughts, thanks
>>>>>>>> for
>>>>>>>>>>>               the input.
>>>>>>>>>>>                >     I believe we agree on the following points then:
>>>>>>>>>>>                >
>>>>>>>>>>>                >     - FRR with BGP (no OSPF)
>>>>>>>>>>>                >     - Route /48 (or/56) down to the VR
>>>>>>>>>>>                >     - /64 per network
>>>>>>>>>>>                >     - SLACC for IP addressing
>>>>>>>>>>>                >
>>>>>>>>>>>                >     I believe the next big question is then "on which level
>>>>>>>>>>>               of ACS do we
>>>>>>>>>>>                >     manage AS numbers?".  I see two options:
>>>>>>>>>>>                >     1) Private AS number on a per-zone basis
>>>>>>>>>>>                >     2) Root Admin assigned AS number on a domain/account
>>>>>>>> basis
>>>>>>>>>>>                >     3) End-user driven AS number on a per network basis
>>>>>>>> (for
>>>>>>>>>>>               bring your
>>>>>>>>>>>                >     own AS and IP scenario)
>>>>>>>>>>>                >
>>>>>>>>>>>                >     Thoughts?
>>>>>>>>>>>                >
>>>>>>>>>>>                >     Cheers
>>>>>>>>>>>                >     Alex
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >     -----Original Message-----
>>>>>>>>>>>                >     From: Wido den Hollander <wido@widodh.nl
>>>>>>>>>>>               <ma...@widodh.nl> <mailto:wido@widodh.nl
>>>>>>>>>>>               <ma...@widodh.nl>>>
>>>>>>>>>>>                >     Sent: 13 July 2021 15:08
>>>>>>>>>>>                >     To: dev@cloudstack.apache.org
>>>>>>>>>>>               <ma...@cloudstack.apache.org>
>>>>>>>>>>>               <mailto:dev@cloudstack.apache.org
>>>>>>>>>>>               <ma...@cloudstack.apache.org>>;
>>>>>>>>>>>                >     Alex Mattioli <Alex.Mattioli@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>
>>>>>>>>>>>                >     <mailto:Alex.Mattioli@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>>>
>>>>>>>>>>>                >     Cc: Wei Zhou <Wei.Zhou@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>
>>>>>>>>>>>                >     <mailto:Wei.Zhou@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>>>; Rohit Yadav
>>>>>>>>>>>                >     <rohit.yadav@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>
>>>>>>>>>>>               <mailto:rohit.yadav@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>>>;
>>>>>>>>>>>                >     Gabriel Beims Bräscher <gabriel@pcextreme.nl
>>>>>>>>>>>               <ma...@pcextreme.nl>
>>>>>>>>>>>                >     <mailto:gabriel@pcextreme.nl <mailto:
>>>>>>>> gabriel@pcextreme.nl
>>>>>>>>>>>>>
>>>>>>>>>>>                >     Subject: Re: IPV6 in Isolated/VPC networks
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >     On 7/7/21 1:16 PM, Alex Mattioli wrote:
>>>>>>>>>>>                >      > Hi all,
>>>>>>>>>>>                >      > @Wei Zhou<mailto:Wei.Zhou@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>
>>>>>>>>>>>                >     <mailto:Wei.Zhou@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>>> @Rohit
>>>>>>>>>>>                >     Yadav<mailto:rohit.yadav@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>
>>>>>>>>>>>                >     <mailto:rohit.yadav@shapeblue.com
>>>>>>>>>>>               <ma...@shapeblue.com>>> and myself are
>>>>>>>>>>>               investigating how
>>>>>>>>>>>                >     to enable IPV6 support on Isolated and VPC networks and
>>>>>>>>>>>               would like
>>>>>>>>>>>                >     your input on it.
>>>>>>>>>>>                >      > At the moment we are looking at implementing FRR
>>>>>>>> with
>>>>>>>>>>>               BGP (and
>>>>>>>>>>>                >     possibly OSPF) on the ACS VR.
>>>>>>>>>>>                >      >
>>>>>>>>>>>                >      > We are looking for requirements, recommendations,
>>>>>>>>>>>               ideas, rants,
>>>>>>>>>>>                >     etc...etc...
>>>>>>>>>>>                >      >
>>>>>>>>>>>                >
>>>>>>>>>>>                >     Ok! Here we go.
>>>>>>>>>>>                >
>>>>>>>>>>>                >     I think that you mean that the VR will actually route
>>>>>>>> the
>>>>>>>>>>>               IPv6
>>>>>>>>>>>                >     traffic and for that you need to have a way of getting
>>>>>>>> a
>>>>>>>>>>>               subnet
>>>>>>>>>>>                >     routed to the VR.
>>>>>>>>>>>                >
>>>>>>>>>>>                >     BGP is probably you best bet here. Although OSPFv3
>>>>>>>>>>>               technically
>>>>>>>>>>>                >     supports this it is very badly implemented in Frr for
>>>>>>>>>>>               example.
>>>>>>>>>>>                >
>>>>>>>>>>>                >     Now FRR is a very good router and one of the fancy
>>>>>>>>>>>               features it
>>>>>>>>>>>                >     supports is BGP Unnumered. This allows for auto
>>>>>>>>>>>               configuration of BGP
>>>>>>>>>>>                >     over a L2 network when both sides are sending Router
>>>>>>>>>>>               Advertisements.
>>>>>>>>>>>                >     This is very easy for flexible BGP configurations where
>>>>>>>>>>>               both sides
>>>>>>>>>>>                >     have dynamic IPs.
>>>>>>>>>>>                >
>>>>>>>>>>>                >     What you want to do is that you get a /56, /48 or
>>>>>>>>>>>               something which is
>>>>>>>>>>>                >      >/64 bits routed to the VR.
>>>>>>>>>>>                >
>>>>>>>>>>>                >     Now you can sub-segment this into separate /64 subnets.
>>>>>>>>>>>               You don't
>>>>>>>>>>>                >     want to go smaller then a /64 is that prevents you from
>>>>>>>>>>>               using SLAAC
>>>>>>>>>>>                >     for IPv6 address configuration. This is how it works
>>>>>>>> for
>>>>>>>>>>>               Shared
>>>>>>>>>>>                >     Networks now in Basic and Advanced Zones.
>>>>>>>>>>>                >
>>>>>>>>>>>                >     FRR can now also send out the Router Advertisements on
>>>>>>>>>>>               the downlinks
>>>>>>>>>>>                >     sending out:
>>>>>>>>>>>                >
>>>>>>>>>>>                >     - DNS servers
>>>>>>>>>>>                >     - DNS domain
>>>>>>>>>>>                >     - Prefix (/64) to be used
>>>>>>>>>>>                >
>>>>>>>>>>>                >     There is no need for DHCPv6. You can calculate the IPv6
>>>>>>>>>>>               address the
>>>>>>>>>>>                >     VM will obtain by using the MAC and the prefix.
>>>>>>>>>>>                >
>>>>>>>>>>>                >     So in short:
>>>>>>>>>>>                >
>>>>>>>>>>>                >     - Using BGP you routed a /48 to the VR
>>>>>>>>>>>                >     - Now you split this into /64 subnets towards the
>>>>>>>>>>>               isolated networks
>>>>>>>>>>>                >
>>>>>>>>>>>                >     Wido
>>>>>>>>>>>                >
>>>>>>>>>>>                >      > Alex Mattioli
>>>>>>>>>>>                >      >
>>>>>>>>>>>                >      >
>>>>>>>>>>>                >      >
>>>>>>>>>>>                >      >
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                >
>>>>>>>>>>>                > --
>>>>>>>>>>>                > Regards,
>>>>>>>>>>>                > Hean Seng
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>           --
>>>>>>>>>>>           Regards,
>>>>>>>>>>>           Hean Seng
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Hean Seng
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>
> 

Re: IPV6 in Isolated/VPC networks

Posted by Wido den Hollander <wi...@widodh.nl>.

Op 17-08-2021 om 11:20 schreef Wei ZHOU:
> Hi Wido,
> 
> (cc to Rohit and Alex)
> 
> It is a good suggestion to use FRR for ipv6. The configuration is quite
> simple and the VMs can get SLAAC, routes, etc.
> 
> Privacy extension looks not the same as what you mentioned. see
> https://datatracker.ietf.org/doc/html/rfc4941
> 
> You are right. To use static routing, the admins need to configure the
> routes in the upstream router, and add some ipv6 ranges (eg /56 for VPCs
> and /64 for isolated networks) and their next-hop  (which will be
> configured in VRs) in CloudStack. CloudStack will pick up an IPv6 range and
> assign it to an isolated network or vpc. @Rohit, correct me if I'm wrong.
> 
> I have a question, it looks stateless dhcpv6 (SLAAC from router/VR,
> router/dns etc via RA messages) will be the only option for now (related to

RA/SLAAC is NOT DHCPv6. Please don't confuse that. DHCPv6 is not 
involved at all when using SLAAC.

> your pr https://github.com/apache/cloudstack/pull/3077) . Would it be good
> to provide stateful dhcpv6 (which can be implemented by dnsmasq) as an
> option in cloudstack ? The advantages are
> (1) support other ipv6 cidr sizes than /64.

Yes, possibly, although hardly used. A /64 for a network is the default 
in most cases.

> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6 addresses can
> be changed

Yes, correct. Although you can now also just add a secondary IPv6 
address to the Instance.

> (4) an Ipv6 addresses can be re-used by multiple vms.

Yes, that is a benefit. Although this can be achieved with secondaire 
IPs as well.

> The problem is, stateful dhcpv6 does not support routers,nameservers, etc.
> we need to figure it out (probably use radvd/frr and dnsmasq both).
> 

You will *always* need RAs, but you will:

- Set the Other Managed flag in the RA
- Do not advertise a prefix

This way the client will learn the IPv6 default gateways, but obtain 
it's address through DHCPv6.

Wido

> -Wei
> 
> 
> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
> 
>> Hi,
>>
>> See my inline responses:
>>
>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>> Hi all,
>>>
>>> Thanks for your feedback and ideas, I've gone ahead with discussing them
>> with Alex and came up with a PoC/design which can be implemented in the
>> following phases:
>>>
>>>     *   Phase1: implement ipv6 support in isolated networks and VPC with
>> static routing
>>>     *   Phase2: discuss and implement support for dynamic routing (TBD)
>>>
>>> For Phase1 here's the high-level proposal:
>>>
>>>     *   IPv6 address management:
>>>        *   At the zone level root-admin specifies a /64 public range that
>> will be used for VRs, then they can add a /48, or /56 IPv6 range for guest
>> networks (to be used by isolated networks and VPC tiers)
>>>        *   On creation of any IPv6 enabled isolated network or VPC tier,
>> from the /48 or /56 block a /64 network is allocated/used
>>>        *   We assume SLAAC and autoconfiguration, no DHCPv6 in the zone
>> (discuss: is privacy a concern, can privacy extensions rfc4941 of slaac be
>> explored?)
>>
>> Privacy Extensions are only a concern for client devices which roam
>> between different IPv6 networks.
>>
>> If you IPv6 address of a client keeps the same suffix (MAC based) and
>> switches network then only the prefix (/64) will change.
>>
>> This way a network like Google, Facebook, etc could track your device
>> moving from network to network if they only look at the last 64-bits of
>> the IPv6 address.
>>
>> For servers this is not a problem as you already know in which network
>> they are.
>>
>>>     *   Network offerings: root-admin can create new network offerings
>> (with VPC too) that specifies a network stack option:
>>>        *   ipv4 only (default, for backward compatibility all
>> networks/offerings post-upgrade migrate to this option)
>>>        *   ipv4-and-ipv6
>>>        *   ipv6-only (this can be phase 1.b)
>>>        *   A new routing option: static (phase1), dynamic (phase2, with
>> multiple sub-options such as ospf/bgp etc...)
>>
>> This means that the network admin will need to statically route the IPv6
>> subnet to the VR's outside IPv6 address, for example, on a JunOS router:
>>
>> set routing-options rib inet6.0 static route 2001:db8:500::/48 next-hop
>> 2001:db8:100::50
>>
>> I'm assuming that 2001:db8:100::50 is the address of the VR on the
>> outside (/64) network. In reality this will probably be a longer
>> address, but this is for just the example.
>>
>>>     *   VR changes:
>>>        *   VR gets its guest and public nics set to inet6 auto
>>>        *   For each /64 allocated to guest network and VPC tiers, radvd
>> is configured to do RA
>>
>> radvd is fine, but looking at phase 2 with dynamic routing you might
>> already want to look into FRRouting. FRR can also advertise RAs while
>> not doing any routing.
>>
>> interface ens4
>>     no ipv6 nd suppress-ra
>>     ipv6 nd prefix 2001:db8:500::/64
>>     ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>
>> See: http://docs.frrouting.org/en/latest/ipv6.html
>>
>>>        *   Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6
>> firewall rules (ACLs, ingress, egress) are implemented; ACLs between VPC
>> tiers are managed/implemented by ipv6 firewall on VR
>>
>> Please take a look at the existing security_group.py script which
>> implements RFC4890
>>
>> https://datatracker.ietf.org/doc/html/rfc4890
>>
>> ICMPv6 is a vital part of IPv6 and certain packets should always be
>> allowed.
>>
>>>        *   It is assumed that static routes are created on the core/main
>> router by the admin or automated using some scripts/tools; for this
>> CloudStack will announce events with details of /64 networks and VR's
>> public IPv6 address that can be consumed by a rabbitmq/message bus client
>> (for example), or a custom cron job or script as part of orchestration.
>> (this wouldn't be necessary for dynamic routing bgp with phase2)\\
>>
>> You would only need to announce the /48 or /56 allocated to the VR,
>> that's all. You don't need to inform the upstream router about the /64
>> subnets created within that larger subnet.
>>
>>>     *   Guest Networking: With SLAAC, it's easy for CloudStack to
>> calculate allocate and use a /64 and determine the IPv6 address of VR nics
>> and guest VM nics
>>>        *   A user create an isolated network/VPC with an offering that is
>> ipv6 enabled
>>>        *   A user can manage firewall for the IPv6 address/guest nics;
>> there'll be no port forward and LB feature though for IPv6
>>>        *   A users can run workloads in the guest VMs that listen on
>> publically routable ipv6 addresses
>>>        *   Usage/billing etc continue to work, no change needed
>>>
>>> Network layout:
>>>
>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] ->
>> [guest VMs/nics]
>>> *core/ISP router needs static routes to be added (manually or
>> automated), assumes a /48 or /56 configured for the zone
>>>
>>> Thoughts, feedback?
>>
>> Looks doable!
>>
>> Side-note: It would be very cool if you could use parts of this
>> implementation to also route /48, /56, or /60 subnets to individual VMs
>> in Shared networks.
>>
>> Why? This allows for running Docker containers with native IPv6 inside
>> the VM or running a (Open)VPN server inside a VM which then assigns
>> public IPv6 addresses to clients connected.
>>
>> Instead of routing the subnet to a VR we route the subnet to a single
>> instance in a shared network.
>>
>> If we could then also move these subnets between Instances easily one
>> can quickly migrate to a different instance while keeping the same IPv6
>> subnet.
>>
>> Wido
>>
>>>
>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>
>>>     *   Created an isolated network and deployed a VM in my home lab
>>> The VR running on KVM has following nics
>>> eth0 - guest network
>>> eth1 - link local
>>> eth2 - public network
>>>
>>>     *   I setup a custom openwrt router on a RPi4 to serve as a toy-core
>> router where I create a wan6 IPv6 tunnel using tunnel broker and I got a
>> /48 allocated. My configuration looks like:
>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>
>>> I create a LAN ipv6 (public network for CloudStack VR): at subnet/prefix
>> 0:
>>> LAN IPv6 address: 2001:470:ed36:0::1/64
>>> Address mode: SLAAC+stateless DHCP (no dhcpv6)
>>>     *
>>>     *
>>> In the isolated VR, I enabled ipv6 as:
>>> net.ipv6.conf.all.disable_ipv6 = 0
>>> net.ipv6.conf.all.forwarding = 1
>>> net.ipv6.conf.all.accept_ra = 1
>>> net.ipv6.conf.all.accept_redirects = 1
>>> net.ipv6.conf.all.autoconf = 1
>>>
>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf
>>> And configured the nics:
>>> echo iface eth0 inet6 auto >> /etc/network/interfaces
>>> echo iface eth2 inet6 auto >> /etc/network/interfaces
>>> /etc/init.d/networking restart
>>> Next, restart ACS isolated network without cleanup to have it
>> reconfigure IPv4 nics, firewall, NAT etc
>>>
>>>     *
>>> Next, I created a /64 network for the isolated guest network on eth0 of
>> VR using radvd:
>>>
>>> # cat /etc/radvd.conf
>>> interface eth0
>>> {
>>>       AdvSendAdvert on;
>>>       MinRtrAdvInterval 5;
>>>       MaxRtrAdvInterval 15;
>>>       prefix 2001:470:ed36:1::/64
>>>       {
>>>           AdvOnLink on;
>>>           AdvAutonomous on;
>>>       };
>>> };
>>> systemctl restart radvd
>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this
>> ...:1::/64 network
>>>     *   Finally I added a static route in toy core-router for the new /64
>> IPv6 range in the isolated network
>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev
>> <local lan nic>
>>>     *
>>> ... and I enabled firewall rules to allow any traffic to pass for the
>> new /64 network
>>>
>>> And voila all done! I create a domain AAAA record that points to my
>> guest VM IPv6 address a test webserver on
>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>>>
>>> (Note: I'll get rid of the tunnel and request a new /48 block after a
>> few days, sharing this solely for testing purposes)
>>>
>>>
>>> Regards.
>>>
>>>
>>
> 

Re: IPV6 in Isolated/VPC networks

Posted by Abhishek Kumar <Ab...@shapeblue.com>.
Hi Wido, Wei,

I've created a PR to have static route in the event description, https://github.com/apache/cloudstack/pull/6389
Can you please check.

Regards,
Abhishek
________________________________
From: Wido den Hollander <wi...@widodh.nl>
Sent: 13 May 2022 15:30
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Wei ZHOU <us...@gmail.com>
Subject: Re: IPV6 in Isolated/VPC networks



Op 13-05-2022 om 11:39 schreef Wei ZHOU:
> Hi Wido,
>
> We do not allocate an ipv6 subnet to a VPC. Instead, we allocate an ipv6
> subnet for each vpc tier.
> These subnets have the same gateway (ipv6 addr in VR).
>
> for example
>
> ipv6 route fd23:313a:2f53:3111::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
> ipv6 route fd23:313a:2f53:3112::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
> ipv6 route fd23:313a:2f53:3113::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
>

Thanks Wei and Alex, that explains! So you can have multiple /64 subnets
being routed to the same VR.

It's up to the organization to only allocate /64s coming from a specific
/56 or /48 to make things logical, but underneath it's all /64s which we
need to route.

Wido

> -Wei
>

 

> On Fri, 13 May 2022 at 11:30, Wido den Hollander <wi...@widodh.nl> wrote:
>
>>
>>
>> Op 12-05-2022 om 16:10 schreef Alex Mattioli:
>>>> ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
>>>
>>> That's correct
>>
>> Ok. So in that case the subnet would be very welcome to be present in
>> the message on the bus.
>>
>>>
>>>> Or a larger subnet:
>>>> ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4
>>>
>>> Not really, the subnets for isolated/VPC networks are always /64.  Which
>> means also no real need to include subnets as well.
>>>
>>
>> But there can be multiple networks behind the VPC router or not? If
>> there are multiple networks you need >/64 as you can then allocate /64s
>> from that larger subnet.
>>
>> Wido
>>
>>> Cheers
>>> Alex
>>>
>>>
>>>
>>>
>>
>

Re: IPV6 in Isolated/VPC networks

Posted by Wido den Hollander <wi...@widodh.nl>.

Op 13-05-2022 om 11:39 schreef Wei ZHOU:
> Hi Wido,
> 
> We do not allocate an ipv6 subnet to a VPC. Instead, we allocate an ipv6
> subnet for each vpc tier.
> These subnets have the same gateway (ipv6 addr in VR).
> 
> for example
> 
> ipv6 route fd23:313a:2f53:3111::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
> ipv6 route fd23:313a:2f53:3112::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
> ipv6 route fd23:313a:2f53:3113::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
> 

Thanks Wei and Alex, that explains! So you can have multiple /64 subnets 
being routed to the same VR.

It's up to the organization to only allocate /64s coming from a specific 
/56 or /48 to make things logical, but underneath it's all /64s which we 
need to route.

Wido

> -Wei
> 
> On Fri, 13 May 2022 at 11:30, Wido den Hollander <wi...@widodh.nl> wrote:
> 
>>
>>
>> Op 12-05-2022 om 16:10 schreef Alex Mattioli:
>>>> ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
>>>
>>> That's correct
>>
>> Ok. So in that case the subnet would be very welcome to be present in
>> the message on the bus.
>>
>>>
>>>> Or a larger subnet:
>>>> ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4
>>>
>>> Not really, the subnets for isolated/VPC networks are always /64.  Which
>> means also no real need to include subnets as well.
>>>
>>
>> But there can be multiple networks behind the VPC router or not? If
>> there are multiple networks you need >/64 as you can then allocate /64s
>> from that larger subnet.
>>
>> Wido
>>
>>> Cheers
>>> Alex
>>>
>>>
>>>
>>>
>>
> 

RE: IPV6 in Isolated/VPC networks

Posted by Alex Mattioli <Al...@shapeblue.com>.
> But there can be multiple networks behind the VPC router or not? If there are multiple networks you need 
>/64 as you can then allocate /64s from that larger subnet.

Yes, there will be multiple /64s in a VPC, but they are very likely to be non-contiguous, so in general we should treat as completely separate networks.  

One could in theory dedicate a range to a domain/account and statically route to that, but I'm not sure if that will happen often.

Cheers
Alex

 


-----Original Message-----
From: Wido den Hollander <wi...@widodh.nl> 
Sent: 13 May 2022 11:30
To: dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks



Op 12-05-2022 om 16:10 schreef Alex Mattioli:
>> ipv6 route fd23:313a:2f53:3cbf::/64 
>> fd23:313a:2f53:3000:1c00:baff:fe00:4
> 
> That's correct

Ok. So in that case the subnet would be very welcome to be present in the message on the bus.

> 
>> Or a larger subnet:
>> ipv6 route fd23:313a:2f53:3c00::/56 
>> fd23:313a:2f53:3000:1c00:baff:fe00:4
> 
> Not really, the subnets for isolated/VPC networks are always /64.  Which means also no real need to include subnets as well.
> 

But there can be multiple networks behind the VPC router or not? If there are multiple networks you need >/64 as you can then allocate /64s from that larger subnet.

Wido

> Cheers
> Alex
> 
> 
>   
> 
> 
> -----Original Message-----
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: 12 May 2022 16:04
> To: Abhishek Kumar <Ab...@shapeblue.com>; 
> dev@cloudstack.apache.org
> Subject: Re: IPV6 in Isolated/VPC networks
> 
> 
> 
> On 5/12/22 09:55, Abhishek Kumar wrote:
>> Hi Wido,
>>
>> I do not understand what you mean by WAB address but
> 
> WAB was a type. I meant WAN.
> 
>> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the public IP of the network
>> (IPv6 of the public NIC of the network VR) in the sample.
>> Yes, route for fd23:313a:2f53:3cbf::/64 need to be added to this IP.
>> fd23:313a:2f53:3cbf::/64 is guest IPv6 CIDR of the network here.
>>
> 
> So that means that I would need to run this command on my upstream router:
> 
> ipv6 route fd23:313a:2f53:3cbf::/64 
> fd23:313a:2f53:3000:1c00:baff:fe00:4
> 
> Or a larger subnet:
> 
> ipv6 route fd23:313a:2f53:3c00::/56 
> fd23:313a:2f53:3000:1c00:baff:fe00:4
> 
>> Currently, the message on event bus does not include subnet. Should 
>> that be included?
> 
> Yes, because then you can pickup those messages and inject the route via ExaBGP into a routing table right away.
> 
>> In case of VPCs, there could be multiple tiers which will need 
>> multiple routes to be added. Will that be an issue if we include 
>> current network/tier subnet in the event message?
> 
> No, as long as it points to the same VR you simply have multiple subnets being routed to the same VR.
> 
> I do wonder what happens if you destroy the VR and create a new one. The WAN address then changes (due to SLAAC) and thus the routes need to be re-programmed.
> 
> Wido
> 
>>
>> Regards,
>> Abhishek
>>
>>
>>    
>>
>>    
>>
>> ---------------------------------------------------------------------
>> -
>> --
>> *From:* Wido den Hollander <wi...@widodh.nl>
>> *Sent:* 10 May 2022 19:01
>> *To:* dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek 
>> Kumar <Ab...@shapeblue.com>
>> *Subject:* Re: IPV6 in Isolated/VPC networks
>>   
>> Hi,
>>
>> Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
>>> Yes. When a public IPv6 is assigned or released, CloudStack will publish event with type NET.IP6ASSIGN, NET.IP6RELEASE.
>>> These event notifications can be tracked. And with improvements in events framework, these event messages will have network uuid as entityuuid and Network as entitytype. Using this network can be queried using to list IPv6 routes that need to be added.
>>>
>>> Sample event message,
>>>
>>> {"eventDateTime":"2022-05-10 09:32:12
>>> +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":
>>> "14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned 
>>> public
>>> IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID:
>>> 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":
>>> "bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec
>>> - af19-1e00320001f3","entity":"Network","status":"Completed"}
>>>
>>>
>>> ?Sample API call,
>>>> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98
>>>> filter=id,name,ip6routes
>>> {
>>>      "count": 1,
>>>      "network": [
>>>        {
>>>          "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
>>>          "ip6routes": [
>>>            {
>>>              "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
>>>              "subnet": "fd23:313a:2f53:3cbf::/64"
>>>            }
>>
>> Looking at this JSON, does this mean that
>> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
>>
>> And that I would need to (statically) route fd23:313a:2f53:3cbf::/64 
>> to that IP?
>>
>> The event message does not include the subnet, that makes it a bit 
>> more difficult as you would then also need to do a API-call to gather 
>> that information.
>>
>> Wido
>>
>> P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an 
>> AAAA-record for IPv6!
>>
>>>          ],
>>>          "name": "routing_test"
>>>        }
>>>      ]
>>> }
>>>
>>>
>>>
>>>
>>> ________________________________
>>> From: Wido den Hollander <wi...@widodh.nl>
>>> Sent: 10 May 2022 13:59
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek 
>>> Kumar <Ab...@shapeblue.com>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
>>>> Hi all,
>>>>
>>>> IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
>>>> Documentation for it is available at 
>>>> http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.ht
>>>> m
>>>> l#isolated-network-and-vpc-tier
>> <http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.htm
>> l #isolated-network-and-vpc-tier> (will be available in the official 
>> docs once 4.17.0 version of docs is built).
>>>>
>>>
>>> Great work!
>>>
>>> I see only static routing is supported. But do we publish something 
>>> on the message bus once a new VR/VPC is created?
>>>
>>> This way you could pick up these messages and have the network 
>>> create a
>>> (static) route based on those.
>>>
>>> ExaBGP for example could be used to inject such routes.
>>>
>>> Wido
>>>
>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support
>>>> +
>>>> in+Isolated+Network+and+VPC+with+Static+Routing
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>> i
>> n+Isolated+Network+and+VPC+with+Static+Routing>
>>>>
>>>> Regards,
>>>> Abhishek
>>>>
>>>> ________________________________
>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>> Sent: 13 September 2021 14:30
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Thanks Alex, Wei. I've updated the docs here:
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support
>>>> +
>>>> in+Isolated+Network+and+VPC+with+Static+Routing
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>> i
>> n+Isolated+Network+and+VPC+with+Static+Routing>
>>>>
>>>> I'll leave the thread open for futher discussion/ideas/feedback. I 
>>>> think we've completed the phase1 design doc including all feedback 
>>>> comments for adding IPv6 support in CloudStack and some initial 
>>>> poc/work can be started. My colleagues and I will keep everyone 
>>>> posted on this thread and/or on a Github PR as and when we're able 
>>>> to
>> start our work on the same (after 4.16, potentially towards 4.17).
>>>>
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Wei ZHOU <us...@gmail.com>
>>>> Sent: Friday, September 10, 2021 20:22
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Agree with Alex.
>>>> We only need to know how many /64 are allocated. We do not care how 
>>>> many
>>>> ipv6 addresses are used by VMs.
>>>>
>>>> -Wei
>>>>
>>>> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli 
>>>> <Al...@shapeblue.com>
>>>> wrote:
>>>>
>>>>> Hi Rohit,
>>>>>
>>>>> I'd go for option 2, don't see a point tracking anything smaller 
>>>>> than a
>>>>> /64 tbh.
>>>>>
>>>>> Cheers
>>>>> Alex
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>>> Sent: 09 September 2021 12:44
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>>>>> agreements:
>>>>>
>>>>>       *   Allocate /64 for both isolated network and VPC tiers, no 
>>>>> large  allocation of prefixes to VPC (cons: more static routing 
>>>>> rules for upstream
>>>>> router/admins)
>>>>>       *   All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address 
>>>>> if zone has a  dedicated /64 prefix/block for systemvms
>>>>>
>>>>> The only outstanding question now is:
>>>>>
>>>>>       *   How do we manage IPv6 usage? Can anyone advise how we do
>>>>> IPv6 usage  for shared network (design, implementation and
>>>>> use-cases?)
>>>>> Option1: We don't do it, all user VMs nics have ipv4 address which 
>>>>> whose  usage we don't track. For public VR/nics/networks, we can 
>>>>> simply add the
>>>>> IPv6 details for a related IPv4 address.
>>>>> Option2: Implement a separate, first-class IPv6 address or /64 
>>>>> prefix  tracking/management and usage for all VMs and systemvms 
>>>>> nic (this means  account/domain level limits and new 
>>>>> billing/records)
>>>>> Option3: other thoughts?
>>>>>
>>>>>
>>>>> Regards.
>>>>>
>>>>> ________________________________
>>>>> From: Alex Mattioli <Al...@shapeblue.com>
>>>>> Sent: Wednesday, September 8, 2021 23:24
>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>> Subject: RE: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hi Rohit, Kristaps,
>>>>>
>>>>> I'd say option 1 as well,  it does create a bit more overhead with 
>>>>> static routes but if that's automated for a VPC it can also be 
>>>>> easily automated for several tiers of a VPC.  We also don't 
>>>>> constrain the amount of tiers in a  VPC.
>>>>> It has the added advantage to be closer to the desired behaviour 
>>>>> with dynamic routing in the future, where a VPC VR can announce 
>>>>> several subnets upstream.
>>>>>
>>>>> Cheers
>>>>> Alex
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>>> Sent: 08 September 2021 19:04
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hi Kristaps,
>>>>>
>>>>> Thanks for sharing, I suppose that means individual tiers should 
>>>>> be allocated /64 instead of larger ipv6 blocks to the whole VPC 
>>>>> which could cause wastage.
>>>>>
>>>>> Any objection from anybody?
>>>>>
>>>>> Regards.
>>>>> ________________________________
>>>>> From: Kristaps Cudars <kr...@gmail.com>
>>>>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hello,
>>>>>
>>>>> I asked networking team to comment on "How should the IPv6 
>>>>> block/allocation work in VPCs?"
>>>>> Option1: They haven't seen lately devices with limits on how many 
>>>>> static routes can be created.
>>>>> Option2: With /60 and /62 assignments and big quantity of routers
>>>>> IPv6 assignment from RIPE NNC can be drained fast.
>>>>>
>>>>> /48 contains 64000 /64
>>>>> /60 contains 16 /64
>>>>> 64000 / 16 = 4000 routers
>>>>>
>>>>>
>>>>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
>>>>>> All,
>>>>>>
>>>>>> After another iteration with Alex, I've updated the design doc.
>>>>>> Kindly
>>>>> review:
>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppo
>>>>>> r
>>>>>> t+in
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>> i
>> n>
>>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>>
>>>>>>
>>>>>> ... and advise on some outstanding questions:
>>>>>>
>>>>>>       *   How should the IPv6 block/allocation work in VPCs?
>>>>>> Option1: Should this be simply /64 allocation on any new tier, 
>>>>>> the cons of this option is one static route/rule per VPC tier. 
>>>>>> (many upstream routers may have limit on no. of static routes?)
>>>>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) 
>>>>>> or
>>>>>> /62
>>>>> (4 tiers) for the VPC, this can be filtered based on the 
>>>>> vpc.max.networks global setting (3 is default). The pros of this 
>>>>> option are less no. of static route/rule and easy programming, but 
>>>>> potential wastage of multiple
>>>>> /64 prefix blocks for unused/uncreated tiers.
>>>>>>       *   How do we manage IPv6 usage? Can anyone advise how we 
>>>>>> do
>>>>>> IPv6
>>>>> usage for shared network (design, implementation and use-cases?)
>>>>>> Option1: We don't do it, all user VMs nics have ipv4 address 
>>>>>> which whose
>>>>> usage we don't track. For public VR/nics/networks, we can simply 
>>>>> add the
>>>>> IPv6 details for a related IPv4 address.
>>>>>> Option2: Implement a separate, first-class IPv6 address or /64 
>>>>>> prefix
>>>>> tracking/management and usage for all VMs and systemvms nic (this 
>>>>> means account/domain level limits and new billing/records)
>>>>>>       *   Enable IPv6 on systemvms (specifically SSVM and CPVM) 
>>>>>> by default
>>>>> if zone has a IPv6 address block allocated/assigned for use for 
>>>>> systemvms (this was mainly thought for VRs, but why no ssvm and 
>>>>> cpvms too - any cons of this?)
>>>>>>       *
>>>>>>
>>>>>> Regards.
>>>>>>
>>>>>> ________________________________
>>>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>>>> Sent: Thursday, August 19, 2021 15:45
>>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I've taken feedback from this thread and wrote this design doc:
>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppo
>>>>>> r
>>>>>> t+in
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>> i
>> n>
>>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>>
>>>>>> Kindly review and advise if I missed anything or anything that 
>>>>>> needs to
>>>>> be changed/updated. You may comment on the wiki directly too.
>>>>>>
>>>>>> Kindly suggest your views on the following (also in the design 
>>>>>> doc
>>>>> above):
>>>>>>
>>>>>> Outstanding Questions:
>>>>>>
>>>>>>       *   Should admin or user be able to specify how VPC super 
>>>>>> CIDRs are
>>>>> created/needed; for example a user can ask for VPC with /60 super 
>>>>> CIDR? Or should CloudStack automatically find/allocate a /64 for a 
>>>>> new VPC tier from the root-admin configured /64-/48 block?
>>>>>>       *   Should we explore FRR and iBGP or other strategies to 
>>>>>> do dynamic
>>>>> routing which may not require advance/complex configuration in the 
>>>>> VR or for the users/admin?
>>>>>>       *   With SLAAC and no dhcpv6, is there a way to support 
>>>>>> secondary IPv6
>>>>> addresses (or floating IPv6 addresses for VR/public traffic) for 
>>>>> guest VM's nics?
>>>>>>       *   Any thoughts on UI/UX for firewall/routing management?
>>>>>>       *   Any other feature/support for isolated network or VPC 
>>>>>> feature that
>>>>> must be explored or supported such as PF, VPN, LB, vpc static 
>>>>> routes, vpc gateway etc.
>>>>>>       *   For usage - should we have any consideration, or should 
>>>>>> we assume
>>>>> that IPv4 and IPv6 address will go together for every nic; so IPv6 
>>>>> usage for a nic is in tandem with Ipv4 address for a nic, i.e. no 
>>>>> explicit/new biling/usage needed?
>>>>>>       *   For smoketests, local dev-test should we explore ULA?
>>>>>> Unique Local
>>>>> Address - in the range fc00::/7. Typically only within the 'local'
>>>>> half fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
>>>>> This prefix can be randomly generated at first install by 
>>>>> CloudStack in a zone using zoneid etc?
>>>>>>       *   Should we expand support for VR diagnostic feature to 
>>>>>> include
>>>>> support for ipv6, traceroute6?
>>>>>>       *   Discuss - expand support/ability to allocate a /60, or 
>>>>>> /56 etc
>>>>> prefix to an individual VM in shared network (Wido's suggestion)
>>>>>>
>>>>>>
>>>>>> Regards.
>>>>>>
>>>>>> ________________________________
>>>>>> From: Wei ZHOU <us...@gmail.com>
>>>>>> Sent: Tuesday, August 17, 2021 21:16
>>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>>
>>>>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>>>>
>>>>>> I am fine with not having stateful dhcpv6 and privacy 
>>>>>> extension/temporary address in phase 1. If community decides not 
>>>>>> to do eventually , it is also ok to me.
>>>>>>
>>>>>> We could explore how to better use secondary ipv6 addresses as 
>>>>>> Wido advised. It would be great if anyone share some user experience.
>>>>>>
>>>>>> -Wei
>>>>>>
>>>>>>
>>>>>> On Tuesday, 17 August 2021, Kristaps Cudars 
>>>>>> <kr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Wei,
>>>>>>>
>>>>>>> Published this month's RFC 9099 and explains in different 
>>>>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>>>>> https://www.rfc-editor.org/rfc/rfc9099.html
>> <https://www.rfc-editor.org/rfc/rfc9099.html>
>>>>>>>
>>>>>>>
>>>>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
>>>>>>>> Hi Wido,
>>>>>>>>
>>>>>>>> (cc to Rohit and Alex)
>>>>>>>>
>>>>>>>> It is a good suggestion to use FRR for ipv6. The configuration 
>>>>>>>> is quite simple and the VMs can get SLAAC, routes, etc.
>>>>>>>>
>>>>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>>>>> https://datatracker.ietf.org/doc/html/rfc4941
>> <https://datatracker.ietf.org/doc/html/rfc4941>
>>>>>>>>
>>>>>>>> You are right. To use static routing, the admins need to 
>>>>>>>> configure the routes in the upstream router, and add some ipv6 
>>>>>>>> ranges (eg
>>>>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop 
>>>>>>>> (which will be configured in VRs) in CloudStack. CloudStack 
>>>>>>>> will pick up an IPv6 range
>>>>>>> and
>>>>>>>> assign it to an isolated network or vpc. @Rohit, correct me if 
>>>>>>>> I'm
>>>>> wrong.
>>>>>>>>
>>>>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from 
>>>>>>>> router/VR, router/dns etc via RA messages) will be the only 
>>>>>>>> option for now (related
>>>>>>> to
>>>>>>>> your pr https://github.com/apache/cloudstack/pull/3077)
>> <https://github.com/apache/cloudstack/pull/3077)> . Would it
>>>>>>>> be
>>>>>>> good
>>>>>>>> to provide stateful dhcpv6 (which can be implemented by 
>>>>>>>> dnsmasq) as an option in cloudstack ? The advantages are
>>>>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6 
>>>>>>>> addresses can be changed
>>>>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>>>>> The problem is, stateful dhcpv6 does not support 
>>>>>>>> routers,nameservers,
>>>>>>> etc.
>>>>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>>>>
>>>>>>>> -Wei
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>     
>>>
>>>> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> See my inline responses:
>>>>>>>>>
>>>>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Thanks for your feedback and ideas, I've gone ahead with 
>>>>>>>>>> discussing
>>>>>>> them
>>>>>>>>> with Alex and came up with a PoC/design which can be 
>>>>>>>>> implemented in the following phases:
>>>>>>>>>>
>>>>>>>>>>        *   Phase1: implement ipv6 support in isolated 
>>>>>>>>>> networks and
>>>>> VPC
>>>>>>> with
>>>>>>>>> static routing
>>>>>>>>>>        *   Phase2: discuss and implement support for dynamic 
>>>>>>>>>> routing
>>>>>>> (TBD)
>>>>>>>>>>
>>>>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>>>>
>>>>>>>>>>        *   IPv6 address management:
>>>>>>>>>>           *   At the zone level root-admin specifies a /64 
>>>>>>>>>> public
>>>>> range
>>>>>>> that
>>>>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6 
>>>>>>>>> range for
>>>>>>> guest
>>>>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>>>>>           *   On creation of any IPv6 enabled isolated 
>>>>>>>>>> network or VPC
>>>>>>> tier,
>>>>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>>>>>           *   We assume SLAAC and autoconfiguration, no 
>>>>>>>>>> DHCPv6 in the
>>>>>>> zone
>>>>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941 
>>>>>>>>> of
>>>>>>> slaac be
>>>>>>>>> explored?)
>>>>>>>>>
>>>>>>>>> Privacy Extensions are only a concern for client devices which 
>>>>>>>>> roam between different IPv6 networks.
>>>>>>>>>
>>>>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>>>>
>>>>>>>>> This way a network like Google, Facebook, etc could track your 
>>>>>>>>> device moving from network to network if they only look at the 
>>>>>>>>> last 64-bits of the IPv6 address.
>>>>>>>>>
>>>>>>>>> For servers this is not a problem as you already know in which 
>>>>>>>>> network they are.
>>>>>>>>>
>>>>>>>>>>        *   Network offerings: root-admin can create new 
>>>>>>>>>> network
>>>>> offerings
>>>>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>>>>>           *   ipv4 only (default, for backward compatibility 
>>>>>>>>>> all
>>>>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>>>>>           *   ipv4-and-ipv6
>>>>>>>>>>           *   ipv6-only (this can be phase 1.b)
>>>>>>>>>>           *   A new routing option: static (phase1), dynamic 
>>>>>>>>>> (phase2,
>>>>>>> with
>>>>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>>>>
>>>>>>>>> This means that the network admin will need to statically 
>>>>>>>>> route the
>>>>>>> IPv6
>>>>>>>>> subnet to the VR's outside IPv6 address, for example, on a 
>>>>>>>>> JunOS
>>>>>>> router:
>>>>>>>>>
>>>>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48 
>>>>>>>>> next-hop
>>>>>>>>> 2001:db8:100::50
>>>>>>>>>
>>>>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on 
>>>>>>>>> the outside (/64) network. In reality this will probably be a 
>>>>>>>>> longer address, but this is for just the example.
>>>>>>>>>
>>>>>>>>>>        *   VR changes:
>>>>>>>>>>           *   VR gets its guest and public nics set to inet6 
>>>>>>>>>> auto
>>>>>>>>>>           *   For each /64 allocated to guest network and VPC 
>>>>>>>>>> tiers,
>>>>>>> radvd
>>>>>>>>> is configured to do RA
>>>>>>>>>
>>>>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you 
>>>>>>>>> might already want to look into FRRouting. FRR can also 
>>>>>>>>> advertise RAs while not doing any routing.
>>>>>>>>>
>>>>>>>>> interface ens4
>>>>>>>>>        no ipv6 nd suppress-ra
>>>>>>>>>        ipv6 nd prefix 2001:db8:500::/64
>>>>>>>>>        ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>>>>
>>>>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
>> <http://docs.frrouting.org/en/latest/ipv6.html>
>>>>>>>>>
>>>>>>>>>>           *   Firewall: a new ipv6 zone/chain is created for
>>>>>>>>>> ipv6
>>>>> where
>>>>>>> ipv6
>>>>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs 
>>>>>>>>> between
>>>>>>> VPC
>>>>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>>>>
>>>>>>>>> Please take a look at the existing security_group.py script 
>>>>>>>>> which implements RFC4890
>>>>>>>>>
>>>>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
>> <https://datatracker.ietf.org/doc/html/rfc4890>
>>>>>>>>>
>>>>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should 
>>>>>>>>> always be allowed.
>>>>>>>>>
>>>>>>>>>>           *   It is assumed that static routes are created on 
>>>>>>>>>> the
>>>>>>> core/main
>>>>>>>>> router by the admin or automated using some scripts/tools; for 
>>>>>>>>> this CloudStack will announce events with details of /64 
>>>>>>>>> networks and VR's public IPv6 address that can be consumed by 
>>>>>>>>> a rabbitmq/message bus
>>>>>>> client
>>>>>>>>> (for example), or a custom cron job or script as part of
>>>>> orchestration.
>>>>>>>>> (this wouldn't be necessary for dynamic routing bgp with 
>>>>>>>>> phase2)\\
>>>>>>>>>
>>>>>>>>> You would only need to announce the /48 or /56 allocated to 
>>>>>>>>> the VR, that's all. You don't need to inform the upstream 
>>>>>>>>> router about the /64 subnets created within that larger subnet.
>>>>>>>>>
>>>>>>>>>>        *   Guest Networking: With SLAAC, it's easy for 
>>>>>>>>>> CloudStack to
>>>>>>>>> calculate allocate and use a /64 and determine the IPv6 
>>>>>>>>> address of VR
>>>>>>> nics
>>>>>>>>> and guest VM nics
>>>>>>>>>>           *   A user create an isolated network/VPC with an 
>>>>>>>>>> offering
>>>>>>> that is
>>>>>>>>> ipv6 enabled
>>>>>>>>>>           *   A user can manage firewall for the IPv6 
>>>>>>>>>> address/guest
>>>>> nics;
>>>>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>>>>>           *   A users can run workloads in the guest VMs that 
>>>>>>>>>> listen
>>>>> on
>>>>>>>>> publically routable ipv6 addresses
>>>>>>>>>>           *   Usage/billing etc continue to work, no change 
>>>>>>>>>> needed
>>>>>>>>>>
>>>>>>>>>> Network layout:
>>>>>>>>>>
>>>>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a 
>>>>>>>>>> VLAN] ->
>>>>>>>>> [guest VMs/nics]
>>>>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>>>>
>>>>>>>>>> Thoughts, feedback?
>>>>>>>>>
>>>>>>>>> Looks doable!
>>>>>>>>>
>>>>>>>>> Side-note: It would be very cool if you could use parts of 
>>>>>>>>> this implementation to also route /48, /56, or /60 subnets to 
>>>>>>>>> individual VMs in Shared networks.
>>>>>>>>>
>>>>>>>>> Why? This allows for running Docker containers with native 
>>>>>>>>> IPv6 inside the VM or running a (Open)VPN server inside a VM 
>>>>>>>>> which then assigns public IPv6 addresses to clients connected.
>>>>>>>>>
>>>>>>>>> Instead of routing the subnet to a VR we route the subnet to a 
>>>>>>>>> single instance in a shared network.
>>>>>>>>>
>>>>>>>>> If we could then also move these subnets between Instances 
>>>>>>>>> easily one can quickly migrate to a different instance while 
>>>>>>>>> keeping the same IPv6 subnet.
>>>>>>>>>
>>>>>>>>> Wido
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>>>>
>>>>>>>>>>        *   Created an isolated network and deployed a VM in 
>>>>>>>>>> my home
>>>>> lab
>>>>>>>>>> The VR running on KVM has following nics
>>>>>>>>>> eth0 - guest network
>>>>>>>>>> eth1 - link local
>>>>>>>>>> eth2 - public network
>>>>>>>>>>
>>>>>>>>>>        *   I setup a custom openwrt router on a RPi4 to serve 
>>>>>>>>>> as a
>>>>>>> toy-core
>>>>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker 
>>>>>>>>> and I got
>>>>>>> a
>>>>>>>>> /48 allocated. My configuration looks like:
>>>>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>>>>
>>>>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>>>>> subnet/prefix
>>>>>>>>> 0:
>>>>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>>>>>        *
>>>>>>>>>>        *
>>>>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0 
>>>>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra 
>>>>>>>>>> =
>>>>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1 
>>>>>>>>>> net.ipv6.conf.all.autoconf = 1
>>>>>>>>>>
>>>>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And 
>>>>>>>>>> configured the nics:
>>>>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo 
>>>>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces 
>>>>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated 
>>>>>>>>>> network without cleanup to have it
>>>>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>>>>
>>>>>>>>>>        *
>>>>>>>>>> Next, I created a /64 network for the isolated guest network 
>>>>>>>>>> on eth0
>>>>>>> of
>>>>>>>>> VR using radvd:
>>>>>>>>>>
>>>>>>>>>> # cat /etc/radvd.conf
>>>>>>>>>> interface eth0
>>>>>>>>>> {
>>>>>>>>>>          AdvSendAdvert on;
>>>>>>>>>>          MinRtrAdvInterval 5;
>>>>>>>>>>          MaxRtrAdvInterval 15;
>>>>>>>>>>          prefix 2001:470:ed36:1::/64
>>>>>>>>>>          {
>>>>>>>>>>              AdvOnLink on;
>>>>>>>>>>              AdvAutonomous on;
>>>>>>>>>>          };
>>>>>>>>>> };
>>>>>>>>>> systemctl restart radvd
>>>>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in 
>>>>>>>>>> this
>>>>>>>>> ...:1::/64 network
>>>>>>>>>>        *   Finally I added a static route in toy core-router 
>>>>>>>>>> for the
>>>>> new
>>>>>>> /64
>>>>>>>>> IPv6 range in the isolated network
>>>>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>>>>> eth2> dev
>>>>>>>>> <local lan nic>
>>>>>>>>>>        *
>>>>>>>>>> ... and I enabled firewall rules to allow any traffic to pass 
>>>>>>>>>> for the
>>>>>>>>> new /64 network
>>>>>>>>>>
>>>>>>>>>> And voila all done! I create a domain AAAA record that points 
>>>>>>>>>> to my
>>>>>>>>> guest VM IPv6 address a test webserver on
>>>>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>> <http://ipv6-isolated-ntwk-demo.yadav.cloud/>
>>>>>>>>>>
>>>>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block 
>>>>>>>>>> after a
>>>>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>

Re: IPV6 in Isolated/VPC networks

Posted by Wei ZHOU <us...@gmail.com>.
Hi Wido,

We do not allocate an ipv6 subnet to a VPC. Instead, we allocate an ipv6
subnet for each vpc tier.
These subnets have the same gateway (ipv6 addr in VR).

for example

ipv6 route fd23:313a:2f53:3111::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
ipv6 route fd23:313a:2f53:3112::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
ipv6 route fd23:313a:2f53:3113::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4

-Wei

On Fri, 13 May 2022 at 11:30, Wido den Hollander <wi...@widodh.nl> wrote:

>
>
> Op 12-05-2022 om 16:10 schreef Alex Mattioli:
> >> ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
> >
> > That's correct
>
> Ok. So in that case the subnet would be very welcome to be present in
> the message on the bus.
>
> >
> >> Or a larger subnet:
> >> ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4
> >
> > Not really, the subnets for isolated/VPC networks are always /64.  Which
> means also no real need to include subnets as well.
> >
>
> But there can be multiple networks behind the VPC router or not? If
> there are multiple networks you need >/64 as you can then allocate /64s
> from that larger subnet.
>
> Wido
>
> > Cheers
> > Alex
> >
> >
> >
> >
>

Re: IPV6 in Isolated/VPC networks

Posted by Wido den Hollander <wi...@widodh.nl>.

Op 12-05-2022 om 16:10 schreef Alex Mattioli:
>> ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
> 
> That's correct

Ok. So in that case the subnet would be very welcome to be present in 
the message on the bus.

> 
>> Or a larger subnet:
>> ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4
> 
> Not really, the subnets for isolated/VPC networks are always /64.  Which means also no real need to include subnets as well.
> 

But there can be multiple networks behind the VPC router or not? If 
there are multiple networks you need >/64 as you can then allocate /64s 
from that larger subnet.

Wido

> Cheers
> Alex
> 
> 
>   
> 
> 
> -----Original Message-----
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: 12 May 2022 16:04
> To: Abhishek Kumar <Ab...@shapeblue.com>; dev@cloudstack.apache.org
> Subject: Re: IPV6 in Isolated/VPC networks
> 
> 
> 
> On 5/12/22 09:55, Abhishek Kumar wrote:
>> Hi Wido,
>>
>> I do not understand what you mean by WAB address but
> 
> WAB was a type. I meant WAN.
> 
>> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the public IP of the network
>> (IPv6 of the public NIC of the network VR) in the sample.
>> Yes, route for fd23:313a:2f53:3cbf::/64 need to be added to this IP.
>> fd23:313a:2f53:3cbf::/64 is guest IPv6 CIDR of the network here.
>>
> 
> So that means that I would need to run this command on my upstream router:
> 
> ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4
> 
> Or a larger subnet:
> 
> ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4
> 
>> Currently, the message on event bus does not include subnet. Should
>> that be included?
> 
> Yes, because then you can pickup those messages and inject the route via ExaBGP into a routing table right away.
> 
>> In case of VPCs, there could be multiple tiers which will need
>> multiple routes to be added. Will that be an issue if we include
>> current network/tier subnet in the event message?
> 
> No, as long as it points to the same VR you simply have multiple subnets being routed to the same VR.
> 
> I do wonder what happens if you destroy the VR and create a new one. The WAN address then changes (due to SLAAC) and thus the routes need to be re-programmed.
> 
> Wido
> 
>>
>> Regards,
>> Abhishek
>>
>>
>>    
>>
>>    
>>
>> ----------------------------------------------------------------------
>> --
>> *From:* Wido den Hollander <wi...@widodh.nl>
>> *Sent:* 10 May 2022 19:01
>> *To:* dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek
>> Kumar <Ab...@shapeblue.com>
>> *Subject:* Re: IPV6 in Isolated/VPC networks
>>   
>> Hi,
>>
>> Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
>>> Yes. When a public IPv6 is assigned or released, CloudStack will publish event with type NET.IP6ASSIGN, NET.IP6RELEASE.
>>> These event notifications can be tracked. And with improvements in events framework, these event messages will have network uuid as entityuuid and Network as entitytype. Using this network can be queried using to list IPv6 routes that need to be added.
>>>
>>> Sample event message,
>>>
>>> {"eventDateTime":"2022-05-10 09:32:12
>>> +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":
>>> "14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned public
>>> IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID:
>>> 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":
>>> "bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-
>>> af19-1e00320001f3","entity":"Network","status":"Completed"}
>>>
>>>
>>> ?Sample API call,
>>>> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98
>>>> filter=id,name,ip6routes
>>> {
>>>      "count": 1,
>>>      "network": [
>>>        {
>>>          "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
>>>          "ip6routes": [
>>>            {
>>>              "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
>>>              "subnet": "fd23:313a:2f53:3cbf::/64"
>>>            }
>>
>> Looking at this JSON, does this mean that
>> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
>>
>> And that I would need to (statically) route fd23:313a:2f53:3cbf::/64
>> to that IP?
>>
>> The event message does not include the subnet, that makes it a bit
>> more difficult as you would then also need to do a API-call to gather
>> that information.
>>
>> Wido
>>
>> P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an
>> AAAA-record for IPv6!
>>
>>>          ],
>>>          "name": "routing_test"
>>>        }
>>>      ]
>>> }
>>>
>>>
>>>
>>>
>>> ________________________________
>>> From: Wido den Hollander <wi...@widodh.nl>
>>> Sent: 10 May 2022 13:59
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek
>>> Kumar <Ab...@shapeblue.com>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
>>>> Hi all,
>>>>
>>>> IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
>>>> Documentation for it is available at
>>>> http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.htm
>>>> l#isolated-network-and-vpc-tier
>> <http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html
>> #isolated-network-and-vpc-tier> (will be available in the official
>> docs once 4.17.0 version of docs is built).
>>>>
>>>
>>> Great work!
>>>
>>> I see only static routing is supported. But do we publish something
>>> on the message bus once a new VR/VPC is created?
>>>
>>> This way you could pick up these messages and have the network create
>>> a
>>> (static) route based on those.
>>>
>>> ExaBGP for example could be used to inject such routes.
>>>
>>> Wido
>>>
>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>>>> in+Isolated+Network+and+VPC+with+Static+Routing
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
>> n+Isolated+Network+and+VPC+with+Static+Routing>
>>>>
>>>> Regards,
>>>> Abhishek
>>>>
>>>> ________________________________
>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>> Sent: 13 September 2021 14:30
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Thanks Alex, Wei. I've updated the docs here:
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>>>> in+Isolated+Network+and+VPC+with+Static+Routing
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
>> n+Isolated+Network+and+VPC+with+Static+Routing>
>>>>
>>>> I'll leave the thread open for futher discussion/ideas/feedback. I
>>>> think we've completed the phase1 design doc including all feedback
>>>> comments for adding IPv6 support in CloudStack and some initial
>>>> poc/work can be started. My colleagues and I will keep everyone
>>>> posted on this thread and/or on a Github PR as and when we're able
>>>> to
>> start our work on the same (after 4.16, potentially towards 4.17).
>>>>
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Wei ZHOU <us...@gmail.com>
>>>> Sent: Friday, September 10, 2021 20:22
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Agree with Alex.
>>>> We only need to know how many /64 are allocated. We do not care how
>>>> many
>>>> ipv6 addresses are used by VMs.
>>>>
>>>> -Wei
>>>>
>>>> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli
>>>> <Al...@shapeblue.com>
>>>> wrote:
>>>>
>>>>> Hi Rohit,
>>>>>
>>>>> I'd go for option 2, don't see a point tracking anything smaller
>>>>> than a
>>>>> /64 tbh.
>>>>>
>>>>> Cheers
>>>>> Alex
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>>> Sent: 09 September 2021 12:44
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>>>>> agreements:
>>>>>
>>>>>       *   Allocate /64 for both isolated network and VPC tiers, no
>>>>> large  allocation of prefixes to VPC (cons: more static routing
>>>>> rules for upstream
>>>>> router/admins)
>>>>>       *   All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if
>>>>> zone has a  dedicated /64 prefix/block for systemvms
>>>>>
>>>>> The only outstanding question now is:
>>>>>
>>>>>       *   How do we manage IPv6 usage? Can anyone advise how we do
>>>>> IPv6 usage  for shared network (design, implementation and
>>>>> use-cases?)
>>>>> Option1: We don't do it, all user VMs nics have ipv4 address which
>>>>> whose  usage we don't track. For public VR/nics/networks, we can
>>>>> simply add the
>>>>> IPv6 details for a related IPv4 address.
>>>>> Option2: Implement a separate, first-class IPv6 address or /64
>>>>> prefix  tracking/management and usage for all VMs and systemvms nic
>>>>> (this means  account/domain level limits and new billing/records)
>>>>> Option3: other thoughts?
>>>>>
>>>>>
>>>>> Regards.
>>>>>
>>>>> ________________________________
>>>>> From: Alex Mattioli <Al...@shapeblue.com>
>>>>> Sent: Wednesday, September 8, 2021 23:24
>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>> Subject: RE: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hi Rohit, Kristaps,
>>>>>
>>>>> I'd say option 1 as well,  it does create a bit more overhead with
>>>>> static routes but if that's automated for a VPC it can also be
>>>>> easily automated for several tiers of a VPC.  We also don't
>>>>> constrain the amount of tiers in a  VPC.
>>>>> It has the added advantage to be closer to the desired behaviour
>>>>> with dynamic routing in the future, where a VPC VR can announce
>>>>> several subnets upstream.
>>>>>
>>>>> Cheers
>>>>> Alex
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>>> Sent: 08 September 2021 19:04
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hi Kristaps,
>>>>>
>>>>> Thanks for sharing, I suppose that means individual tiers should be
>>>>> allocated /64 instead of larger ipv6 blocks to the whole VPC which
>>>>> could cause wastage.
>>>>>
>>>>> Any objection from anybody?
>>>>>
>>>>> Regards.
>>>>> ________________________________
>>>>> From: Kristaps Cudars <kr...@gmail.com>
>>>>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hello,
>>>>>
>>>>> I asked networking team to comment on "How should the IPv6
>>>>> block/allocation work in VPCs?"
>>>>> Option1: They haven't seen lately devices with limits on how many
>>>>> static routes can be created.
>>>>> Option2: With /60 and /62 assignments and big quantity of routers
>>>>> IPv6 assignment from RIPE NNC can be drained fast.
>>>>>
>>>>> /48 contains 64000 /64
>>>>> /60 contains 16 /64
>>>>> 64000 / 16 = 4000 routers
>>>>>
>>>>>
>>>>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
>>>>>> All,
>>>>>>
>>>>>> After another iteration with Alex, I've updated the design doc.
>>>>>> Kindly
>>>>> review:
>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppor
>>>>>> t+in
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
>> n>
>>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>>
>>>>>>
>>>>>> ... and advise on some outstanding questions:
>>>>>>
>>>>>>       *   How should the IPv6 block/allocation work in VPCs?
>>>>>> Option1: Should this be simply /64 allocation on any new tier, the
>>>>>> cons of this option is one static route/rule per VPC tier. (many
>>>>>> upstream routers may have limit on no. of static routes?)
>>>>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or
>>>>>> /62
>>>>> (4 tiers) for the VPC, this can be filtered based on the
>>>>> vpc.max.networks global setting (3 is default). The pros of this
>>>>> option are less no. of static route/rule and easy programming, but
>>>>> potential wastage of multiple
>>>>> /64 prefix blocks for unused/uncreated tiers.
>>>>>>       *   How do we manage IPv6 usage? Can anyone advise how we do
>>>>>> IPv6
>>>>> usage for shared network (design, implementation and use-cases?)
>>>>>> Option1: We don't do it, all user VMs nics have ipv4 address which
>>>>>> whose
>>>>> usage we don't track. For public VR/nics/networks, we can simply
>>>>> add the
>>>>> IPv6 details for a related IPv4 address.
>>>>>> Option2: Implement a separate, first-class IPv6 address or /64
>>>>>> prefix
>>>>> tracking/management and usage for all VMs and systemvms nic (this
>>>>> means account/domain level limits and new billing/records)
>>>>>>       *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by
>>>>>> default
>>>>> if zone has a IPv6 address block allocated/assigned for use for
>>>>> systemvms (this was mainly thought for VRs, but why no ssvm and
>>>>> cpvms too - any cons of this?)
>>>>>>       *
>>>>>>
>>>>>> Regards.
>>>>>>
>>>>>> ________________________________
>>>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>>>> Sent: Thursday, August 19, 2021 15:45
>>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I've taken feedback from this thread and wrote this design doc:
>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppor
>>>>>> t+in
>> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
>> n>
>>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>>
>>>>>> Kindly review and advise if I missed anything or anything that
>>>>>> needs to
>>>>> be changed/updated. You may comment on the wiki directly too.
>>>>>>
>>>>>> Kindly suggest your views on the following (also in the design doc
>>>>> above):
>>>>>>
>>>>>> Outstanding Questions:
>>>>>>
>>>>>>       *   Should admin or user be able to specify how VPC super
>>>>>> CIDRs are
>>>>> created/needed; for example a user can ask for VPC with /60 super
>>>>> CIDR? Or should CloudStack automatically find/allocate a /64 for a
>>>>> new VPC tier from the root-admin configured /64-/48 block?
>>>>>>       *   Should we explore FRR and iBGP or other strategies to do
>>>>>> dynamic
>>>>> routing which may not require advance/complex configuration in the
>>>>> VR or for the users/admin?
>>>>>>       *   With SLAAC and no dhcpv6, is there a way to support
>>>>>> secondary IPv6
>>>>> addresses (or floating IPv6 addresses for VR/public traffic) for
>>>>> guest VM's nics?
>>>>>>       *   Any thoughts on UI/UX for firewall/routing management?
>>>>>>       *   Any other feature/support for isolated network or VPC
>>>>>> feature that
>>>>> must be explored or supported such as PF, VPN, LB, vpc static
>>>>> routes, vpc gateway etc.
>>>>>>       *   For usage - should we have any consideration, or should we
>>>>>> assume
>>>>> that IPv4 and IPv6 address will go together for every nic; so IPv6
>>>>> usage for a nic is in tandem with Ipv4 address for a nic, i.e. no
>>>>> explicit/new biling/usage needed?
>>>>>>       *   For smoketests, local dev-test should we explore ULA?
>>>>>> Unique Local
>>>>> Address - in the range fc00::/7. Typically only within the 'local'
>>>>> half fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
>>>>> This prefix can be randomly generated at first install by
>>>>> CloudStack in a zone using zoneid etc?
>>>>>>       *   Should we expand support for VR diagnostic feature to
>>>>>> include
>>>>> support for ipv6, traceroute6?
>>>>>>       *   Discuss - expand support/ability to allocate a /60, or /56
>>>>>> etc
>>>>> prefix to an individual VM in shared network (Wido's suggestion)
>>>>>>
>>>>>>
>>>>>> Regards.
>>>>>>
>>>>>> ________________________________
>>>>>> From: Wei ZHOU <us...@gmail.com>
>>>>>> Sent: Tuesday, August 17, 2021 21:16
>>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>>
>>>>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>>>>
>>>>>> I am fine with not having stateful dhcpv6 and privacy
>>>>>> extension/temporary address in phase 1. If community decides not
>>>>>> to do eventually , it is also ok to me.
>>>>>>
>>>>>> We could explore how to better use secondary ipv6 addresses as
>>>>>> Wido advised. It would be great if anyone share some user experience.
>>>>>>
>>>>>> -Wei
>>>>>>
>>>>>>
>>>>>> On Tuesday, 17 August 2021, Kristaps Cudars
>>>>>> <kr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Wei,
>>>>>>>
>>>>>>> Published this month's RFC 9099 and explains in different
>>>>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>>>>> https://www.rfc-editor.org/rfc/rfc9099.html
>> <https://www.rfc-editor.org/rfc/rfc9099.html>
>>>>>>>
>>>>>>>
>>>>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
>>>>>>>> Hi Wido,
>>>>>>>>
>>>>>>>> (cc to Rohit and Alex)
>>>>>>>>
>>>>>>>> It is a good suggestion to use FRR for ipv6. The configuration
>>>>>>>> is quite simple and the VMs can get SLAAC, routes, etc.
>>>>>>>>
>>>>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>>>>> https://datatracker.ietf.org/doc/html/rfc4941
>> <https://datatracker.ietf.org/doc/html/rfc4941>
>>>>>>>>
>>>>>>>> You are right. To use static routing, the admins need to
>>>>>>>> configure the routes in the upstream router, and add some ipv6
>>>>>>>> ranges (eg
>>>>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
>>>>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
>>>>>>>> pick up an IPv6 range
>>>>>>> and
>>>>>>>> assign it to an isolated network or vpc. @Rohit, correct me if
>>>>>>>> I'm
>>>>> wrong.
>>>>>>>>
>>>>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
>>>>>>>> router/VR, router/dns etc via RA messages) will be the only
>>>>>>>> option for now (related
>>>>>>> to
>>>>>>>> your pr https://github.com/apache/cloudstack/pull/3077)
>> <https://github.com/apache/cloudstack/pull/3077)> . Would it
>>>>>>>> be
>>>>>>> good
>>>>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
>>>>>>>> as an option in cloudstack ? The advantages are
>>>>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
>>>>>>>> addresses can be changed
>>>>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>>>>> The problem is, stateful dhcpv6 does not support
>>>>>>>> routers,nameservers,
>>>>>>> etc.
>>>>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>>>>
>>>>>>>> -Wei
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>     
>>>
>>>> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> See my inline responses:
>>>>>>>>>
>>>>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
>>>>>>>>>> discussing
>>>>>>> them
>>>>>>>>> with Alex and came up with a PoC/design which can be
>>>>>>>>> implemented in the following phases:
>>>>>>>>>>
>>>>>>>>>>        *   Phase1: implement ipv6 support in isolated networks
>>>>>>>>>> and
>>>>> VPC
>>>>>>> with
>>>>>>>>> static routing
>>>>>>>>>>        *   Phase2: discuss and implement support for dynamic
>>>>>>>>>> routing
>>>>>>> (TBD)
>>>>>>>>>>
>>>>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>>>>
>>>>>>>>>>        *   IPv6 address management:
>>>>>>>>>>           *   At the zone level root-admin specifies a /64
>>>>>>>>>> public
>>>>> range
>>>>>>> that
>>>>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6
>>>>>>>>> range for
>>>>>>> guest
>>>>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>>>>>           *   On creation of any IPv6 enabled isolated network
>>>>>>>>>> or VPC
>>>>>>> tier,
>>>>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>>>>>           *   We assume SLAAC and autoconfiguration, no DHCPv6
>>>>>>>>>> in the
>>>>>>> zone
>>>>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
>>>>>>>>> of
>>>>>>> slaac be
>>>>>>>>> explored?)
>>>>>>>>>
>>>>>>>>> Privacy Extensions are only a concern for client devices which
>>>>>>>>> roam between different IPv6 networks.
>>>>>>>>>
>>>>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>>>>
>>>>>>>>> This way a network like Google, Facebook, etc could track your
>>>>>>>>> device moving from network to network if they only look at the
>>>>>>>>> last 64-bits of the IPv6 address.
>>>>>>>>>
>>>>>>>>> For servers this is not a problem as you already know in which
>>>>>>>>> network they are.
>>>>>>>>>
>>>>>>>>>>        *   Network offerings: root-admin can create new network
>>>>> offerings
>>>>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>>>>>           *   ipv4 only (default, for backward compatibility all
>>>>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>>>>>           *   ipv4-and-ipv6
>>>>>>>>>>           *   ipv6-only (this can be phase 1.b)
>>>>>>>>>>           *   A new routing option: static (phase1), dynamic
>>>>>>>>>> (phase2,
>>>>>>> with
>>>>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>>>>
>>>>>>>>> This means that the network admin will need to statically route
>>>>>>>>> the
>>>>>>> IPv6
>>>>>>>>> subnet to the VR's outside IPv6 address, for example, on a
>>>>>>>>> JunOS
>>>>>>> router:
>>>>>>>>>
>>>>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
>>>>>>>>> next-hop
>>>>>>>>> 2001:db8:100::50
>>>>>>>>>
>>>>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
>>>>>>>>> the outside (/64) network. In reality this will probably be a
>>>>>>>>> longer address, but this is for just the example.
>>>>>>>>>
>>>>>>>>>>        *   VR changes:
>>>>>>>>>>           *   VR gets its guest and public nics set to inet6
>>>>>>>>>> auto
>>>>>>>>>>           *   For each /64 allocated to guest network and VPC
>>>>>>>>>> tiers,
>>>>>>> radvd
>>>>>>>>> is configured to do RA
>>>>>>>>>
>>>>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
>>>>>>>>> might already want to look into FRRouting. FRR can also
>>>>>>>>> advertise RAs while not doing any routing.
>>>>>>>>>
>>>>>>>>> interface ens4
>>>>>>>>>        no ipv6 nd suppress-ra
>>>>>>>>>        ipv6 nd prefix 2001:db8:500::/64
>>>>>>>>>        ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>>>>
>>>>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
>> <http://docs.frrouting.org/en/latest/ipv6.html>
>>>>>>>>>
>>>>>>>>>>           *   Firewall: a new ipv6 zone/chain is created for
>>>>>>>>>> ipv6
>>>>> where
>>>>>>> ipv6
>>>>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
>>>>>>>>> between
>>>>>>> VPC
>>>>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>>>>
>>>>>>>>> Please take a look at the existing security_group.py script
>>>>>>>>> which implements RFC4890
>>>>>>>>>
>>>>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
>> <https://datatracker.ietf.org/doc/html/rfc4890>
>>>>>>>>>
>>>>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should
>>>>>>>>> always be allowed.
>>>>>>>>>
>>>>>>>>>>           *   It is assumed that static routes are created on
>>>>>>>>>> the
>>>>>>> core/main
>>>>>>>>> router by the admin or automated using some scripts/tools; for
>>>>>>>>> this CloudStack will announce events with details of /64
>>>>>>>>> networks and VR's public IPv6 address that can be consumed by a
>>>>>>>>> rabbitmq/message bus
>>>>>>> client
>>>>>>>>> (for example), or a custom cron job or script as part of
>>>>> orchestration.
>>>>>>>>> (this wouldn't be necessary for dynamic routing bgp with
>>>>>>>>> phase2)\\
>>>>>>>>>
>>>>>>>>> You would only need to announce the /48 or /56 allocated to the
>>>>>>>>> VR, that's all. You don't need to inform the upstream router
>>>>>>>>> about the /64 subnets created within that larger subnet.
>>>>>>>>>
>>>>>>>>>>        *   Guest Networking: With SLAAC, it's easy for
>>>>>>>>>> CloudStack to
>>>>>>>>> calculate allocate and use a /64 and determine the IPv6 address
>>>>>>>>> of VR
>>>>>>> nics
>>>>>>>>> and guest VM nics
>>>>>>>>>>           *   A user create an isolated network/VPC with an
>>>>>>>>>> offering
>>>>>>> that is
>>>>>>>>> ipv6 enabled
>>>>>>>>>>           *   A user can manage firewall for the IPv6
>>>>>>>>>> address/guest
>>>>> nics;
>>>>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>>>>>           *   A users can run workloads in the guest VMs that
>>>>>>>>>> listen
>>>>> on
>>>>>>>>> publically routable ipv6 addresses
>>>>>>>>>>           *   Usage/billing etc continue to work, no change
>>>>>>>>>> needed
>>>>>>>>>>
>>>>>>>>>> Network layout:
>>>>>>>>>>
>>>>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
>>>>>>>>>> VLAN] ->
>>>>>>>>> [guest VMs/nics]
>>>>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>>>>
>>>>>>>>>> Thoughts, feedback?
>>>>>>>>>
>>>>>>>>> Looks doable!
>>>>>>>>>
>>>>>>>>> Side-note: It would be very cool if you could use parts of this
>>>>>>>>> implementation to also route /48, /56, or /60 subnets to
>>>>>>>>> individual VMs in Shared networks.
>>>>>>>>>
>>>>>>>>> Why? This allows for running Docker containers with native IPv6
>>>>>>>>> inside the VM or running a (Open)VPN server inside a VM which
>>>>>>>>> then assigns public IPv6 addresses to clients connected.
>>>>>>>>>
>>>>>>>>> Instead of routing the subnet to a VR we route the subnet to a
>>>>>>>>> single instance in a shared network.
>>>>>>>>>
>>>>>>>>> If we could then also move these subnets between Instances
>>>>>>>>> easily one can quickly migrate to a different instance while
>>>>>>>>> keeping the same IPv6 subnet.
>>>>>>>>>
>>>>>>>>> Wido
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>>>>
>>>>>>>>>>        *   Created an isolated network and deployed a VM in my
>>>>>>>>>> home
>>>>> lab
>>>>>>>>>> The VR running on KVM has following nics
>>>>>>>>>> eth0 - guest network
>>>>>>>>>> eth1 - link local
>>>>>>>>>> eth2 - public network
>>>>>>>>>>
>>>>>>>>>>        *   I setup a custom openwrt router on a RPi4 to serve as
>>>>>>>>>> a
>>>>>>> toy-core
>>>>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker
>>>>>>>>> and I got
>>>>>>> a
>>>>>>>>> /48 allocated. My configuration looks like:
>>>>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>>>>
>>>>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>>>>> subnet/prefix
>>>>>>>>> 0:
>>>>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>>>>>        *
>>>>>>>>>>        *
>>>>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
>>>>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
>>>>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
>>>>>>>>>> net.ipv6.conf.all.autoconf = 1
>>>>>>>>>>
>>>>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
>>>>>>>>>> configured the nics:
>>>>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
>>>>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
>>>>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
>>>>>>>>>> network without cleanup to have it
>>>>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>>>>
>>>>>>>>>>        *
>>>>>>>>>> Next, I created a /64 network for the isolated guest network
>>>>>>>>>> on eth0
>>>>>>> of
>>>>>>>>> VR using radvd:
>>>>>>>>>>
>>>>>>>>>> # cat /etc/radvd.conf
>>>>>>>>>> interface eth0
>>>>>>>>>> {
>>>>>>>>>>          AdvSendAdvert on;
>>>>>>>>>>          MinRtrAdvInterval 5;
>>>>>>>>>>          MaxRtrAdvInterval 15;
>>>>>>>>>>          prefix 2001:470:ed36:1::/64
>>>>>>>>>>          {
>>>>>>>>>>              AdvOnLink on;
>>>>>>>>>>              AdvAutonomous on;
>>>>>>>>>>          };
>>>>>>>>>> };
>>>>>>>>>> systemctl restart radvd
>>>>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
>>>>>>>>>> this
>>>>>>>>> ...:1::/64 network
>>>>>>>>>>        *   Finally I added a static route in toy core-router for
>>>>>>>>>> the
>>>>> new
>>>>>>> /64
>>>>>>>>> IPv6 range in the isolated network
>>>>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>>>>> eth2> dev
>>>>>>>>> <local lan nic>
>>>>>>>>>>        *
>>>>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
>>>>>>>>>> for the
>>>>>>>>> new /64 network
>>>>>>>>>>
>>>>>>>>>> And voila all done! I create a domain AAAA record that points
>>>>>>>>>> to my
>>>>>>>>> guest VM IPv6 address a test webserver on
>>>>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>> <http://ipv6-isolated-ntwk-demo.yadav.cloud/>
>>>>>>>>>>
>>>>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
>>>>>>>>>> after a
>>>>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>

RE: IPV6 in Isolated/VPC networks

Posted by Alex Mattioli <Al...@shapeblue.com>.
> ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4

That's correct

> Or a larger subnet:
> ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4

Not really, the subnets for isolated/VPC networks are always /64.  Which means also no real need to include subnets as well.

Cheers
Alex


 


-----Original Message-----
From: Wido den Hollander <wi...@widodh.nl> 
Sent: 12 May 2022 16:04
To: Abhishek Kumar <Ab...@shapeblue.com>; dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks



On 5/12/22 09:55, Abhishek Kumar wrote:
> Hi Wido,
> 
> I do not understand what you mean by WAB address but

WAB was a type. I meant WAN.

> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the public IP of the network
> (IPv6 of the public NIC of the network VR) in the sample.
> Yes, route for fd23:313a:2f53:3cbf::/64 need to be added to this IP.
> fd23:313a:2f53:3cbf::/64 is guest IPv6 CIDR of the network here.
> 

So that means that I would need to run this command on my upstream router:

ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4

Or a larger subnet:

ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4

> Currently, the message on event bus does not include subnet. Should 
> that be included?

Yes, because then you can pickup those messages and inject the route via ExaBGP into a routing table right away.

> In case of VPCs, there could be multiple tiers which will need 
> multiple routes to be added. Will that be an issue if we include 
> current network/tier subnet in the event message?

No, as long as it points to the same VR you simply have multiple subnets being routed to the same VR.

I do wonder what happens if you destroy the VR and create a new one. The WAN address then changes (due to SLAAC) and thus the routes need to be re-programmed.

Wido

> 
> Regards,
> Abhishek
> 
> 
>   
> 
>   
> 
> ----------------------------------------------------------------------
> --
> *From:* Wido den Hollander <wi...@widodh.nl>
> *Sent:* 10 May 2022 19:01
> *To:* dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek 
> Kumar <Ab...@shapeblue.com>
> *Subject:* Re: IPV6 in Isolated/VPC networks
>  
> Hi,
> 
> Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
>> Yes. When a public IPv6 is assigned or released, CloudStack will publish event with type NET.IP6ASSIGN, NET.IP6RELEASE.
>> These event notifications can be tracked. And with improvements in events framework, these event messages will have network uuid as entityuuid and Network as entitytype. Using this network can be queried using to list IPv6 routes that need to be added.
>> 
>> Sample event message,
>> 
>> {"eventDateTime":"2022-05-10 09:32:12 
>> +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":
>> "14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned public 
>> IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID: 
>> 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":
>> "bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-
>> af19-1e00320001f3","entity":"Network","status":"Completed"}
>> 
>> 
>> ?Sample API call,
>>> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98 
>>> filter=id,name,ip6routes
>> {
>>    "count": 1,
>>    "network": [
>>      {
>>        "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
>>        "ip6routes": [
>>          {
>>            "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
>>            "subnet": "fd23:313a:2f53:3cbf::/64"
>>          }
> 
> Looking at this JSON, does this mean that
> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
> 
> And that I would need to (statically) route fd23:313a:2f53:3cbf::/64 
> to that IP?
> 
> The event message does not include the subnet, that makes it a bit 
> more difficult as you would then also need to do a API-call to gather 
> that information.
> 
> Wido
> 
> P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an 
> AAAA-record for IPv6!
> 
>>        ],
>>        "name": "routing_test"
>>      }
>>    ]
>> }
>> 
>> 
>> 
>> 
>> ________________________________
>> From: Wido den Hollander <wi...@widodh.nl>
>> Sent: 10 May 2022 13:59
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek 
>> Kumar <Ab...@shapeblue.com>
>> Subject: Re: IPV6 in Isolated/VPC networks
>> 
>> Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
>>> Hi all,
>>>
>>> IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
>>> Documentation for it is available at 
>>> http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.htm
>>> l#isolated-network-and-vpc-tier
> <http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html
> #isolated-network-and-vpc-tier> (will be available in the official 
> docs once 4.17.0 version of docs is built).
>>>
>> 
>> Great work!
>> 
>> I see only static routing is supported. But do we publish something 
>> on the message bus once a new VR/VPC is created?
>> 
>> This way you could pick up these messages and have the network create 
>> a
>> (static) route based on those.
>> 
>> ExaBGP for example could be used to inject such routes.
>> 
>> Wido
>> 
>>> [1] 
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>>> in+Isolated+Network+and+VPC+with+Static+Routing
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
> n+Isolated+Network+and+VPC+with+Static+Routing>
>>>
>>> Regards,
>>> Abhishek
>>>
>>> ________________________________
>>> From: Rohit Yadav <ro...@shapeblue.com>
>>> Sent: 13 September 2021 14:30
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Thanks Alex, Wei. I've updated the docs here: 
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+
>>> in+Isolated+Network+and+VPC+with+Static+Routing
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
> n+Isolated+Network+and+VPC+with+Static+Routing>
>>>
>>> I'll leave the thread open for futher discussion/ideas/feedback. I 
>>> think we've completed the phase1 design doc including all feedback 
>>> comments for adding IPv6 support in CloudStack and some initial 
>>> poc/work can be started. My colleagues and I will keep everyone 
>>> posted on this thread and/or on a Github PR as and when we're able 
>>> to
> start our work on the same (after 4.16, potentially towards 4.17).
>>>
>>>
>>> Regards.
>>>
>>> ________________________________
>>> From: Wei ZHOU <us...@gmail.com>
>>> Sent: Friday, September 10, 2021 20:22
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Agree with Alex.
>>> We only need to know how many /64 are allocated. We do not care how 
>>> many
>>> ipv6 addresses are used by VMs.
>>>
>>> -Wei
>>>
>>> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli 
>>> <Al...@shapeblue.com>
>>> wrote:
>>>
>>>> Hi Rohit,
>>>>
>>>> I'd go for option 2, don't see a point tracking anything smaller 
>>>> than a
>>>> /64 tbh.
>>>>
>>>> Cheers
>>>> Alex
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>> Sent: 09 September 2021 12:44
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>>>> agreements:
>>>>
>>>>     *   Allocate /64 for both isolated network and VPC tiers, no 
>>>>large  allocation of prefixes to VPC (cons: more static routing 
>>>>rules for upstream
>>>> router/admins)
>>>>     *   All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if 
>>>>zone has a  dedicated /64 prefix/block for systemvms
>>>>
>>>> The only outstanding question now is:
>>>>
>>>>     *   How do we manage IPv6 usage? Can anyone advise how we do 
>>>>IPv6 usage  for shared network (design, implementation and 
>>>>use-cases?)
>>>> Option1: We don't do it, all user VMs nics have ipv4 address which 
>>>>whose  usage we don't track. For public VR/nics/networks, we can 
>>>>simply add the
>>>> IPv6 details for a related IPv4 address.
>>>> Option2: Implement a separate, first-class IPv6 address or /64 
>>>>prefix  tracking/management and usage for all VMs and systemvms nic 
>>>>(this means  account/domain level limits and new billing/records)
>>>> Option3: other thoughts?
>>>>
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Alex Mattioli <Al...@shapeblue.com>
>>>> Sent: Wednesday, September 8, 2021 23:24
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: RE: IPV6 in Isolated/VPC networks
>>>>
>>>> Hi Rohit, Kristaps,
>>>>
>>>> I'd say option 1 as well,  it does create a bit more overhead with 
>>>> static routes but if that's automated for a VPC it can also be 
>>>> easily automated for several tiers of a VPC.  We also don't 
>>>> constrain the amount of tiers in a  VPC.
>>>> It has the added advantage to be closer to the desired behaviour 
>>>> with dynamic routing in the future, where a VPC VR can announce 
>>>> several subnets upstream.
>>>>
>>>> Cheers
>>>> Alex
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>> Sent: 08 September 2021 19:04
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Hi Kristaps,
>>>>
>>>> Thanks for sharing, I suppose that means individual tiers should be 
>>>> allocated /64 instead of larger ipv6 blocks to the whole VPC which 
>>>> could cause wastage.
>>>>
>>>> Any objection from anybody?
>>>>
>>>> Regards.
>>>> ________________________________
>>>> From: Kristaps Cudars <kr...@gmail.com>
>>>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Hello,
>>>>
>>>> I asked networking team to comment on "How should the IPv6 
>>>> block/allocation work in VPCs?"
>>>> Option1: They haven't seen lately devices with limits on how many 
>>>> static routes can be created.
>>>> Option2: With /60 and /62 assignments and big quantity of routers 
>>>> IPv6 assignment from RIPE NNC can be drained fast.
>>>>
>>>> /48 contains 64000 /64
>>>> /60 contains 16 /64
>>>> 64000 / 16 = 4000 routers
>>>>
>>>>
>>>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
>>>>> All,
>>>>>
>>>>> After another iteration with Alex, I've updated the design doc. 
>>>>> Kindly
>>>> review:
>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppor
>>>>> t+in
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
> n>
>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>
>>>>>
>>>>> ... and advise on some outstanding questions:
>>>>>
>>>>>     *   How should the IPv6 block/allocation work in VPCs?
>>>>> Option1: Should this be simply /64 allocation on any new tier, the  
>>>>>cons of this option is one static route/rule per VPC tier. (many  
>>>>>upstream routers may have limit on no. of static routes?)
>>>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or 
>>>>>/62
>>>> (4 tiers) for the VPC, this can be filtered based on the 
>>>> vpc.max.networks global setting (3 is default). The pros of this 
>>>> option are less no. of static route/rule and easy programming, but 
>>>> potential wastage of multiple
>>>> /64 prefix blocks for unused/uncreated tiers.
>>>>>     *   How do we manage IPv6 usage? Can anyone advise how we do 
>>>>>IPv6
>>>> usage for shared network (design, implementation and use-cases?)
>>>>> Option1: We don't do it, all user VMs nics have ipv4 address which 
>>>>> whose
>>>> usage we don't track. For public VR/nics/networks, we can simply 
>>>> add the
>>>> IPv6 details for a related IPv4 address.
>>>>> Option2: Implement a separate, first-class IPv6 address or /64 
>>>>> prefix
>>>> tracking/management and usage for all VMs and systemvms nic (this 
>>>> means account/domain level limits and new billing/records)
>>>>>     *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by 
>>>>>default
>>>> if zone has a IPv6 address block allocated/assigned for use for 
>>>> systemvms (this was mainly thought for VRs, but why no ssvm and 
>>>> cpvms too - any cons of this?)
>>>>>     *
>>>>>
>>>>> Regards.
>>>>>
>>>>> ________________________________
>>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>>> Sent: Thursday, August 19, 2021 15:45
>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I've taken feedback from this thread and wrote this design doc:
>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Suppor
>>>>> t+in
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+i
> n>
>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>
>>>>> Kindly review and advise if I missed anything or anything that 
>>>>> needs to
>>>> be changed/updated. You may comment on the wiki directly too.
>>>>>
>>>>> Kindly suggest your views on the following (also in the design doc
>>>> above):
>>>>>
>>>>> Outstanding Questions:
>>>>>
>>>>>     *   Should admin or user be able to specify how VPC super 
>>>>>CIDRs are
>>>> created/needed; for example a user can ask for VPC with /60 super 
>>>> CIDR? Or should CloudStack automatically find/allocate a /64 for a 
>>>> new VPC tier from the root-admin configured /64-/48 block?
>>>>>     *   Should we explore FRR and iBGP or other strategies to do 
>>>>>dynamic
>>>> routing which may not require advance/complex configuration in the 
>>>> VR or for the users/admin?
>>>>>     *   With SLAAC and no dhcpv6, is there a way to support 
>>>>>secondary IPv6
>>>> addresses (or floating IPv6 addresses for VR/public traffic) for 
>>>> guest VM's nics?
>>>>>     *   Any thoughts on UI/UX for firewall/routing management?
>>>>>     *   Any other feature/support for isolated network or VPC 
>>>>>feature that
>>>> must be explored or supported such as PF, VPN, LB, vpc static 
>>>> routes, vpc gateway etc.
>>>>>     *   For usage - should we have any consideration, or should we 
>>>>>assume
>>>> that IPv4 and IPv6 address will go together for every nic; so IPv6 
>>>> usage for a nic is in tandem with Ipv4 address for a nic, i.e. no 
>>>> explicit/new biling/usage needed?
>>>>>     *   For smoketests, local dev-test should we explore ULA? 
>>>>>Unique Local
>>>> Address - in the range fc00::/7. Typically only within the 'local' 
>>>> half fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
>>>> This prefix can be randomly generated at first install by 
>>>> CloudStack in a zone using zoneid etc?
>>>>>     *   Should we expand support for VR diagnostic feature to 
>>>>>include
>>>> support for ipv6, traceroute6?
>>>>>     *   Discuss - expand support/ability to allocate a /60, or /56 
>>>>>etc
>>>> prefix to an individual VM in shared network (Wido's suggestion)
>>>>>
>>>>>
>>>>> Regards.
>>>>>
>>>>> ________________________________
>>>>> From: Wei ZHOU <us...@gmail.com>
>>>>> Sent: Tuesday, August 17, 2021 21:16
>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>>>
>>>>> I am fine with not having stateful dhcpv6 and privacy 
>>>>> extension/temporary address in phase 1. If community decides not 
>>>>> to do eventually , it is also ok to me.
>>>>>
>>>>> We could explore how to better use secondary ipv6 addresses as 
>>>>> Wido advised. It would be great if anyone share some user experience.
>>>>>
>>>>> -Wei
>>>>>
>>>>>
>>>>> On Tuesday, 17 August 2021, Kristaps Cudars 
>>>>> <kr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Wei,
>>>>>>
>>>>>> Published this month's RFC 9099 and explains in different 
>>>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>>>> https://www.rfc-editor.org/rfc/rfc9099.html
> <https://www.rfc-editor.org/rfc/rfc9099.html>
>>>>>>
>>>>>>
>>>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
>>>>>>> Hi Wido,
>>>>>>>
>>>>>>> (cc to Rohit and Alex)
>>>>>>>
>>>>>>> It is a good suggestion to use FRR for ipv6. The configuration 
>>>>>>> is quite simple and the VMs can get SLAAC, routes, etc.
>>>>>>>
>>>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>>>> https://datatracker.ietf.org/doc/html/rfc4941
> <https://datatracker.ietf.org/doc/html/rfc4941>
>>>>>>>
>>>>>>> You are right. To use static routing, the admins need to 
>>>>>>> configure the routes in the upstream router, and add some ipv6 
>>>>>>> ranges (eg
>>>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop 
>>>>>>> (which will be configured in VRs) in CloudStack. CloudStack will 
>>>>>>> pick up an IPv6 range
>>>>>> and
>>>>>>> assign it to an isolated network or vpc. @Rohit, correct me if 
>>>>>>> I'm
>>>> wrong.
>>>>>>>
>>>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from 
>>>>>>> router/VR, router/dns etc via RA messages) will be the only 
>>>>>>> option for now (related
>>>>>> to
>>>>>>> your pr https://github.com/apache/cloudstack/pull/3077)
> <https://github.com/apache/cloudstack/pull/3077)> . Would it
>>>>>>> be
>>>>>> good
>>>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq) 
>>>>>>> as an option in cloudstack ? The advantages are
>>>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6 
>>>>>>> addresses can be changed
>>>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>>>> The problem is, stateful dhcpv6 does not support 
>>>>>>> routers,nameservers,
>>>>>> etc.
>>>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>>>
>>>>>>> -Wei
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>   
>> 
>>> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> See my inline responses:
>>>>>>>>
>>>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> Thanks for your feedback and ideas, I've gone ahead with 
>>>>>>>>> discussing
>>>>>> them
>>>>>>>> with Alex and came up with a PoC/design which can be 
>>>>>>>> implemented in the following phases:
>>>>>>>>>
>>>>>>>>>      *   Phase1: implement ipv6 support in isolated networks 
>>>>>>>>>and
>>>> VPC
>>>>>> with
>>>>>>>> static routing
>>>>>>>>>      *   Phase2: discuss and implement support for dynamic 
>>>>>>>>>routing
>>>>>> (TBD)
>>>>>>>>>
>>>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>>>
>>>>>>>>>      *   IPv6 address management:
>>>>>>>>>         *   At the zone level root-admin specifies a /64 
>>>>>>>>>public
>>>> range
>>>>>> that
>>>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6 
>>>>>>>> range for
>>>>>> guest
>>>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>>>>         *   On creation of any IPv6 enabled isolated network 
>>>>>>>>>or VPC
>>>>>> tier,
>>>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>>>>         *   We assume SLAAC and autoconfiguration, no DHCPv6 
>>>>>>>>>in the
>>>>>> zone
>>>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941 
>>>>>>>> of
>>>>>> slaac be
>>>>>>>> explored?)
>>>>>>>>
>>>>>>>> Privacy Extensions are only a concern for client devices which 
>>>>>>>> roam between different IPv6 networks.
>>>>>>>>
>>>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>>>
>>>>>>>> This way a network like Google, Facebook, etc could track your 
>>>>>>>> device moving from network to network if they only look at the 
>>>>>>>> last 64-bits of the IPv6 address.
>>>>>>>>
>>>>>>>> For servers this is not a problem as you already know in which 
>>>>>>>> network they are.
>>>>>>>>
>>>>>>>>>      *   Network offerings: root-admin can create new network
>>>> offerings
>>>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>>>>         *   ipv4 only (default, for backward compatibility all
>>>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>>>>         *   ipv4-and-ipv6
>>>>>>>>>         *   ipv6-only (this can be phase 1.b)
>>>>>>>>>         *   A new routing option: static (phase1), dynamic 
>>>>>>>>>(phase2,
>>>>>> with
>>>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>>>
>>>>>>>> This means that the network admin will need to statically route 
>>>>>>>> the
>>>>>> IPv6
>>>>>>>> subnet to the VR's outside IPv6 address, for example, on a 
>>>>>>>> JunOS
>>>>>> router:
>>>>>>>>
>>>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48 
>>>>>>>> next-hop
>>>>>>>> 2001:db8:100::50
>>>>>>>>
>>>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on 
>>>>>>>> the outside (/64) network. In reality this will probably be a 
>>>>>>>> longer address, but this is for just the example.
>>>>>>>>
>>>>>>>>>      *   VR changes:
>>>>>>>>>         *   VR gets its guest and public nics set to inet6 
>>>>>>>>>auto
>>>>>>>>>         *   For each /64 allocated to guest network and VPC 
>>>>>>>>>tiers,
>>>>>> radvd
>>>>>>>> is configured to do RA
>>>>>>>>
>>>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you 
>>>>>>>> might already want to look into FRRouting. FRR can also 
>>>>>>>> advertise RAs while not doing any routing.
>>>>>>>>
>>>>>>>> interface ens4
>>>>>>>>      no ipv6 nd suppress-ra
>>>>>>>>      ipv6 nd prefix 2001:db8:500::/64
>>>>>>>>      ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>>>
>>>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
> <http://docs.frrouting.org/en/latest/ipv6.html>
>>>>>>>>
>>>>>>>>>         *   Firewall: a new ipv6 zone/chain is created for 
>>>>>>>>>ipv6
>>>> where
>>>>>> ipv6
>>>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs 
>>>>>>>> between
>>>>>> VPC
>>>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>>>
>>>>>>>> Please take a look at the existing security_group.py script 
>>>>>>>> which implements RFC4890
>>>>>>>>
>>>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
> <https://datatracker.ietf.org/doc/html/rfc4890>
>>>>>>>>
>>>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should 
>>>>>>>> always be allowed.
>>>>>>>>
>>>>>>>>>         *   It is assumed that static routes are created on 
>>>>>>>>>the
>>>>>> core/main
>>>>>>>> router by the admin or automated using some scripts/tools; for 
>>>>>>>> this CloudStack will announce events with details of /64 
>>>>>>>> networks and VR's public IPv6 address that can be consumed by a 
>>>>>>>> rabbitmq/message bus
>>>>>> client
>>>>>>>> (for example), or a custom cron job or script as part of
>>>> orchestration.
>>>>>>>> (this wouldn't be necessary for dynamic routing bgp with 
>>>>>>>> phase2)\\
>>>>>>>>
>>>>>>>> You would only need to announce the /48 or /56 allocated to the 
>>>>>>>> VR, that's all. You don't need to inform the upstream router 
>>>>>>>> about the /64 subnets created within that larger subnet.
>>>>>>>>
>>>>>>>>>      *   Guest Networking: With SLAAC, it's easy for 
>>>>>>>>>CloudStack to
>>>>>>>> calculate allocate and use a /64 and determine the IPv6 address 
>>>>>>>> of VR
>>>>>> nics
>>>>>>>> and guest VM nics
>>>>>>>>>         *   A user create an isolated network/VPC with an 
>>>>>>>>>offering
>>>>>> that is
>>>>>>>> ipv6 enabled
>>>>>>>>>         *   A user can manage firewall for the IPv6 
>>>>>>>>>address/guest
>>>> nics;
>>>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>>>>         *   A users can run workloads in the guest VMs that 
>>>>>>>>>listen
>>>> on
>>>>>>>> publically routable ipv6 addresses
>>>>>>>>>         *   Usage/billing etc continue to work, no change 
>>>>>>>>>needed
>>>>>>>>>
>>>>>>>>> Network layout:
>>>>>>>>>
>>>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a 
>>>>>>>>> VLAN] ->
>>>>>>>> [guest VMs/nics]
>>>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>>>
>>>>>>>>> Thoughts, feedback?
>>>>>>>>
>>>>>>>> Looks doable!
>>>>>>>>
>>>>>>>> Side-note: It would be very cool if you could use parts of this 
>>>>>>>> implementation to also route /48, /56, or /60 subnets to 
>>>>>>>> individual VMs in Shared networks.
>>>>>>>>
>>>>>>>> Why? This allows for running Docker containers with native IPv6 
>>>>>>>> inside the VM or running a (Open)VPN server inside a VM which 
>>>>>>>> then assigns public IPv6 addresses to clients connected.
>>>>>>>>
>>>>>>>> Instead of routing the subnet to a VR we route the subnet to a 
>>>>>>>> single instance in a shared network.
>>>>>>>>
>>>>>>>> If we could then also move these subnets between Instances 
>>>>>>>> easily one can quickly migrate to a different instance while 
>>>>>>>> keeping the same IPv6 subnet.
>>>>>>>>
>>>>>>>> Wido
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>>>
>>>>>>>>>      *   Created an isolated network and deployed a VM in my 
>>>>>>>>>home
>>>> lab
>>>>>>>>> The VR running on KVM has following nics
>>>>>>>>> eth0 - guest network
>>>>>>>>> eth1 - link local
>>>>>>>>> eth2 - public network
>>>>>>>>>
>>>>>>>>>      *   I setup a custom openwrt router on a RPi4 to serve as 
>>>>>>>>>a
>>>>>> toy-core
>>>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker 
>>>>>>>> and I got
>>>>>> a
>>>>>>>> /48 allocated. My configuration looks like:
>>>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>>>
>>>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>>>> subnet/prefix
>>>>>>>> 0:
>>>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>>>>      *
>>>>>>>>>      *
>>>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0  
>>>>>>>>>net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
>>>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1  
>>>>>>>>>net.ipv6.conf.all.autoconf = 1
>>>>>>>>>
>>>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And 
>>>>>>>>> configured the nics:
>>>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo 
>>>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces 
>>>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated 
>>>>>>>>> network without cleanup to have it
>>>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>>>
>>>>>>>>>      *
>>>>>>>>> Next, I created a /64 network for the isolated guest network  
>>>>>>>>>on eth0
>>>>>> of
>>>>>>>> VR using radvd:
>>>>>>>>>
>>>>>>>>> # cat /etc/radvd.conf
>>>>>>>>> interface eth0
>>>>>>>>> {
>>>>>>>>>        AdvSendAdvert on;
>>>>>>>>>        MinRtrAdvInterval 5;
>>>>>>>>>        MaxRtrAdvInterval 15;
>>>>>>>>>        prefix 2001:470:ed36:1::/64
>>>>>>>>>        {
>>>>>>>>>            AdvOnLink on;
>>>>>>>>>            AdvAutonomous on;
>>>>>>>>>        };
>>>>>>>>> };
>>>>>>>>> systemctl restart radvd
>>>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in  
>>>>>>>>>this
>>>>>>>> ...:1::/64 network
>>>>>>>>>      *   Finally I added a static route in toy core-router for 
>>>>>>>>>the
>>>> new
>>>>>> /64
>>>>>>>> IPv6 range in the isolated network
>>>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>>>> eth2> dev
>>>>>>>> <local lan nic>
>>>>>>>>>      *
>>>>>>>>> ... and I enabled firewall rules to allow any traffic to pass  
>>>>>>>>>for the
>>>>>>>> new /64 network
>>>>>>>>>
>>>>>>>>> And voila all done! I create a domain AAAA record that points 
>>>>>>>>> to my
>>>>>>>> guest VM IPv6 address a test webserver on
>>>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
> <http://ipv6-isolated-ntwk-demo.yadav.cloud/>
>>>>>>>>>
>>>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block 
>>>>>>>>> after a
>>>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>> 

Re: IPV6 in Isolated/VPC networks

Posted by Wido den Hollander <wi...@widodh.nl>.

On 5/12/22 09:55, Abhishek Kumar wrote:
> Hi Wido,
> 
> I do not understand what you mean by WAB address but

WAB was a type. I meant WAN.

> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the public IP of the network
> (IPv6 of the public NIC of the network VR) in the sample.
> Yes, route for fd23:313a:2f53:3cbf::/64 need to be added to this IP.
> fd23:313a:2f53:3cbf::/64 is guest IPv6 CIDR of the network here.
> 

So that means that I would need to run this command on my upstream router:

ipv6 route fd23:313a:2f53:3cbf::/64 fd23:313a:2f53:3000:1c00:baff:fe00:4

Or a larger subnet:

ipv6 route fd23:313a:2f53:3c00::/56 fd23:313a:2f53:3000:1c00:baff:fe00:4

> Currently, the message on event bus does not include subnet. Should that
> be included?

Yes, because then you can pickup those messages and inject the route via
ExaBGP into a routing table right away.

> In case of VPCs, there could be multiple tiers which will need multiple
> routes to be added. Will that be an issue if we include current
> network/tier subnet in the event message?

No, as long as it points to the same VR you simply have multiple subnets
being routed to the same VR.

I do wonder what happens if you destroy the VR and create a new one. The
WAN address then changes (due to SLAAC) and thus the routes need to be
re-programmed.

Wido

> 
> Regards,
> Abhishek
> 
> 
>   
> 
>   
> 
> ------------------------------------------------------------------------
> *From:* Wido den Hollander <wi...@widodh.nl>
> *Sent:* 10 May 2022 19:01
> *To:* dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek
> Kumar <Ab...@shapeblue.com>
> *Subject:* Re: IPV6 in Isolated/VPC networks
>  
> Hi,
> 
> Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
>> Yes. When a public IPv6 is assigned or released, CloudStack will publish event with type NET.IP6ASSIGN, NET.IP6RELEASE.
>> These event notifications can be tracked. And with improvements in events framework, these event messages will have network uuid as entityuuid and Network as entitytype. Using this network can be queried using to list IPv6 routes that need to be added.
>> 
>> Sample event message,
>> 
>> {"eventDateTime":"2022-05-10 09:32:12 +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":"14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned public IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID: 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":"bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-af19-1e00320001f3","entity":"Network","status":"Completed"}
>> 
>> 
>> ?Sample API call,
>>> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98 filter=id,name,ip6routes
>> {
>>    "count": 1,
>>    "network": [
>>      {
>>        "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
>>        "ip6routes": [
>>          {
>>            "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
>>            "subnet": "fd23:313a:2f53:3cbf::/64"
>>          }
> 
> Looking at this JSON, does this mean that
> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
> 
> And that I would need to (statically) route fd23:313a:2f53:3cbf::/64 to
> that IP?
> 
> The event message does not include the subnet, that makes it a bit more
> difficult as you would then also need to do a API-call to gather that
> information.
> 
> Wido
> 
> P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an
> AAAA-record for IPv6!
> 
>>        ],
>>        "name": "routing_test"
>>      }
>>    ]
>> }
>> 
>> 
>> 
>> 
>> ________________________________
>> From: Wido den Hollander <wi...@widodh.nl>
>> Sent: 10 May 2022 13:59
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek Kumar <Ab...@shapeblue.com>
>> Subject: Re: IPV6 in Isolated/VPC networks
>> 
>> Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
>>> Hi all,
>>>
>>> IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
>>> Documentation for it is available at http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier
> <http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier>
> (will be available in the official docs once 4.17.0 version of docs is
> built).
>>>
>> 
>> Great work!
>> 
>> I see only static routing is supported. But do we publish something on
>> the message bus once a new VR/VPC is created?
>> 
>> This way you could pick up these messages and have the network create a
>> (static) route based on those.
>> 
>> ExaBGP for example could be used to inject such routes.
>> 
>> Wido
>> 
>>> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing>
>>>
>>> Regards,
>>> Abhishek
>>>
>>> ________________________________
>>> From: Rohit Yadav <ro...@shapeblue.com>
>>> Sent: 13 September 2021 14:30
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Thanks Alex, Wei. I've updated the docs here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing>
>>>
>>> I'll leave the thread open for futher discussion/ideas/feedback. I think we've completed the phase1 design doc including all feedback comments for adding IPv6 support in CloudStack and some initial poc/work can be started. My colleagues and I will keep everyone posted on this thread and/or on a Github PR as and when we're able to
> start our work on the same (after 4.16, potentially towards 4.17).
>>>
>>>
>>> Regards.
>>>
>>> ________________________________
>>> From: Wei ZHOU <us...@gmail.com>
>>> Sent: Friday, September 10, 2021 20:22
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Agree with Alex.
>>> We only need to know how many /64 are allocated. We do not care how many
>>> ipv6 addresses are used by VMs.
>>>
>>> -Wei
>>>
>>> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <Al...@shapeblue.com>
>>> wrote:
>>>
>>>> Hi Rohit,
>>>>
>>>> I'd go for option 2, don't see a point tracking anything smaller than a
>>>> /64 tbh.
>>>>
>>>> Cheers
>>>> Alex
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>> Sent: 09 September 2021 12:44
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>>>> agreements:
>>>>
>>>>     *   Allocate /64 for both isolated network and VPC tiers, no large
>>>> allocation of prefixes to VPC (cons: more static routing rules for upstream
>>>> router/admins)
>>>>     *   All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a
>>>> dedicated /64 prefix/block for systemvms
>>>>
>>>> The only outstanding question now is:
>>>>
>>>>     *   How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage
>>>> for shared network (design, implementation and use-cases?)
>>>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>>>> usage we don't track. For public VR/nics/networks, we can simply add the
>>>> IPv6 details for a related IPv4 address.
>>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>>>> tracking/management and usage for all VMs and systemvms nic (this means
>>>> account/domain level limits and new billing/records)
>>>> Option3: other thoughts?
>>>>
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Alex Mattioli <Al...@shapeblue.com>
>>>> Sent: Wednesday, September 8, 2021 23:24
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: RE: IPV6 in Isolated/VPC networks
>>>>
>>>> Hi Rohit, Kristaps,
>>>>
>>>> I'd say option 1 as well,  it does create a bit more overhead with static
>>>> routes but if that's automated for a VPC it can also be easily automated
>>>> for several tiers of a VPC.  We also don't constrain the amount of tiers in
>>>> a  VPC.
>>>> It has the added advantage to be closer to the desired behaviour with
>>>> dynamic routing in the future, where a VPC VR can announce several subnets
>>>> upstream.
>>>>
>>>> Cheers
>>>> Alex
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>> Sent: 08 September 2021 19:04
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Hi Kristaps,
>>>>
>>>> Thanks for sharing, I suppose that means individual tiers should be
>>>> allocated /64 instead of larger ipv6 blocks to the whole VPC which could
>>>> cause wastage.
>>>>
>>>> Any objection from anybody?
>>>>
>>>> Regards.
>>>> ________________________________
>>>> From: Kristaps Cudars <kr...@gmail.com>
>>>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Hello,
>>>>
>>>> I asked networking team to comment on "How should the IPv6
>>>> block/allocation work in VPCs?"
>>>> Option1: They haven't seen lately devices with limits on how many static
>>>> routes can be created.
>>>> Option2: With /60 and /62 assignments and big quantity of routers IPv6
>>>> assignment from RIPE NNC can be drained fast.
>>>>
>>>> /48 contains 64000 /64
>>>> /60 contains 16 /64
>>>> 64000 / 16 = 4000 routers
>>>>
>>>>
>>>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
>>>>> All,
>>>>>
>>>>> After another iteration with Alex, I've updated the design doc. Kindly
>>>> review:
>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in>
>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>
>>>>>
>>>>> ... and advise on some outstanding questions:
>>>>>
>>>>>     *   How should the IPv6 block/allocation work in VPCs?
>>>>> Option1: Should this be simply /64 allocation on any new tier, the
>>>>> cons of this option is one static route/rule per VPC tier. (many
>>>>> upstream routers may have limit on no. of static routes?)
>>>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
>>>> (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks
>>>> global setting (3 is default). The pros of this option are less no. of
>>>> static route/rule and easy programming, but potential wastage of multiple
>>>> /64 prefix blocks for unused/uncreated tiers.
>>>>>     *   How do we manage IPv6 usage? Can anyone advise how we do IPv6
>>>> usage for shared network (design, implementation and use-cases?)
>>>>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>>>> usage we don't track. For public VR/nics/networks, we can simply add the
>>>> IPv6 details for a related IPv4 address.
>>>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>>>> tracking/management and usage for all VMs and systemvms nic (this means
>>>> account/domain level limits and new billing/records)
>>>>>     *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by default
>>>> if zone has a IPv6 address block allocated/assigned for use for systemvms
>>>> (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons
>>>> of this?)
>>>>>     *
>>>>>
>>>>> Regards.
>>>>>
>>>>> ________________________________
>>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>>> Sent: Thursday, August 19, 2021 15:45
>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I've taken feedback from this thread and wrote this design doc:
>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> <https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in>
>>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>>
>>>>> Kindly review and advise if I missed anything or anything that needs to
>>>> be changed/updated. You may comment on the wiki directly too.
>>>>>
>>>>> Kindly suggest your views on the following (also in the design doc
>>>> above):
>>>>>
>>>>> Outstanding Questions:
>>>>>
>>>>>     *   Should admin or user be able to specify how VPC super CIDRs are
>>>> created/needed; for example a user can ask for VPC with /60 super CIDR? Or
>>>> should CloudStack automatically find/allocate a /64 for a new VPC tier from
>>>> the root-admin configured /64-/48 block?
>>>>>     *   Should we explore FRR and iBGP or other strategies to do dynamic
>>>> routing which may not require advance/complex configuration in the VR or
>>>> for the users/admin?
>>>>>     *   With SLAAC and no dhcpv6, is there a way to support secondary IPv6
>>>> addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's
>>>> nics?
>>>>>     *   Any thoughts on UI/UX for firewall/routing management?
>>>>>     *   Any other feature/support for isolated network or VPC feature that
>>>> must be explored or supported such as PF, VPN, LB, vpc static routes, vpc
>>>> gateway etc.
>>>>>     *   For usage - should we have any consideration, or should we assume
>>>> that IPv4 and IPv6 address will go together for every nic; so IPv6 usage
>>>> for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new
>>>> biling/usage needed?
>>>>>     *   For smoketests, local dev-test should we explore ULA? Unique Local
>>>> Address - in the range fc00::/7. Typically only within the 'local' half
>>>> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
>>>> This prefix can be randomly generated at first install by CloudStack in a
>>>> zone using zoneid etc?
>>>>>     *   Should we expand support for VR diagnostic feature to include
>>>> support for ipv6, traceroute6?
>>>>>     *   Discuss - expand support/ability to allocate a /60, or /56 etc
>>>> prefix to an individual VM in shared network (Wido's suggestion)
>>>>>
>>>>>
>>>>> Regards.
>>>>>
>>>>> ________________________________
>>>>> From: Wei ZHOU <us...@gmail.com>
>>>>> Sent: Tuesday, August 17, 2021 21:16
>>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>>
>>>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>>>
>>>>> I am fine with not having stateful dhcpv6 and privacy
>>>>> extension/temporary address in phase 1. If community decides not to do
>>>>> eventually , it is also ok to me.
>>>>>
>>>>> We could explore how to better use secondary ipv6 addresses as Wido
>>>>> advised. It would be great if anyone share some user experience.
>>>>>
>>>>> -Wei
>>>>>
>>>>>
>>>>> On Tuesday, 17 August 2021, Kristaps Cudars
>>>>> <kr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Wei,
>>>>>>
>>>>>> Published this month's RFC 9099 and explains in different
>>>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>>>> https://www.rfc-editor.org/rfc/rfc9099.html
> <https://www.rfc-editor.org/rfc/rfc9099.html>
>>>>>>
>>>>>>
>>>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
>>>>>>> Hi Wido,
>>>>>>>
>>>>>>> (cc to Rohit and Alex)
>>>>>>>
>>>>>>> It is a good suggestion to use FRR for ipv6. The configuration is
>>>>>>> quite simple and the VMs can get SLAAC, routes, etc.
>>>>>>>
>>>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>>>> https://datatracker.ietf.org/doc/html/rfc4941
> <https://datatracker.ietf.org/doc/html/rfc4941>
>>>>>>>
>>>>>>> You are right. To use static routing, the admins need to configure
>>>>>>> the routes in the upstream router, and add some ipv6 ranges (eg
>>>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
>>>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
>>>>>>> pick up an IPv6 range
>>>>>> and
>>>>>>> assign it to an isolated network or vpc. @Rohit, correct me if I'm
>>>> wrong.
>>>>>>>
>>>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
>>>>>>> router/VR, router/dns etc via RA messages) will be the only option
>>>>>>> for now (related
>>>>>> to
>>>>>>> your pr https://github.com/apache/cloudstack/pull/3077)
> <https://github.com/apache/cloudstack/pull/3077)> . Would it
>>>>>>> be
>>>>>> good
>>>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
>>>>>>> as an option in cloudstack ? The advantages are
>>>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
>>>>>>> addresses can be changed
>>>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>>>> The problem is, stateful dhcpv6 does not support
>>>>>>> routers,nameservers,
>>>>>> etc.
>>>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>>>
>>>>>>> -Wei
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>   
>> 
>>> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> See my inline responses:
>>>>>>>>
>>>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
>>>>>>>>> discussing
>>>>>> them
>>>>>>>> with Alex and came up with a PoC/design which can be implemented
>>>>>>>> in the following phases:
>>>>>>>>>
>>>>>>>>>      *   Phase1: implement ipv6 support in isolated networks and
>>>> VPC
>>>>>> with
>>>>>>>> static routing
>>>>>>>>>      *   Phase2: discuss and implement support for dynamic routing
>>>>>> (TBD)
>>>>>>>>>
>>>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>>>
>>>>>>>>>      *   IPv6 address management:
>>>>>>>>>         *   At the zone level root-admin specifies a /64 public
>>>> range
>>>>>> that
>>>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6 range
>>>>>>>> for
>>>>>> guest
>>>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>>>>         *   On creation of any IPv6 enabled isolated network or VPC
>>>>>> tier,
>>>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>>>>         *   We assume SLAAC and autoconfiguration, no DHCPv6 in the
>>>>>> zone
>>>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
>>>>>>>> of
>>>>>> slaac be
>>>>>>>> explored?)
>>>>>>>>
>>>>>>>> Privacy Extensions are only a concern for client devices which
>>>>>>>> roam between different IPv6 networks.
>>>>>>>>
>>>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>>>
>>>>>>>> This way a network like Google, Facebook, etc could track your
>>>>>>>> device moving from network to network if they only look at the
>>>>>>>> last 64-bits of the IPv6 address.
>>>>>>>>
>>>>>>>> For servers this is not a problem as you already know in which
>>>>>>>> network they are.
>>>>>>>>
>>>>>>>>>      *   Network offerings: root-admin can create new network
>>>> offerings
>>>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>>>>         *   ipv4 only (default, for backward compatibility all
>>>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>>>>         *   ipv4-and-ipv6
>>>>>>>>>         *   ipv6-only (this can be phase 1.b)
>>>>>>>>>         *   A new routing option: static (phase1), dynamic (phase2,
>>>>>> with
>>>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>>>
>>>>>>>> This means that the network admin will need to statically route
>>>>>>>> the
>>>>>> IPv6
>>>>>>>> subnet to the VR's outside IPv6 address, for example, on a JunOS
>>>>>> router:
>>>>>>>>
>>>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
>>>>>>>> next-hop
>>>>>>>> 2001:db8:100::50
>>>>>>>>
>>>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
>>>>>>>> the outside (/64) network. In reality this will probably be a
>>>>>>>> longer address, but this is for just the example.
>>>>>>>>
>>>>>>>>>      *   VR changes:
>>>>>>>>>         *   VR gets its guest and public nics set to inet6 auto
>>>>>>>>>         *   For each /64 allocated to guest network and VPC tiers,
>>>>>> radvd
>>>>>>>> is configured to do RA
>>>>>>>>
>>>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
>>>>>>>> might already want to look into FRRouting. FRR can also
>>>>>>>> advertise RAs while not doing any routing.
>>>>>>>>
>>>>>>>> interface ens4
>>>>>>>>      no ipv6 nd suppress-ra
>>>>>>>>      ipv6 nd prefix 2001:db8:500::/64
>>>>>>>>      ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>>>
>>>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
> <http://docs.frrouting.org/en/latest/ipv6.html>
>>>>>>>>
>>>>>>>>>         *   Firewall: a new ipv6 zone/chain is created for ipv6
>>>> where
>>>>>> ipv6
>>>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
>>>>>>>> between
>>>>>> VPC
>>>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>>>
>>>>>>>> Please take a look at the existing security_group.py script
>>>>>>>> which implements RFC4890
>>>>>>>>
>>>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
> <https://datatracker.ietf.org/doc/html/rfc4890>
>>>>>>>>
>>>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should always
>>>>>>>> be allowed.
>>>>>>>>
>>>>>>>>>         *   It is assumed that static routes are created on the
>>>>>> core/main
>>>>>>>> router by the admin or automated using some scripts/tools; for
>>>>>>>> this CloudStack will announce events with details of /64
>>>>>>>> networks and VR's public IPv6 address that can be consumed by a
>>>>>>>> rabbitmq/message bus
>>>>>> client
>>>>>>>> (for example), or a custom cron job or script as part of
>>>> orchestration.
>>>>>>>> (this wouldn't be necessary for dynamic routing bgp with
>>>>>>>> phase2)\\
>>>>>>>>
>>>>>>>> You would only need to announce the /48 or /56 allocated to the
>>>>>>>> VR, that's all. You don't need to inform the upstream router
>>>>>>>> about the /64 subnets created within that larger subnet.
>>>>>>>>
>>>>>>>>>      *   Guest Networking: With SLAAC, it's easy for CloudStack to
>>>>>>>> calculate allocate and use a /64 and determine the IPv6 address
>>>>>>>> of VR
>>>>>> nics
>>>>>>>> and guest VM nics
>>>>>>>>>         *   A user create an isolated network/VPC with an offering
>>>>>> that is
>>>>>>>> ipv6 enabled
>>>>>>>>>         *   A user can manage firewall for the IPv6 address/guest
>>>> nics;
>>>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>>>>         *   A users can run workloads in the guest VMs that listen
>>>> on
>>>>>>>> publically routable ipv6 addresses
>>>>>>>>>         *   Usage/billing etc continue to work, no change needed
>>>>>>>>>
>>>>>>>>> Network layout:
>>>>>>>>>
>>>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
>>>>>>>>> VLAN] ->
>>>>>>>> [guest VMs/nics]
>>>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>>>
>>>>>>>>> Thoughts, feedback?
>>>>>>>>
>>>>>>>> Looks doable!
>>>>>>>>
>>>>>>>> Side-note: It would be very cool if you could use parts of this
>>>>>>>> implementation to also route /48, /56, or /60 subnets to
>>>>>>>> individual VMs in Shared networks.
>>>>>>>>
>>>>>>>> Why? This allows for running Docker containers with native IPv6
>>>>>>>> inside the VM or running a (Open)VPN server inside a VM which
>>>>>>>> then assigns public IPv6 addresses to clients connected.
>>>>>>>>
>>>>>>>> Instead of routing the subnet to a VR we route the subnet to a
>>>>>>>> single instance in a shared network.
>>>>>>>>
>>>>>>>> If we could then also move these subnets between Instances
>>>>>>>> easily one can quickly migrate to a different instance while
>>>>>>>> keeping the same IPv6 subnet.
>>>>>>>>
>>>>>>>> Wido
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>>>
>>>>>>>>>      *   Created an isolated network and deployed a VM in my home
>>>> lab
>>>>>>>>> The VR running on KVM has following nics
>>>>>>>>> eth0 - guest network
>>>>>>>>> eth1 - link local
>>>>>>>>> eth2 - public network
>>>>>>>>>
>>>>>>>>>      *   I setup a custom openwrt router on a RPi4 to serve as a
>>>>>> toy-core
>>>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker and
>>>>>>>> I got
>>>>>> a
>>>>>>>> /48 allocated. My configuration looks like:
>>>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>>>
>>>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>>>> subnet/prefix
>>>>>>>> 0:
>>>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>>>>      *
>>>>>>>>>      *
>>>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
>>>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
>>>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
>>>>>>>>> net.ipv6.conf.all.autoconf = 1
>>>>>>>>>
>>>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
>>>>>>>>> configured the nics:
>>>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
>>>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
>>>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
>>>>>>>>> network without cleanup to have it
>>>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>>>
>>>>>>>>>      *
>>>>>>>>> Next, I created a /64 network for the isolated guest network
>>>>>>>>> on eth0
>>>>>> of
>>>>>>>> VR using radvd:
>>>>>>>>>
>>>>>>>>> # cat /etc/radvd.conf
>>>>>>>>> interface eth0
>>>>>>>>> {
>>>>>>>>>        AdvSendAdvert on;
>>>>>>>>>        MinRtrAdvInterval 5;
>>>>>>>>>        MaxRtrAdvInterval 15;
>>>>>>>>>        prefix 2001:470:ed36:1::/64
>>>>>>>>>        {
>>>>>>>>>            AdvOnLink on;
>>>>>>>>>            AdvAutonomous on;
>>>>>>>>>        };
>>>>>>>>> };
>>>>>>>>> systemctl restart radvd
>>>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
>>>>>>>>> this
>>>>>>>> ...:1::/64 network
>>>>>>>>>      *   Finally I added a static route in toy core-router for the
>>>> new
>>>>>> /64
>>>>>>>> IPv6 range in the isolated network
>>>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>>>> eth2> dev
>>>>>>>> <local lan nic>
>>>>>>>>>      *
>>>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
>>>>>>>>> for the
>>>>>>>> new /64 network
>>>>>>>>>
>>>>>>>>> And voila all done! I create a domain AAAA record that points
>>>>>>>>> to my
>>>>>>>> guest VM IPv6 address a test webserver on
>>>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
> <http://ipv6-isolated-ntwk-demo.yadav.cloud/>
>>>>>>>>>
>>>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
>>>>>>>>> after a
>>>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>> 

Re: IPV6 in Isolated/VPC networks

Posted by Wei ZHOU <us...@gmail.com>.
Hi Ahbishek,

Yes, the message should include the subnet.

The message should be sent when the IPv6 subnets are allocated or released,
right ? If so, there will not be multiple subnets/routes in the message.

-Wei

On Thu, 12 May 2022 at 09:56, Abhishek Kumar <Ab...@shapeblue.com>
wrote:

> Hi Wido,
>
> I do not understand what you mean by WAB address but
> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the public IP of the network (IPv6
> of the public NIC of the network VR) in the sample.
> Yes, route for fd23:313a:2f53:3cbf::/64 need to be added to this IP.
> fd23:313a:2f53:3cbf::/64 is guest IPv6 CIDR of the network here.
>
> Currently, the message on event bus does not include subnet. Should that
> be included?
> In case of VPCs, there could be multiple tiers which will need multiple
> routes to be added. Will that be an issue if we include current
> network/tier subnet in the event message?
>
> Regards,
> Abhishek
> ________________________________
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: 10 May 2022 19:01
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek Kumar
> <Ab...@shapeblue.com>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Hi,
>
> Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
> > Yes. When a public IPv6 is assigned or released, CloudStack will publish
> event with type NET.IP6ASSIGN, NET.IP6RELEASE.
> > These event notifications can be tracked. And with improvements in
> events framework, these event messages will have network uuid as entityuuid
> and Network as entitytype. Using this network can be queried using to list
> IPv6 routes that need to be added.
> >
> > Sample event message,
> >
> > {"eventDateTime":"2022-05-10 09:32:12
> +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":"14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned
> public IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID:
> 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":"bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-af19-1e00320001f3","entity":"Network","status":"Completed"}
> >
> >
> > ?Sample API call,
> >> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98
> filter=id,name,ip6routes
> > {
> >    "count": 1,
> >    "network": [
> >      {
> >        "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
> >        "ip6routes": [
> >          {
> >            "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
> >            "subnet": "fd23:313a:2f53:3cbf::/64"
> >          }
>
> Looking at this JSON, does this mean that
> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
>
> And that I would need to (statically) route fd23:313a:2f53:3cbf::/64 to
> that IP?
>
> The event message does not include the subnet, that makes it a bit more
> difficult as you would then also need to do a API-call to gather that
> information.
>
> Wido
>
> P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an
> AAAA-record for IPv6!
>
> >        ],
> >        "name": "routing_test"
> >      }
> >    ]
> > }
> >
> >
> >
> >
> > ________________________________
> > From: Wido den Hollander <wi...@widodh.nl>
> > Sent: 10 May 2022 13:59
> > To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek
> Kumar <Ab...@shapeblue.com>
> > Subject: Re: IPV6 in Isolated/VPC networks
> >
> > Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
> >> Hi all,
> >>
> >> IPv6 Support in Isolated Network and VPC with Static Routing based on
> the design doc [1] has been implemented and is available in 4.17.0 RC2. I
> hope while testing 4.17.0 RC2 you will also try to test it ?
> >> Documentation for it is available at
> http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier
> (will be available in the official docs once 4.17.0 version of docs is
> built).
> >>
> >
> > Great work!
> >
> > I see only static routing is supported. But do we publish something on
> > the message bus once a new VR/VPC is created?
> >
> > This way you could pick up these messages and have the network create a
> > (static) route based on those.
> >
> > ExaBGP for example could be used to inject such routes.
> >
> > Wido
> >
> >> [1]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> >>
> >> Regards,
> >> Abhishek
> >>
> >> ________________________________
> >> From: Rohit Yadav <ro...@shapeblue.com>
> >> Sent: 13 September 2021 14:30
> >> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >> Subject: Re: IPV6 in Isolated/VPC networks
> >>
> >> Thanks Alex, Wei. I've updated the docs here:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> >>
> >> I'll leave the thread open for futher discussion/ideas/feedback. I
> think we've completed the phase1 design doc including all feedback comments
> for adding IPv6 support in CloudStack and some initial poc/work can be
> started. My colleagues and I will keep everyone posted on this thread
> and/or on a Github PR as and when we're able to start our work on the same
> (after 4.16, potentially towards 4.17).
> >>
> >>
> >> Regards.
> >>
> >> ________________________________
> >> From: Wei ZHOU <us...@gmail.com>
> >> Sent: Friday, September 10, 2021 20:22
> >> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >> Subject: Re: IPV6 in Isolated/VPC networks
> >>
> >> Agree with Alex.
> >> We only need to know how many /64 are allocated. We do not care how many
> >> ipv6 addresses are used by VMs.
> >>
> >> -Wei
> >>
> >> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <
> Alex.Mattioli@shapeblue.com>
> >> wrote:
> >>
> >>> Hi Rohit,
> >>>
> >>> I'd go for option 2, don't see a point tracking anything smaller than a
> >>> /64 tbh.
> >>>
> >>> Cheers
> >>> Alex
> >>>
> >>>
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Rohit Yadav <ro...@shapeblue.com>
> >>> Sent: 09 September 2021 12:44
> >>> To: dev@cloudstack.apache.org
> >>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>
> >>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
> >>> agreements:
> >>>
> >>>     *   Allocate /64 for both isolated network and VPC tiers, no large
> >>> allocation of prefixes to VPC (cons: more static routing rules for
> upstream
> >>> router/admins)
> >>>     *   All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone
> has a
> >>> dedicated /64 prefix/block for systemvms
> >>>
> >>> The only outstanding question now is:
> >>>
> >>>     *   How do we manage IPv6 usage? Can anyone advise how we do IPv6
> usage
> >>> for shared network (design, implementation and use-cases?)
> >>> Option1: We don't do it, all user VMs nics have ipv4 address which
> whose
> >>> usage we don't track. For public VR/nics/networks, we can simply add
> the
> >>> IPv6 details for a related IPv4 address.
> >>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
> >>> tracking/management and usage for all VMs and systemvms nic (this means
> >>> account/domain level limits and new billing/records)
> >>> Option3: other thoughts?
> >>>
> >>>
> >>> Regards.
> >>>
> >>> ________________________________
> >>> From: Alex Mattioli <Al...@shapeblue.com>
> >>> Sent: Wednesday, September 8, 2021 23:24
> >>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >>> Subject: RE: IPV6 in Isolated/VPC networks
> >>>
> >>> Hi Rohit, Kristaps,
> >>>
> >>> I'd say option 1 as well,  it does create a bit more overhead with
> static
> >>> routes but if that's automated for a VPC it can also be easily
> automated
> >>> for several tiers of a VPC.  We also don't constrain the amount of
> tiers in
> >>> a  VPC.
> >>> It has the added advantage to be closer to the desired behaviour with
> >>> dynamic routing in the future, where a VPC VR can announce several
> subnets
> >>> upstream.
> >>>
> >>> Cheers
> >>> Alex
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Rohit Yadav <ro...@shapeblue.com>
> >>> Sent: 08 September 2021 19:04
> >>> To: dev@cloudstack.apache.org
> >>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>
> >>> Hi Kristaps,
> >>>
> >>> Thanks for sharing, I suppose that means individual tiers should be
> >>> allocated /64 instead of larger ipv6 blocks to the whole VPC which
> could
> >>> cause wastage.
> >>>
> >>> Any objection from anybody?
> >>>
> >>> Regards.
> >>> ________________________________
> >>> From: Kristaps Cudars <kr...@gmail.com>
> >>> Sent: Wednesday, September 8, 2021 9:24:01 PM
> >>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>
> >>> Hello,
> >>>
> >>> I asked networking team to comment on "How should the IPv6
> >>> block/allocation work in VPCs?"
> >>> Option1: They haven't seen lately devices with limits on how many
> static
> >>> routes can be created.
> >>> Option2: With /60 and /62 assignments and big quantity of routers IPv6
> >>> assignment from RIPE NNC can be drained fast.
> >>>
> >>> /48 contains 64000 /64
> >>> /60 contains 16 /64
> >>> 64000 / 16 = 4000 routers
> >>>
> >>>
> >>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
> >>>> All,
> >>>>
> >>>> After another iteration with Alex, I've updated the design doc. Kindly
> >>> review:
> >>>>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> >>>> +Isolated+Network+and+VPC+with+Static+Routing
> >>>>
> >>>>
> >>>> ... and advise on some outstanding questions:
> >>>>
> >>>>     *   How should the IPv6 block/allocation work in VPCs?
> >>>> Option1: Should this be simply /64 allocation on any new tier, the
> >>>> cons of this option is one static route/rule per VPC tier. (many
> >>>> upstream routers may have limit on no. of static routes?)
> >>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
> >>> (4 tiers) for the VPC, this can be filtered based on the
> vpc.max.networks
> >>> global setting (3 is default). The pros of this option are less no. of
> >>> static route/rule and easy programming, but potential wastage of
> multiple
> >>> /64 prefix blocks for unused/uncreated tiers.
> >>>>     *   How do we manage IPv6 usage? Can anyone advise how we do IPv6
> >>> usage for shared network (design, implementation and use-cases?)
> >>>> Option1: We don't do it, all user VMs nics have ipv4 address which
> whose
> >>> usage we don't track. For public VR/nics/networks, we can simply add
> the
> >>> IPv6 details for a related IPv4 address.
> >>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
> >>> tracking/management and usage for all VMs and systemvms nic (this means
> >>> account/domain level limits and new billing/records)
> >>>>     *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by
> default
> >>> if zone has a IPv6 address block allocated/assigned for use for
> systemvms
> >>> (this was mainly thought for VRs, but why no ssvm and cpvms too - any
> cons
> >>> of this?)
> >>>>     *
> >>>>
> >>>> Regards.
> >>>>
> >>>> ________________________________
> >>>> From: Rohit Yadav <ro...@shapeblue.com>
> >>>> Sent: Thursday, August 19, 2021 15:45
> >>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >>>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I've taken feedback from this thread and wrote this design doc:
> >>>>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> >>>> +Isolated+Network+and+VPC+with+Static+Routing
> >>>>
> >>>> Kindly review and advise if I missed anything or anything that needs
> to
> >>> be changed/updated. You may comment on the wiki directly too.
> >>>>
> >>>> Kindly suggest your views on the following (also in the design doc
> >>> above):
> >>>>
> >>>> Outstanding Questions:
> >>>>
> >>>>     *   Should admin or user be able to specify how VPC super CIDRs
> are
> >>> created/needed; for example a user can ask for VPC with /60 super
> CIDR? Or
> >>> should CloudStack automatically find/allocate a /64 for a new VPC tier
> from
> >>> the root-admin configured /64-/48 block?
> >>>>     *   Should we explore FRR and iBGP or other strategies to do
> dynamic
> >>> routing which may not require advance/complex configuration in the VR
> or
> >>> for the users/admin?
> >>>>     *   With SLAAC and no dhcpv6, is there a way to support secondary
> IPv6
> >>> addresses (or floating IPv6 addresses for VR/public traffic) for guest
> VM's
> >>> nics?
> >>>>     *   Any thoughts on UI/UX for firewall/routing management?
> >>>>     *   Any other feature/support for isolated network or VPC feature
> that
> >>> must be explored or supported such as PF, VPN, LB, vpc static routes,
> vpc
> >>> gateway etc.
> >>>>     *   For usage - should we have any consideration, or should we
> assume
> >>> that IPv4 and IPv6 address will go together for every nic; so IPv6
> usage
> >>> for a nic is in tandem with Ipv4 address for a nic, i.e. no
> explicit/new
> >>> biling/usage needed?
> >>>>     *   For smoketests, local dev-test should we explore ULA? Unique
> Local
> >>> Address - in the range fc00::/7. Typically only within the 'local' half
> >>> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
> >>> This prefix can be randomly generated at first install by CloudStack
> in a
> >>> zone using zoneid etc?
> >>>>     *   Should we expand support for VR diagnostic feature to include
> >>> support for ipv6, traceroute6?
> >>>>     *   Discuss - expand support/ability to allocate a /60, or /56 etc
> >>> prefix to an individual VM in shared network (Wido's suggestion)
> >>>>
> >>>>
> >>>> Regards.
> >>>>
> >>>> ________________________________
> >>>> From: Wei ZHOU <us...@gmail.com>
> >>>> Sent: Tuesday, August 17, 2021 21:16
> >>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >>>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>>
> >>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
> >>>>
> >>>> I am fine with not having stateful dhcpv6 and privacy
> >>>> extension/temporary address in phase 1. If community decides not to do
> >>>> eventually , it is also ok to me.
> >>>>
> >>>> We could explore how to better use secondary ipv6 addresses as Wido
> >>>> advised. It would be great if anyone share some user experience.
> >>>>
> >>>> -Wei
> >>>>
> >>>>
> >>>> On Tuesday, 17 August 2021, Kristaps Cudars
> >>>> <kr...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi Wei,
> >>>>>
> >>>>> Published this month's RFC 9099 and explains in different
> >>>>> words/perspective what has been written by Alex, Rohit and Wido.
> >>>>> https://www.rfc-editor.org/rfc/rfc9099.html
> >>>>>
> >>>>>
> >>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
> >>>>>> Hi Wido,
> >>>>>>
> >>>>>> (cc to Rohit and Alex)
> >>>>>>
> >>>>>> It is a good suggestion to use FRR for ipv6. The configuration is
> >>>>>> quite simple and the VMs can get SLAAC, routes, etc.
> >>>>>>
> >>>>>> Privacy extension looks not the same as what you mentioned. see
> >>>>>> https://datatracker.ietf.org/doc/html/rfc4941
> >>>>>>
> >>>>>> You are right. To use static routing, the admins need to configure
> >>>>>> the routes in the upstream router, and add some ipv6 ranges (eg
> >>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
> >>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
> >>>>>> pick up an IPv6 range
> >>>>> and
> >>>>>> assign it to an isolated network or vpc. @Rohit, correct me if I'm
> >>> wrong.
> >>>>>>
> >>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
> >>>>>> router/VR, router/dns etc via RA messages) will be the only option
> >>>>>> for now (related
> >>>>> to
> >>>>>> your pr https://github.com/apache/cloudstack/pull/3077) . Would it
> >>>>>> be
> >>>>> good
> >>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
> >>>>>> as an option in cloudstack ? The advantages are
> >>>>>> (1) support other ipv6 cidr sizes than /64.
> >>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
> >>>>>> addresses can be changed
> >>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
> >>>>>> The problem is, stateful dhcpv6 does not support
> >>>>>> routers,nameservers,
> >>>>> etc.
> >>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
> >>>>>>
> >>>>>> -Wei
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
> >
>
>
> > On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> See my inline responses:
> >>>>>>>
> >>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
> >>>>>>>> discussing
> >>>>> them
> >>>>>>> with Alex and came up with a PoC/design which can be implemented
> >>>>>>> in the following phases:
> >>>>>>>>
> >>>>>>>>      *   Phase1: implement ipv6 support in isolated networks and
> >>> VPC
> >>>>> with
> >>>>>>> static routing
> >>>>>>>>      *   Phase2: discuss and implement support for dynamic routing
> >>>>> (TBD)
> >>>>>>>>
> >>>>>>>> For Phase1 here's the high-level proposal:
> >>>>>>>>
> >>>>>>>>      *   IPv6 address management:
> >>>>>>>>         *   At the zone level root-admin specifies a /64 public
> >>> range
> >>>>> that
> >>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6 range
> >>>>>>> for
> >>>>> guest
> >>>>>>> networks (to be used by isolated networks and VPC tiers)
> >>>>>>>>         *   On creation of any IPv6 enabled isolated network or
> VPC
> >>>>> tier,
> >>>>>>> from the /48 or /56 block a /64 network is allocated/used
> >>>>>>>>         *   We assume SLAAC and autoconfiguration, no DHCPv6 in
> the
> >>>>> zone
> >>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
> >>>>>>> of
> >>>>> slaac be
> >>>>>>> explored?)
> >>>>>>>
> >>>>>>> Privacy Extensions are only a concern for client devices which
> >>>>>>> roam between different IPv6 networks.
> >>>>>>>
> >>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
> >>>>>>> based) and switches network then only the prefix (/64) will change.
> >>>>>>>
> >>>>>>> This way a network like Google, Facebook, etc could track your
> >>>>>>> device moving from network to network if they only look at the
> >>>>>>> last 64-bits of the IPv6 address.
> >>>>>>>
> >>>>>>> For servers this is not a problem as you already know in which
> >>>>>>> network they are.
> >>>>>>>
> >>>>>>>>      *   Network offerings: root-admin can create new network
> >>> offerings
> >>>>>>> (with VPC too) that specifies a network stack option:
> >>>>>>>>         *   ipv4 only (default, for backward compatibility all
> >>>>>>> networks/offerings post-upgrade migrate to this option)
> >>>>>>>>         *   ipv4-and-ipv6
> >>>>>>>>         *   ipv6-only (this can be phase 1.b)
> >>>>>>>>         *   A new routing option: static (phase1), dynamic
> (phase2,
> >>>>> with
> >>>>>>> multiple sub-options such as ospf/bgp etc...)
> >>>>>>>
> >>>>>>> This means that the network admin will need to statically route
> >>>>>>> the
> >>>>> IPv6
> >>>>>>> subnet to the VR's outside IPv6 address, for example, on a JunOS
> >>>>> router:
> >>>>>>>
> >>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
> >>>>>>> next-hop
> >>>>>>> 2001:db8:100::50
> >>>>>>>
> >>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
> >>>>>>> the outside (/64) network. In reality this will probably be a
> >>>>>>> longer address, but this is for just the example.
> >>>>>>>
> >>>>>>>>      *   VR changes:
> >>>>>>>>         *   VR gets its guest and public nics set to inet6 auto
> >>>>>>>>         *   For each /64 allocated to guest network and VPC tiers,
> >>>>> radvd
> >>>>>>> is configured to do RA
> >>>>>>>
> >>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
> >>>>>>> might already want to look into FRRouting. FRR can also
> >>>>>>> advertise RAs while not doing any routing.
> >>>>>>>
> >>>>>>> interface ens4
> >>>>>>>      no ipv6 nd suppress-ra
> >>>>>>>      ipv6 nd prefix 2001:db8:500::/64
> >>>>>>>      ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> >>>>>>>
> >>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
> >>>>>>>
> >>>>>>>>         *   Firewall: a new ipv6 zone/chain is created for ipv6
> >>> where
> >>>>> ipv6
> >>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
> >>>>>>> between
> >>>>> VPC
> >>>>>>> tiers are managed/implemented by ipv6 firewall on VR
> >>>>>>>
> >>>>>>> Please take a look at the existing security_group.py script
> >>>>>>> which implements RFC4890
> >>>>>>>
> >>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
> >>>>>>>
> >>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should always
> >>>>>>> be allowed.
> >>>>>>>
> >>>>>>>>         *   It is assumed that static routes are created on the
> >>>>> core/main
> >>>>>>> router by the admin or automated using some scripts/tools; for
> >>>>>>> this CloudStack will announce events with details of /64
> >>>>>>> networks and VR's public IPv6 address that can be consumed by a
> >>>>>>> rabbitmq/message bus
> >>>>> client
> >>>>>>> (for example), or a custom cron job or script as part of
> >>> orchestration.
> >>>>>>> (this wouldn't be necessary for dynamic routing bgp with
> >>>>>>> phase2)\\
> >>>>>>>
> >>>>>>> You would only need to announce the /48 or /56 allocated to the
> >>>>>>> VR, that's all. You don't need to inform the upstream router
> >>>>>>> about the /64 subnets created within that larger subnet.
> >>>>>>>
> >>>>>>>>      *   Guest Networking: With SLAAC, it's easy for CloudStack to
> >>>>>>> calculate allocate and use a /64 and determine the IPv6 address
> >>>>>>> of VR
> >>>>> nics
> >>>>>>> and guest VM nics
> >>>>>>>>         *   A user create an isolated network/VPC with an offering
> >>>>> that is
> >>>>>>> ipv6 enabled
> >>>>>>>>         *   A user can manage firewall for the IPv6 address/guest
> >>> nics;
> >>>>>>> there'll be no port forward and LB feature though for IPv6
> >>>>>>>>         *   A users can run workloads in the guest VMs that listen
> >>> on
> >>>>>>> publically routable ipv6 addresses
> >>>>>>>>         *   Usage/billing etc continue to work, no change needed
> >>>>>>>>
> >>>>>>>> Network layout:
> >>>>>>>>
> >>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
> >>>>>>>> VLAN] ->
> >>>>>>> [guest VMs/nics]
> >>>>>>>> *core/ISP router needs static routes to be added (manually or
> >>>>>>> automated), assumes a /48 or /56 configured for the zone
> >>>>>>>>
> >>>>>>>> Thoughts, feedback?
> >>>>>>>
> >>>>>>> Looks doable!
> >>>>>>>
> >>>>>>> Side-note: It would be very cool if you could use parts of this
> >>>>>>> implementation to also route /48, /56, or /60 subnets to
> >>>>>>> individual VMs in Shared networks.
> >>>>>>>
> >>>>>>> Why? This allows for running Docker containers with native IPv6
> >>>>>>> inside the VM or running a (Open)VPN server inside a VM which
> >>>>>>> then assigns public IPv6 addresses to clients connected.
> >>>>>>>
> >>>>>>> Instead of routing the subnet to a VR we route the subnet to a
> >>>>>>> single instance in a shared network.
> >>>>>>>
> >>>>>>> If we could then also move these subnets between Instances
> >>>>>>> easily one can quickly migrate to a different instance while
> >>>>>>> keeping the same IPv6 subnet.
> >>>>>>>
> >>>>>>> Wido
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
> >>>>>>>>
> >>>>>>>>      *   Created an isolated network and deployed a VM in my home
> >>> lab
> >>>>>>>> The VR running on KVM has following nics
> >>>>>>>> eth0 - guest network
> >>>>>>>> eth1 - link local
> >>>>>>>> eth2 - public network
> >>>>>>>>
> >>>>>>>>      *   I setup a custom openwrt router on a RPi4 to serve as a
> >>>>> toy-core
> >>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker and
> >>>>>>> I got
> >>>>> a
> >>>>>>> /48 allocated. My configuration looks like:
> >>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> >>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
> >>>>>>>>
> >>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
> >>>>> subnet/prefix
> >>>>>>> 0:
> >>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
> >>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
> >>>>>>>>      *
> >>>>>>>>      *
> >>>>>>>> In the isolated VR, I enabled ipv6 as:
> >>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
> >>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
> >>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
> >>>>>>>> net.ipv6.conf.all.autoconf = 1
> >>>>>>>>
> >>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
> >>>>>>>> configured the nics:
> >>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
> >>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
> >>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
> >>>>>>>> network without cleanup to have it
> >>>>>>> reconfigure IPv4 nics, firewall, NAT etc
> >>>>>>>>
> >>>>>>>>      *
> >>>>>>>> Next, I created a /64 network for the isolated guest network
> >>>>>>>> on eth0
> >>>>> of
> >>>>>>> VR using radvd:
> >>>>>>>>
> >>>>>>>> # cat /etc/radvd.conf
> >>>>>>>> interface eth0
> >>>>>>>> {
> >>>>>>>>        AdvSendAdvert on;
> >>>>>>>>        MinRtrAdvInterval 5;
> >>>>>>>>        MaxRtrAdvInterval 15;
> >>>>>>>>        prefix 2001:470:ed36:1::/64
> >>>>>>>>        {
> >>>>>>>>            AdvOnLink on;
> >>>>>>>>            AdvAutonomous on;
> >>>>>>>>        };
> >>>>>>>> };
> >>>>>>>> systemctl restart radvd
> >>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
> >>>>>>>> this
> >>>>>>> ...:1::/64 network
> >>>>>>>>      *   Finally I added a static route in toy core-router for the
> >>> new
> >>>>> /64
> >>>>>>> IPv6 range in the isolated network
> >>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
> >>>>>>>> eth2> dev
> >>>>>>> <local lan nic>
> >>>>>>>>      *
> >>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
> >>>>>>>> for the
> >>>>>>> new /64 network
> >>>>>>>>
> >>>>>>>> And voila all done! I create a domain AAAA record that points
> >>>>>>>> to my
> >>>>>>> guest VM IPv6 address a test webserver on
> >>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
> >>>>>>>>
> >>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
> >>>>>>>> after a
> >>>>>>> few days, sharing this solely for testing purposes)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Regards.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >
>

Re: IPV6 in Isolated/VPC networks

Posted by Abhishek Kumar <Ab...@shapeblue.com>.
Hi Wido,

I do not understand what you mean by WAB address but fd23:313a:2f53:3000:1c00:baff:fe00:4 is the public IP of the network (IPv6 of the public NIC of the network VR) in the sample.
Yes, route for fd23:313a:2f53:3cbf::/64 need to be added to this IP. fd23:313a:2f53:3cbf::/64 is guest IPv6 CIDR of the network here.

Currently, the message on event bus does not include subnet. Should that be included?
In case of VPCs, there could be multiple tiers which will need multiple routes to be added. Will that be an issue if we include current network/tier subnet in the event message?

Regards,
Abhishek
________________________________
From: Wido den Hollander <wi...@widodh.nl>
Sent: 10 May 2022 19:01
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek Kumar <Ab...@shapeblue.com>
Subject: Re: IPV6 in Isolated/VPC networks

Hi,

Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
> Yes. When a public IPv6 is assigned or released, CloudStack will publish event with type NET.IP6ASSIGN, NET.IP6RELEASE.
> These event notifications can be tracked. And with improvements in events framework, these event messages will have network uuid as entityuuid and Network as entitytype. Using this network can be queried using to list IPv6 routes that need to be added.
>
> Sample event message,
>
> {"eventDateTime":"2022-05-10 09:32:12 +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":"14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned public IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID: 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":"bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-af19-1e00320001f3","entity":"Network","status":"Completed"}
>
>
> ?Sample API call,
>> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98 filter=id,name,ip6routes
> {
>    "count": 1,
>    "network": [
>      {
>        "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
>        "ip6routes": [
>          {
>            "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
>            "subnet": "fd23:313a:2f53:3cbf::/64"
>          }

Looking at this JSON, does this mean that
fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?

And that I would need to (statically) route fd23:313a:2f53:3cbf::/64 to
that IP?

The event message does not include the subnet, that makes it a bit more
difficult as you would then also need to do a API-call to gather that
information.

Wido

P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an
AAAA-record for IPv6!

>        ],
>        "name": "routing_test"
>      }
>    ]
> }
>
>
>
>
> ________________________________
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: 10 May 2022 13:59
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek Kumar <Ab...@shapeblue.com>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
>> Hi all,
>>
>> IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
>> Documentation for it is available at http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier (will be available in the official docs once 4.17.0 version of docs is built).
>>
>
> Great work!
>
> I see only static routing is supported. But do we publish something on
> the message bus once a new VR/VPC is created?
>
> This way you could pick up these messages and have the network create a
> (static) route based on those.
>
> ExaBGP for example could be used to inject such routes.
>
> Wido
>
>> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
>>
>> Regards,
>> Abhishek
>>
>> ________________________________
>> From: Rohit Yadav <ro...@shapeblue.com>
>> Sent: 13 September 2021 14:30
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Thanks Alex, Wei. I've updated the docs here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
>>
>> I'll leave the thread open for futher discussion/ideas/feedback. I think we've completed the phase1 design doc including all feedback comments for adding IPv6 support in CloudStack and some initial poc/work can be started. My colleagues and I will keep everyone posted on this thread and/or on a Github PR as and when we're able to start our work on the same (after 4.16, potentially towards 4.17).
>>
>>
>> Regards.
>>
>> ________________________________
>> From: Wei ZHOU <us...@gmail.com>
>> Sent: Friday, September 10, 2021 20:22
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Agree with Alex.
>> We only need to know how many /64 are allocated. We do not care how many
>> ipv6 addresses are used by VMs.
>>
>> -Wei
>>
>> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <Al...@shapeblue.com>
>> wrote:
>>
>>> Hi Rohit,
>>>
>>> I'd go for option 2, don't see a point tracking anything smaller than a
>>> /64 tbh.
>>>
>>> Cheers
>>> Alex
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Rohit Yadav <ro...@shapeblue.com>
>>> Sent: 09 September 2021 12:44
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>>> agreements:
>>>
>>>     *   Allocate /64 for both isolated network and VPC tiers, no large
>>> allocation of prefixes to VPC (cons: more static routing rules for upstream
>>> router/admins)
>>>     *   All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a
>>> dedicated /64 prefix/block for systemvms
>>>
>>> The only outstanding question now is:
>>>
>>>     *   How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage
>>> for shared network (design, implementation and use-cases?)
>>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>>> usage we don't track. For public VR/nics/networks, we can simply add the
>>> IPv6 details for a related IPv4 address.
>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>>> tracking/management and usage for all VMs and systemvms nic (this means
>>> account/domain level limits and new billing/records)
>>> Option3: other thoughts?
>>>
>>>
>>> Regards.
>>>
>>> ________________________________
>>> From: Alex Mattioli <Al...@shapeblue.com>
>>> Sent: Wednesday, September 8, 2021 23:24
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: RE: IPV6 in Isolated/VPC networks
>>>
>>> Hi Rohit, Kristaps,
>>>
>>> I'd say option 1 as well,  it does create a bit more overhead with static
>>> routes but if that's automated for a VPC it can also be easily automated
>>> for several tiers of a VPC.  We also don't constrain the amount of tiers in
>>> a  VPC.
>>> It has the added advantage to be closer to the desired behaviour with
>>> dynamic routing in the future, where a VPC VR can announce several subnets
>>> upstream.
>>>
>>> Cheers
>>> Alex
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Rohit Yadav <ro...@shapeblue.com>
>>> Sent: 08 September 2021 19:04
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Hi Kristaps,
>>>
>>> Thanks for sharing, I suppose that means individual tiers should be
>>> allocated /64 instead of larger ipv6 blocks to the whole VPC which could
>>> cause wastage.
>>>
>>> Any objection from anybody?
>>>
>>> Regards.
>>> ________________________________
>>> From: Kristaps Cudars <kr...@gmail.com>
>>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Hello,
>>>
>>> I asked networking team to comment on "How should the IPv6
>>> block/allocation work in VPCs?"
>>> Option1: They haven't seen lately devices with limits on how many static
>>> routes can be created.
>>> Option2: With /60 and /62 assignments and big quantity of routers IPv6
>>> assignment from RIPE NNC can be drained fast.
>>>
>>> /48 contains 64000 /64
>>> /60 contains 16 /64
>>> 64000 / 16 = 4000 routers
>>>
>>>
>>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
>>>> All,
>>>>
>>>> After another iteration with Alex, I've updated the design doc. Kindly
>>> review:
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>
>>>>
>>>> ... and advise on some outstanding questions:
>>>>
>>>>     *   How should the IPv6 block/allocation work in VPCs?
>>>> Option1: Should this be simply /64 allocation on any new tier, the
>>>> cons of this option is one static route/rule per VPC tier. (many
>>>> upstream routers may have limit on no. of static routes?)
>>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
>>> (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks
>>> global setting (3 is default). The pros of this option are less no. of
>>> static route/rule and easy programming, but potential wastage of multiple
>>> /64 prefix blocks for unused/uncreated tiers.
>>>>     *   How do we manage IPv6 usage? Can anyone advise how we do IPv6
>>> usage for shared network (design, implementation and use-cases?)
>>>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>>> usage we don't track. For public VR/nics/networks, we can simply add the
>>> IPv6 details for a related IPv4 address.
>>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>>> tracking/management and usage for all VMs and systemvms nic (this means
>>> account/domain level limits and new billing/records)
>>>>     *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by default
>>> if zone has a IPv6 address block allocated/assigned for use for systemvms
>>> (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons
>>> of this?)
>>>>     *
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>> Sent: Thursday, August 19, 2021 15:45
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Hi all,
>>>>
>>>> I've taken feedback from this thread and wrote this design doc:
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>
>>>> Kindly review and advise if I missed anything or anything that needs to
>>> be changed/updated. You may comment on the wiki directly too.
>>>>
>>>> Kindly suggest your views on the following (also in the design doc
>>> above):
>>>>
>>>> Outstanding Questions:
>>>>
>>>>     *   Should admin or user be able to specify how VPC super CIDRs are
>>> created/needed; for example a user can ask for VPC with /60 super CIDR? Or
>>> should CloudStack automatically find/allocate a /64 for a new VPC tier from
>>> the root-admin configured /64-/48 block?
>>>>     *   Should we explore FRR and iBGP or other strategies to do dynamic
>>> routing which may not require advance/complex configuration in the VR or
>>> for the users/admin?
>>>>     *   With SLAAC and no dhcpv6, is there a way to support secondary IPv6
>>> addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's
>>> nics?
>>>>     *   Any thoughts on UI/UX for firewall/routing management?
>>>>     *   Any other feature/support for isolated network or VPC feature that
>>> must be explored or supported such as PF, VPN, LB, vpc static routes, vpc
>>> gateway etc.
>>>>     *   For usage - should we have any consideration, or should we assume
>>> that IPv4 and IPv6 address will go together for every nic; so IPv6 usage
>>> for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new
>>> biling/usage needed?
>>>>     *   For smoketests, local dev-test should we explore ULA? Unique Local
>>> Address - in the range fc00::/7. Typically only within the 'local' half
>>> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
>>> This prefix can be randomly generated at first install by CloudStack in a
>>> zone using zoneid etc?
>>>>     *   Should we expand support for VR diagnostic feature to include
>>> support for ipv6, traceroute6?
>>>>     *   Discuss - expand support/ability to allocate a /60, or /56 etc
>>> prefix to an individual VM in shared network (Wido's suggestion)
>>>>
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Wei ZHOU <us...@gmail.com>
>>>> Sent: Tuesday, August 17, 2021 21:16
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>>
>>>> I am fine with not having stateful dhcpv6 and privacy
>>>> extension/temporary address in phase 1. If community decides not to do
>>>> eventually , it is also ok to me.
>>>>
>>>> We could explore how to better use secondary ipv6 addresses as Wido
>>>> advised. It would be great if anyone share some user experience.
>>>>
>>>> -Wei
>>>>
>>>>
>>>> On Tuesday, 17 August 2021, Kristaps Cudars
>>>> <kr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Wei,
>>>>>
>>>>> Published this month's RFC 9099 and explains in different
>>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>>> https://www.rfc-editor.org/rfc/rfc9099.html
>>>>>
>>>>>
>>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
>>>>>> Hi Wido,
>>>>>>
>>>>>> (cc to Rohit and Alex)
>>>>>>
>>>>>> It is a good suggestion to use FRR for ipv6. The configuration is
>>>>>> quite simple and the VMs can get SLAAC, routes, etc.
>>>>>>
>>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>>> https://datatracker.ietf.org/doc/html/rfc4941
>>>>>>
>>>>>> You are right. To use static routing, the admins need to configure
>>>>>> the routes in the upstream router, and add some ipv6 ranges (eg
>>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
>>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
>>>>>> pick up an IPv6 range
>>>>> and
>>>>>> assign it to an isolated network or vpc. @Rohit, correct me if I'm
>>> wrong.
>>>>>>
>>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
>>>>>> router/VR, router/dns etc via RA messages) will be the only option
>>>>>> for now (related
>>>>> to
>>>>>> your pr https://github.com/apache/cloudstack/pull/3077) . Would it
>>>>>> be
>>>>> good
>>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
>>>>>> as an option in cloudstack ? The advantages are
>>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
>>>>>> addresses can be changed
>>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>>> The problem is, stateful dhcpv6 does not support
>>>>>> routers,nameservers,
>>>>> etc.
>>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>>
>>>>>> -Wei
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>>
>
>
>
 

> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> See my inline responses:
>>>>>>>
>>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
>>>>>>>> discussing
>>>>> them
>>>>>>> with Alex and came up with a PoC/design which can be implemented
>>>>>>> in the following phases:
>>>>>>>>
>>>>>>>>      *   Phase1: implement ipv6 support in isolated networks and
>>> VPC
>>>>> with
>>>>>>> static routing
>>>>>>>>      *   Phase2: discuss and implement support for dynamic routing
>>>>> (TBD)
>>>>>>>>
>>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>>
>>>>>>>>      *   IPv6 address management:
>>>>>>>>         *   At the zone level root-admin specifies a /64 public
>>> range
>>>>> that
>>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6 range
>>>>>>> for
>>>>> guest
>>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>>>         *   On creation of any IPv6 enabled isolated network or VPC
>>>>> tier,
>>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>>>         *   We assume SLAAC and autoconfiguration, no DHCPv6 in the
>>>>> zone
>>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
>>>>>>> of
>>>>> slaac be
>>>>>>> explored?)
>>>>>>>
>>>>>>> Privacy Extensions are only a concern for client devices which
>>>>>>> roam between different IPv6 networks.
>>>>>>>
>>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>>
>>>>>>> This way a network like Google, Facebook, etc could track your
>>>>>>> device moving from network to network if they only look at the
>>>>>>> last 64-bits of the IPv6 address.
>>>>>>>
>>>>>>> For servers this is not a problem as you already know in which
>>>>>>> network they are.
>>>>>>>
>>>>>>>>      *   Network offerings: root-admin can create new network
>>> offerings
>>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>>>         *   ipv4 only (default, for backward compatibility all
>>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>>>         *   ipv4-and-ipv6
>>>>>>>>         *   ipv6-only (this can be phase 1.b)
>>>>>>>>         *   A new routing option: static (phase1), dynamic (phase2,
>>>>> with
>>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>>
>>>>>>> This means that the network admin will need to statically route
>>>>>>> the
>>>>> IPv6
>>>>>>> subnet to the VR's outside IPv6 address, for example, on a JunOS
>>>>> router:
>>>>>>>
>>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
>>>>>>> next-hop
>>>>>>> 2001:db8:100::50
>>>>>>>
>>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
>>>>>>> the outside (/64) network. In reality this will probably be a
>>>>>>> longer address, but this is for just the example.
>>>>>>>
>>>>>>>>      *   VR changes:
>>>>>>>>         *   VR gets its guest and public nics set to inet6 auto
>>>>>>>>         *   For each /64 allocated to guest network and VPC tiers,
>>>>> radvd
>>>>>>> is configured to do RA
>>>>>>>
>>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
>>>>>>> might already want to look into FRRouting. FRR can also
>>>>>>> advertise RAs while not doing any routing.
>>>>>>>
>>>>>>> interface ens4
>>>>>>>      no ipv6 nd suppress-ra
>>>>>>>      ipv6 nd prefix 2001:db8:500::/64
>>>>>>>      ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>>
>>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
>>>>>>>
>>>>>>>>         *   Firewall: a new ipv6 zone/chain is created for ipv6
>>> where
>>>>> ipv6
>>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
>>>>>>> between
>>>>> VPC
>>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>>
>>>>>>> Please take a look at the existing security_group.py script
>>>>>>> which implements RFC4890
>>>>>>>
>>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
>>>>>>>
>>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should always
>>>>>>> be allowed.
>>>>>>>
>>>>>>>>         *   It is assumed that static routes are created on the
>>>>> core/main
>>>>>>> router by the admin or automated using some scripts/tools; for
>>>>>>> this CloudStack will announce events with details of /64
>>>>>>> networks and VR's public IPv6 address that can be consumed by a
>>>>>>> rabbitmq/message bus
>>>>> client
>>>>>>> (for example), or a custom cron job or script as part of
>>> orchestration.
>>>>>>> (this wouldn't be necessary for dynamic routing bgp with
>>>>>>> phase2)\\
>>>>>>>
>>>>>>> You would only need to announce the /48 or /56 allocated to the
>>>>>>> VR, that's all. You don't need to inform the upstream router
>>>>>>> about the /64 subnets created within that larger subnet.
>>>>>>>
>>>>>>>>      *   Guest Networking: With SLAAC, it's easy for CloudStack to
>>>>>>> calculate allocate and use a /64 and determine the IPv6 address
>>>>>>> of VR
>>>>> nics
>>>>>>> and guest VM nics
>>>>>>>>         *   A user create an isolated network/VPC with an offering
>>>>> that is
>>>>>>> ipv6 enabled
>>>>>>>>         *   A user can manage firewall for the IPv6 address/guest
>>> nics;
>>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>>>         *   A users can run workloads in the guest VMs that listen
>>> on
>>>>>>> publically routable ipv6 addresses
>>>>>>>>         *   Usage/billing etc continue to work, no change needed
>>>>>>>>
>>>>>>>> Network layout:
>>>>>>>>
>>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
>>>>>>>> VLAN] ->
>>>>>>> [guest VMs/nics]
>>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>>
>>>>>>>> Thoughts, feedback?
>>>>>>>
>>>>>>> Looks doable!
>>>>>>>
>>>>>>> Side-note: It would be very cool if you could use parts of this
>>>>>>> implementation to also route /48, /56, or /60 subnets to
>>>>>>> individual VMs in Shared networks.
>>>>>>>
>>>>>>> Why? This allows for running Docker containers with native IPv6
>>>>>>> inside the VM or running a (Open)VPN server inside a VM which
>>>>>>> then assigns public IPv6 addresses to clients connected.
>>>>>>>
>>>>>>> Instead of routing the subnet to a VR we route the subnet to a
>>>>>>> single instance in a shared network.
>>>>>>>
>>>>>>> If we could then also move these subnets between Instances
>>>>>>> easily one can quickly migrate to a different instance while
>>>>>>> keeping the same IPv6 subnet.
>>>>>>>
>>>>>>> Wido
>>>>>>>
>>>>>>>>
>>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>>
>>>>>>>>      *   Created an isolated network and deployed a VM in my home
>>> lab
>>>>>>>> The VR running on KVM has following nics
>>>>>>>> eth0 - guest network
>>>>>>>> eth1 - link local
>>>>>>>> eth2 - public network
>>>>>>>>
>>>>>>>>      *   I setup a custom openwrt router on a RPi4 to serve as a
>>>>> toy-core
>>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker and
>>>>>>> I got
>>>>> a
>>>>>>> /48 allocated. My configuration looks like:
>>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>>
>>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>>> subnet/prefix
>>>>>>> 0:
>>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>>>      *
>>>>>>>>      *
>>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
>>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
>>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
>>>>>>>> net.ipv6.conf.all.autoconf = 1
>>>>>>>>
>>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
>>>>>>>> configured the nics:
>>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
>>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
>>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
>>>>>>>> network without cleanup to have it
>>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>>
>>>>>>>>      *
>>>>>>>> Next, I created a /64 network for the isolated guest network
>>>>>>>> on eth0
>>>>> of
>>>>>>> VR using radvd:
>>>>>>>>
>>>>>>>> # cat /etc/radvd.conf
>>>>>>>> interface eth0
>>>>>>>> {
>>>>>>>>        AdvSendAdvert on;
>>>>>>>>        MinRtrAdvInterval 5;
>>>>>>>>        MaxRtrAdvInterval 15;
>>>>>>>>        prefix 2001:470:ed36:1::/64
>>>>>>>>        {
>>>>>>>>            AdvOnLink on;
>>>>>>>>            AdvAutonomous on;
>>>>>>>>        };
>>>>>>>> };
>>>>>>>> systemctl restart radvd
>>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
>>>>>>>> this
>>>>>>> ...:1::/64 network
>>>>>>>>      *   Finally I added a static route in toy core-router for the
>>> new
>>>>> /64
>>>>>>> IPv6 range in the isolated network
>>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>>> eth2> dev
>>>>>>> <local lan nic>
>>>>>>>>      *
>>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
>>>>>>>> for the
>>>>>>> new /64 network
>>>>>>>>
>>>>>>>> And voila all done! I create a domain AAAA record that points
>>>>>>>> to my
>>>>>>> guest VM IPv6 address a test webserver on
>>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>>>>>>>>
>>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
>>>>>>>> after a
>>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>

Re: IPV6 in Isolated/VPC networks

Posted by Wei ZHOU <us...@gmail.com>.
Hi Wido,

In Abhishek's example, `fd23:313a:2f53:3000:1c00:baff:fe00:4` (gateway) is
used for communication between CloudStack VR and the upstream router.
In current implementation, it uses the same VLAN as the public IPv4 address
of the network, it is associated with the public NIC in CloudStack VR.
it can be a public or private IPv6 address. the upstream router should have
an Ipv6 in the network (e.g. upstream router has IP
fd23:313a:2f53:3000::1/64)

the guest vms are reachable when a static route is added in upstream
router, similar as `ip -6 route add <guest subnet> via <gateway>"

-Wei

On Tue, 10 May 2022 at 15:31, Wido den Hollander <wi...@widodh.nl> wrote:

> Hi,
>
> Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
> > Yes. When a public IPv6 is assigned or released, CloudStack will publish
> event with type NET.IP6ASSIGN, NET.IP6RELEASE.
> > These event notifications can be tracked. And with improvements in
> events framework, these event messages will have network uuid as entityuuid
> and Network as entitytype. Using this network can be queried using to list
> IPv6 routes that need to be added.
> >
> > Sample event message,
> >
> > {"eventDateTime":"2022-05-10 09:32:12
> +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":"14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned
> public IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID:
> 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":"bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-af19-1e00320001f3","entity":"Network","status":"Completed"}
> >
> >
> > ?Sample API call,
> >> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98
> filter=id,name,ip6routes
> > {
> >    "count": 1,
> >    "network": [
> >      {
> >        "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
> >        "ip6routes": [
> >          {
> >            "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
> >            "subnet": "fd23:313a:2f53:3cbf::/64"
> >          }
>
> Looking at this JSON, does this mean that
> fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
>
> And that I would need to (statically) route fd23:313a:2f53:3cbf::/64 to
> that IP?
>
> The event message does not include the subnet, that makes it a bit more
> difficult as you would then also need to do a API-call to gather that
> information.
>
> Wido
>
> P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an
> AAAA-record for IPv6!
>
> >        ],
> >        "name": "routing_test"
> >      }
> >    ]
> > }
> >
> >
> >
> >
> > ________________________________
> > From: Wido den Hollander <wi...@widodh.nl>
> > Sent: 10 May 2022 13:59
> > To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek
> Kumar <Ab...@shapeblue.com>
> > Subject: Re: IPV6 in Isolated/VPC networks
> >
> > Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
> >> Hi all,
> >>
> >> IPv6 Support in Isolated Network and VPC with Static Routing based on
> the design doc [1] has been implemented and is available in 4.17.0 RC2. I
> hope while testing 4.17.0 RC2 you will also try to test it ?
> >> Documentation for it is available at
> http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier
> (will be available in the official docs once 4.17.0 version of docs is
> built).
> >>
> >
> > Great work!
> >
> > I see only static routing is supported. But do we publish something on
> > the message bus once a new VR/VPC is created?
> >
> > This way you could pick up these messages and have the network create a
> > (static) route based on those.
> >
> > ExaBGP for example could be used to inject such routes.
> >
> > Wido
> >
> >> [1]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> >>
> >> Regards,
> >> Abhishek
> >>
> >> ________________________________
> >> From: Rohit Yadav <ro...@shapeblue.com>
> >> Sent: 13 September 2021 14:30
> >> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >> Subject: Re: IPV6 in Isolated/VPC networks
> >>
> >> Thanks Alex, Wei. I've updated the docs here:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> >>
> >> I'll leave the thread open for futher discussion/ideas/feedback. I
> think we've completed the phase1 design doc including all feedback comments
> for adding IPv6 support in CloudStack and some initial poc/work can be
> started. My colleagues and I will keep everyone posted on this thread
> and/or on a Github PR as and when we're able to start our work on the same
> (after 4.16, potentially towards 4.17).
> >>
> >>
> >> Regards.
> >>
> >> ________________________________
> >> From: Wei ZHOU <us...@gmail.com>
> >> Sent: Friday, September 10, 2021 20:22
> >> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >> Subject: Re: IPV6 in Isolated/VPC networks
> >>
> >> Agree with Alex.
> >> We only need to know how many /64 are allocated. We do not care how many
> >> ipv6 addresses are used by VMs.
> >>
> >> -Wei
> >>
> >> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <
> Alex.Mattioli@shapeblue.com>
> >> wrote:
> >>
> >>> Hi Rohit,
> >>>
> >>> I'd go for option 2, don't see a point tracking anything smaller than a
> >>> /64 tbh.
> >>>
> >>> Cheers
> >>> Alex
> >>>
> >>>
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Rohit Yadav <ro...@shapeblue.com>
> >>> Sent: 09 September 2021 12:44
> >>> To: dev@cloudstack.apache.org
> >>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>
> >>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
> >>> agreements:
> >>>
> >>>     *   Allocate /64 for both isolated network and VPC tiers, no large
> >>> allocation of prefixes to VPC (cons: more static routing rules for
> upstream
> >>> router/admins)
> >>>     *   All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone
> has a
> >>> dedicated /64 prefix/block for systemvms
> >>>
> >>> The only outstanding question now is:
> >>>
> >>>     *   How do we manage IPv6 usage? Can anyone advise how we do IPv6
> usage
> >>> for shared network (design, implementation and use-cases?)
> >>> Option1: We don't do it, all user VMs nics have ipv4 address which
> whose
> >>> usage we don't track. For public VR/nics/networks, we can simply add
> the
> >>> IPv6 details for a related IPv4 address.
> >>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
> >>> tracking/management and usage for all VMs and systemvms nic (this means
> >>> account/domain level limits and new billing/records)
> >>> Option3: other thoughts?
> >>>
> >>>
> >>> Regards.
> >>>
> >>> ________________________________
> >>> From: Alex Mattioli <Al...@shapeblue.com>
> >>> Sent: Wednesday, September 8, 2021 23:24
> >>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >>> Subject: RE: IPV6 in Isolated/VPC networks
> >>>
> >>> Hi Rohit, Kristaps,
> >>>
> >>> I'd say option 1 as well,  it does create a bit more overhead with
> static
> >>> routes but if that's automated for a VPC it can also be easily
> automated
> >>> for several tiers of a VPC.  We also don't constrain the amount of
> tiers in
> >>> a  VPC.
> >>> It has the added advantage to be closer to the desired behaviour with
> >>> dynamic routing in the future, where a VPC VR can announce several
> subnets
> >>> upstream.
> >>>
> >>> Cheers
> >>> Alex
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Rohit Yadav <ro...@shapeblue.com>
> >>> Sent: 08 September 2021 19:04
> >>> To: dev@cloudstack.apache.org
> >>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>
> >>> Hi Kristaps,
> >>>
> >>> Thanks for sharing, I suppose that means individual tiers should be
> >>> allocated /64 instead of larger ipv6 blocks to the whole VPC which
> could
> >>> cause wastage.
> >>>
> >>> Any objection from anybody?
> >>>
> >>> Regards.
> >>> ________________________________
> >>> From: Kristaps Cudars <kr...@gmail.com>
> >>> Sent: Wednesday, September 8, 2021 9:24:01 PM
> >>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>
> >>> Hello,
> >>>
> >>> I asked networking team to comment on "How should the IPv6
> >>> block/allocation work in VPCs?"
> >>> Option1: They haven't seen lately devices with limits on how many
> static
> >>> routes can be created.
> >>> Option2: With /60 and /62 assignments and big quantity of routers IPv6
> >>> assignment from RIPE NNC can be drained fast.
> >>>
> >>> /48 contains 64000 /64
> >>> /60 contains 16 /64
> >>> 64000 / 16 = 4000 routers
> >>>
> >>>
> >>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
> >>>> All,
> >>>>
> >>>> After another iteration with Alex, I've updated the design doc. Kindly
> >>> review:
> >>>>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> >>>> +Isolated+Network+and+VPC+with+Static+Routing
> >>>>
> >>>>
> >>>> ... and advise on some outstanding questions:
> >>>>
> >>>>     *   How should the IPv6 block/allocation work in VPCs?
> >>>> Option1: Should this be simply /64 allocation on any new tier, the
> >>>> cons of this option is one static route/rule per VPC tier. (many
> >>>> upstream routers may have limit on no. of static routes?)
> >>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
> >>> (4 tiers) for the VPC, this can be filtered based on the
> vpc.max.networks
> >>> global setting (3 is default). The pros of this option are less no. of
> >>> static route/rule and easy programming, but potential wastage of
> multiple
> >>> /64 prefix blocks for unused/uncreated tiers.
> >>>>     *   How do we manage IPv6 usage? Can anyone advise how we do IPv6
> >>> usage for shared network (design, implementation and use-cases?)
> >>>> Option1: We don't do it, all user VMs nics have ipv4 address which
> whose
> >>> usage we don't track. For public VR/nics/networks, we can simply add
> the
> >>> IPv6 details for a related IPv4 address.
> >>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
> >>> tracking/management and usage for all VMs and systemvms nic (this means
> >>> account/domain level limits and new billing/records)
> >>>>     *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by
> default
> >>> if zone has a IPv6 address block allocated/assigned for use for
> systemvms
> >>> (this was mainly thought for VRs, but why no ssvm and cpvms too - any
> cons
> >>> of this?)
> >>>>     *
> >>>>
> >>>> Regards.
> >>>>
> >>>> ________________________________
> >>>> From: Rohit Yadav <ro...@shapeblue.com>
> >>>> Sent: Thursday, August 19, 2021 15:45
> >>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >>>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I've taken feedback from this thread and wrote this design doc:
> >>>>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> >>>> +Isolated+Network+and+VPC+with+Static+Routing
> >>>>
> >>>> Kindly review and advise if I missed anything or anything that needs
> to
> >>> be changed/updated. You may comment on the wiki directly too.
> >>>>
> >>>> Kindly suggest your views on the following (also in the design doc
> >>> above):
> >>>>
> >>>> Outstanding Questions:
> >>>>
> >>>>     *   Should admin or user be able to specify how VPC super CIDRs
> are
> >>> created/needed; for example a user can ask for VPC with /60 super
> CIDR? Or
> >>> should CloudStack automatically find/allocate a /64 for a new VPC tier
> from
> >>> the root-admin configured /64-/48 block?
> >>>>     *   Should we explore FRR and iBGP or other strategies to do
> dynamic
> >>> routing which may not require advance/complex configuration in the VR
> or
> >>> for the users/admin?
> >>>>     *   With SLAAC and no dhcpv6, is there a way to support secondary
> IPv6
> >>> addresses (or floating IPv6 addresses for VR/public traffic) for guest
> VM's
> >>> nics?
> >>>>     *   Any thoughts on UI/UX for firewall/routing management?
> >>>>     *   Any other feature/support for isolated network or VPC feature
> that
> >>> must be explored or supported such as PF, VPN, LB, vpc static routes,
> vpc
> >>> gateway etc.
> >>>>     *   For usage - should we have any consideration, or should we
> assume
> >>> that IPv4 and IPv6 address will go together for every nic; so IPv6
> usage
> >>> for a nic is in tandem with Ipv4 address for a nic, i.e. no
> explicit/new
> >>> biling/usage needed?
> >>>>     *   For smoketests, local dev-test should we explore ULA? Unique
> Local
> >>> Address - in the range fc00::/7. Typically only within the 'local' half
> >>> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
> >>> This prefix can be randomly generated at first install by CloudStack
> in a
> >>> zone using zoneid etc?
> >>>>     *   Should we expand support for VR diagnostic feature to include
> >>> support for ipv6, traceroute6?
> >>>>     *   Discuss - expand support/ability to allocate a /60, or /56 etc
> >>> prefix to an individual VM in shared network (Wido's suggestion)
> >>>>
> >>>>
> >>>> Regards.
> >>>>
> >>>> ________________________________
> >>>> From: Wei ZHOU <us...@gmail.com>
> >>>> Sent: Tuesday, August 17, 2021 21:16
> >>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> >>>> Subject: Re: IPV6 in Isolated/VPC networks
> >>>>
> >>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
> >>>>
> >>>> I am fine with not having stateful dhcpv6 and privacy
> >>>> extension/temporary address in phase 1. If community decides not to do
> >>>> eventually , it is also ok to me.
> >>>>
> >>>> We could explore how to better use secondary ipv6 addresses as Wido
> >>>> advised. It would be great if anyone share some user experience.
> >>>>
> >>>> -Wei
> >>>>
> >>>>
> >>>> On Tuesday, 17 August 2021, Kristaps Cudars
> >>>> <kr...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi Wei,
> >>>>>
> >>>>> Published this month's RFC 9099 and explains in different
> >>>>> words/perspective what has been written by Alex, Rohit and Wido.
> >>>>> https://www.rfc-editor.org/rfc/rfc9099.html
> >>>>>
> >>>>>
> >>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
> >>>>>> Hi Wido,
> >>>>>>
> >>>>>> (cc to Rohit and Alex)
> >>>>>>
> >>>>>> It is a good suggestion to use FRR for ipv6. The configuration is
> >>>>>> quite simple and the VMs can get SLAAC, routes, etc.
> >>>>>>
> >>>>>> Privacy extension looks not the same as what you mentioned. see
> >>>>>> https://datatracker.ietf.org/doc/html/rfc4941
> >>>>>>
> >>>>>> You are right. To use static routing, the admins need to configure
> >>>>>> the routes in the upstream router, and add some ipv6 ranges (eg
> >>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
> >>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
> >>>>>> pick up an IPv6 range
> >>>>> and
> >>>>>> assign it to an isolated network or vpc. @Rohit, correct me if I'm
> >>> wrong.
> >>>>>>
> >>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
> >>>>>> router/VR, router/dns etc via RA messages) will be the only option
> >>>>>> for now (related
> >>>>> to
> >>>>>> your pr https://github.com/apache/cloudstack/pull/3077) . Would it
> >>>>>> be
> >>>>> good
> >>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
> >>>>>> as an option in cloudstack ? The advantages are
> >>>>>> (1) support other ipv6 cidr sizes than /64.
> >>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
> >>>>>> addresses can be changed
> >>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
> >>>>>> The problem is, stateful dhcpv6 does not support
> >>>>>> routers,nameservers,
> >>>>> etc.
> >>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
> >>>>>>
> >>>>>> -Wei
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
> >> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl>
> wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> See my inline responses:
> >>>>>>>
> >>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
> >>>>>>>> discussing
> >>>>> them
> >>>>>>> with Alex and came up with a PoC/design which can be implemented
> >>>>>>> in the following phases:
> >>>>>>>>
> >>>>>>>>      *   Phase1: implement ipv6 support in isolated networks and
> >>> VPC
> >>>>> with
> >>>>>>> static routing
> >>>>>>>>      *   Phase2: discuss and implement support for dynamic routing
> >>>>> (TBD)
> >>>>>>>>
> >>>>>>>> For Phase1 here's the high-level proposal:
> >>>>>>>>
> >>>>>>>>      *   IPv6 address management:
> >>>>>>>>         *   At the zone level root-admin specifies a /64 public
> >>> range
> >>>>> that
> >>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6 range
> >>>>>>> for
> >>>>> guest
> >>>>>>> networks (to be used by isolated networks and VPC tiers)
> >>>>>>>>         *   On creation of any IPv6 enabled isolated network or
> VPC
> >>>>> tier,
> >>>>>>> from the /48 or /56 block a /64 network is allocated/used
> >>>>>>>>         *   We assume SLAAC and autoconfiguration, no DHCPv6 in
> the
> >>>>> zone
> >>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
> >>>>>>> of
> >>>>> slaac be
> >>>>>>> explored?)
> >>>>>>>
> >>>>>>> Privacy Extensions are only a concern for client devices which
> >>>>>>> roam between different IPv6 networks.
> >>>>>>>
> >>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
> >>>>>>> based) and switches network then only the prefix (/64) will change.
> >>>>>>>
> >>>>>>> This way a network like Google, Facebook, etc could track your
> >>>>>>> device moving from network to network if they only look at the
> >>>>>>> last 64-bits of the IPv6 address.
> >>>>>>>
> >>>>>>> For servers this is not a problem as you already know in which
> >>>>>>> network they are.
> >>>>>>>
> >>>>>>>>      *   Network offerings: root-admin can create new network
> >>> offerings
> >>>>>>> (with VPC too) that specifies a network stack option:
> >>>>>>>>         *   ipv4 only (default, for backward compatibility all
> >>>>>>> networks/offerings post-upgrade migrate to this option)
> >>>>>>>>         *   ipv4-and-ipv6
> >>>>>>>>         *   ipv6-only (this can be phase 1.b)
> >>>>>>>>         *   A new routing option: static (phase1), dynamic
> (phase2,
> >>>>> with
> >>>>>>> multiple sub-options such as ospf/bgp etc...)
> >>>>>>>
> >>>>>>> This means that the network admin will need to statically route
> >>>>>>> the
> >>>>> IPv6
> >>>>>>> subnet to the VR's outside IPv6 address, for example, on a JunOS
> >>>>> router:
> >>>>>>>
> >>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
> >>>>>>> next-hop
> >>>>>>> 2001:db8:100::50
> >>>>>>>
> >>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
> >>>>>>> the outside (/64) network. In reality this will probably be a
> >>>>>>> longer address, but this is for just the example.
> >>>>>>>
> >>>>>>>>      *   VR changes:
> >>>>>>>>         *   VR gets its guest and public nics set to inet6 auto
> >>>>>>>>         *   For each /64 allocated to guest network and VPC tiers,
> >>>>> radvd
> >>>>>>> is configured to do RA
> >>>>>>>
> >>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
> >>>>>>> might already want to look into FRRouting. FRR can also
> >>>>>>> advertise RAs while not doing any routing.
> >>>>>>>
> >>>>>>> interface ens4
> >>>>>>>      no ipv6 nd suppress-ra
> >>>>>>>      ipv6 nd prefix 2001:db8:500::/64
> >>>>>>>      ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> >>>>>>>
> >>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
> >>>>>>>
> >>>>>>>>         *   Firewall: a new ipv6 zone/chain is created for ipv6
> >>> where
> >>>>> ipv6
> >>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
> >>>>>>> between
> >>>>> VPC
> >>>>>>> tiers are managed/implemented by ipv6 firewall on VR
> >>>>>>>
> >>>>>>> Please take a look at the existing security_group.py script
> >>>>>>> which implements RFC4890
> >>>>>>>
> >>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
> >>>>>>>
> >>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should always
> >>>>>>> be allowed.
> >>>>>>>
> >>>>>>>>         *   It is assumed that static routes are created on the
> >>>>> core/main
> >>>>>>> router by the admin or automated using some scripts/tools; for
> >>>>>>> this CloudStack will announce events with details of /64
> >>>>>>> networks and VR's public IPv6 address that can be consumed by a
> >>>>>>> rabbitmq/message bus
> >>>>> client
> >>>>>>> (for example), or a custom cron job or script as part of
> >>> orchestration.
> >>>>>>> (this wouldn't be necessary for dynamic routing bgp with
> >>>>>>> phase2)\\
> >>>>>>>
> >>>>>>> You would only need to announce the /48 or /56 allocated to the
> >>>>>>> VR, that's all. You don't need to inform the upstream router
> >>>>>>> about the /64 subnets created within that larger subnet.
> >>>>>>>
> >>>>>>>>      *   Guest Networking: With SLAAC, it's easy for CloudStack to
> >>>>>>> calculate allocate and use a /64 and determine the IPv6 address
> >>>>>>> of VR
> >>>>> nics
> >>>>>>> and guest VM nics
> >>>>>>>>         *   A user create an isolated network/VPC with an offering
> >>>>> that is
> >>>>>>> ipv6 enabled
> >>>>>>>>         *   A user can manage firewall for the IPv6 address/guest
> >>> nics;
> >>>>>>> there'll be no port forward and LB feature though for IPv6
> >>>>>>>>         *   A users can run workloads in the guest VMs that listen
> >>> on
> >>>>>>> publically routable ipv6 addresses
> >>>>>>>>         *   Usage/billing etc continue to work, no change needed
> >>>>>>>>
> >>>>>>>> Network layout:
> >>>>>>>>
> >>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
> >>>>>>>> VLAN] ->
> >>>>>>> [guest VMs/nics]
> >>>>>>>> *core/ISP router needs static routes to be added (manually or
> >>>>>>> automated), assumes a /48 or /56 configured for the zone
> >>>>>>>>
> >>>>>>>> Thoughts, feedback?
> >>>>>>>
> >>>>>>> Looks doable!
> >>>>>>>
> >>>>>>> Side-note: It would be very cool if you could use parts of this
> >>>>>>> implementation to also route /48, /56, or /60 subnets to
> >>>>>>> individual VMs in Shared networks.
> >>>>>>>
> >>>>>>> Why? This allows for running Docker containers with native IPv6
> >>>>>>> inside the VM or running a (Open)VPN server inside a VM which
> >>>>>>> then assigns public IPv6 addresses to clients connected.
> >>>>>>>
> >>>>>>> Instead of routing the subnet to a VR we route the subnet to a
> >>>>>>> single instance in a shared network.
> >>>>>>>
> >>>>>>> If we could then also move these subnets between Instances
> >>>>>>> easily one can quickly migrate to a different instance while
> >>>>>>> keeping the same IPv6 subnet.
> >>>>>>>
> >>>>>>> Wido
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
> >>>>>>>>
> >>>>>>>>      *   Created an isolated network and deployed a VM in my home
> >>> lab
> >>>>>>>> The VR running on KVM has following nics
> >>>>>>>> eth0 - guest network
> >>>>>>>> eth1 - link local
> >>>>>>>> eth2 - public network
> >>>>>>>>
> >>>>>>>>      *   I setup a custom openwrt router on a RPi4 to serve as a
> >>>>> toy-core
> >>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker and
> >>>>>>> I got
> >>>>> a
> >>>>>>> /48 allocated. My configuration looks like:
> >>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> >>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
> >>>>>>>>
> >>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
> >>>>> subnet/prefix
> >>>>>>> 0:
> >>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
> >>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
> >>>>>>>>      *
> >>>>>>>>      *
> >>>>>>>> In the isolated VR, I enabled ipv6 as:
> >>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
> >>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
> >>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
> >>>>>>>> net.ipv6.conf.all.autoconf = 1
> >>>>>>>>
> >>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
> >>>>>>>> configured the nics:
> >>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
> >>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
> >>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
> >>>>>>>> network without cleanup to have it
> >>>>>>> reconfigure IPv4 nics, firewall, NAT etc
> >>>>>>>>
> >>>>>>>>      *
> >>>>>>>> Next, I created a /64 network for the isolated guest network
> >>>>>>>> on eth0
> >>>>> of
> >>>>>>> VR using radvd:
> >>>>>>>>
> >>>>>>>> # cat /etc/radvd.conf
> >>>>>>>> interface eth0
> >>>>>>>> {
> >>>>>>>>        AdvSendAdvert on;
> >>>>>>>>        MinRtrAdvInterval 5;
> >>>>>>>>        MaxRtrAdvInterval 15;
> >>>>>>>>        prefix 2001:470:ed36:1::/64
> >>>>>>>>        {
> >>>>>>>>            AdvOnLink on;
> >>>>>>>>            AdvAutonomous on;
> >>>>>>>>        };
> >>>>>>>> };
> >>>>>>>> systemctl restart radvd
> >>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
> >>>>>>>> this
> >>>>>>> ...:1::/64 network
> >>>>>>>>      *   Finally I added a static route in toy core-router for the
> >>> new
> >>>>> /64
> >>>>>>> IPv6 range in the isolated network
> >>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
> >>>>>>>> eth2> dev
> >>>>>>> <local lan nic>
> >>>>>>>>      *
> >>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
> >>>>>>>> for the
> >>>>>>> new /64 network
> >>>>>>>>
> >>>>>>>> And voila all done! I create a domain AAAA record that points
> >>>>>>>> to my
> >>>>>>> guest VM IPv6 address a test webserver on
> >>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
> >>>>>>>>
> >>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
> >>>>>>>> after a
> >>>>>>> few days, sharing this solely for testing purposes)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Regards.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >
>

Re: IPV6 in Isolated/VPC networks

Posted by Wido den Hollander <wi...@widodh.nl>.
Hi,

Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
> Yes. When a public IPv6 is assigned or released, CloudStack will publish event with type NET.IP6ASSIGN, NET.IP6RELEASE.
> These event notifications can be tracked. And with improvements in events framework, these event messages will have network uuid as entityuuid and Network as entitytype. Using this network can be queried using to list IPv6 routes that need to be added.
> 
> Sample event message,
> 
> {"eventDateTime":"2022-05-10 09:32:12 +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":"14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned public IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID: 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":"bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-af19-1e00320001f3","entity":"Network","status":"Completed"}
> 
> 
> ?Sample API call,
>> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98 filter=id,name,ip6routes
> {
>    "count": 1,
>    "network": [
>      {
>        "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
>        "ip6routes": [
>          {
>            "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
>            "subnet": "fd23:313a:2f53:3cbf::/64"
>          }

Looking at this JSON, does this mean that 
fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?

And that I would need to (statically) route fd23:313a:2f53:3cbf::/64 to 
that IP?

The event message does not include the subnet, that makes it a bit more 
difficult as you would then also need to do a API-call to gather that 
information.

Wido

P.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an 
AAAA-record for IPv6!

>        ],
>        "name": "routing_test"
>      }
>    ]
> }
> 
> 
> 
> 
> ________________________________
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: 10 May 2022 13:59
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek Kumar <Ab...@shapeblue.com>
> Subject: Re: IPV6 in Isolated/VPC networks
> 
> Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
>> Hi all,
>>
>> IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
>> Documentation for it is available at http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier (will be available in the official docs once 4.17.0 version of docs is built).
>>
> 
> Great work!
> 
> I see only static routing is supported. But do we publish something on
> the message bus once a new VR/VPC is created?
> 
> This way you could pick up these messages and have the network create a
> (static) route based on those.
> 
> ExaBGP for example could be used to inject such routes.
> 
> Wido
> 
>> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
>>
>> Regards,
>> Abhishek
>>
>> ________________________________
>> From: Rohit Yadav <ro...@shapeblue.com>
>> Sent: 13 September 2021 14:30
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Thanks Alex, Wei. I've updated the docs here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
>>
>> I'll leave the thread open for futher discussion/ideas/feedback. I think we've completed the phase1 design doc including all feedback comments for adding IPv6 support in CloudStack and some initial poc/work can be started. My colleagues and I will keep everyone posted on this thread and/or on a Github PR as and when we're able to start our work on the same (after 4.16, potentially towards 4.17).
>>
>>
>> Regards.
>>
>> ________________________________
>> From: Wei ZHOU <us...@gmail.com>
>> Sent: Friday, September 10, 2021 20:22
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Agree with Alex.
>> We only need to know how many /64 are allocated. We do not care how many
>> ipv6 addresses are used by VMs.
>>
>> -Wei
>>
>> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <Al...@shapeblue.com>
>> wrote:
>>
>>> Hi Rohit,
>>>
>>> I'd go for option 2, don't see a point tracking anything smaller than a
>>> /64 tbh.
>>>
>>> Cheers
>>> Alex
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Rohit Yadav <ro...@shapeblue.com>
>>> Sent: 09 September 2021 12:44
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>>> agreements:
>>>
>>>     *   Allocate /64 for both isolated network and VPC tiers, no large
>>> allocation of prefixes to VPC (cons: more static routing rules for upstream
>>> router/admins)
>>>     *   All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a
>>> dedicated /64 prefix/block for systemvms
>>>
>>> The only outstanding question now is:
>>>
>>>     *   How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage
>>> for shared network (design, implementation and use-cases?)
>>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>>> usage we don't track. For public VR/nics/networks, we can simply add the
>>> IPv6 details for a related IPv4 address.
>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>>> tracking/management and usage for all VMs and systemvms nic (this means
>>> account/domain level limits and new billing/records)
>>> Option3: other thoughts?
>>>
>>>
>>> Regards.
>>>
>>> ________________________________
>>> From: Alex Mattioli <Al...@shapeblue.com>
>>> Sent: Wednesday, September 8, 2021 23:24
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: RE: IPV6 in Isolated/VPC networks
>>>
>>> Hi Rohit, Kristaps,
>>>
>>> I'd say option 1 as well,  it does create a bit more overhead with static
>>> routes but if that's automated for a VPC it can also be easily automated
>>> for several tiers of a VPC.  We also don't constrain the amount of tiers in
>>> a  VPC.
>>> It has the added advantage to be closer to the desired behaviour with
>>> dynamic routing in the future, where a VPC VR can announce several subnets
>>> upstream.
>>>
>>> Cheers
>>> Alex
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Rohit Yadav <ro...@shapeblue.com>
>>> Sent: 08 September 2021 19:04
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Hi Kristaps,
>>>
>>> Thanks for sharing, I suppose that means individual tiers should be
>>> allocated /64 instead of larger ipv6 blocks to the whole VPC which could
>>> cause wastage.
>>>
>>> Any objection from anybody?
>>>
>>> Regards.
>>> ________________________________
>>> From: Kristaps Cudars <kr...@gmail.com>
>>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Hello,
>>>
>>> I asked networking team to comment on "How should the IPv6
>>> block/allocation work in VPCs?"
>>> Option1: They haven't seen lately devices with limits on how many static
>>> routes can be created.
>>> Option2: With /60 and /62 assignments and big quantity of routers IPv6
>>> assignment from RIPE NNC can be drained fast.
>>>
>>> /48 contains 64000 /64
>>> /60 contains 16 /64
>>> 64000 / 16 = 4000 routers
>>>
>>>
>>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
>>>> All,
>>>>
>>>> After another iteration with Alex, I've updated the design doc. Kindly
>>> review:
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>
>>>>
>>>> ... and advise on some outstanding questions:
>>>>
>>>>     *   How should the IPv6 block/allocation work in VPCs?
>>>> Option1: Should this be simply /64 allocation on any new tier, the
>>>> cons of this option is one static route/rule per VPC tier. (many
>>>> upstream routers may have limit on no. of static routes?)
>>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
>>> (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks
>>> global setting (3 is default). The pros of this option are less no. of
>>> static route/rule and easy programming, but potential wastage of multiple
>>> /64 prefix blocks for unused/uncreated tiers.
>>>>     *   How do we manage IPv6 usage? Can anyone advise how we do IPv6
>>> usage for shared network (design, implementation and use-cases?)
>>>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>>> usage we don't track. For public VR/nics/networks, we can simply add the
>>> IPv6 details for a related IPv4 address.
>>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>>> tracking/management and usage for all VMs and systemvms nic (this means
>>> account/domain level limits and new billing/records)
>>>>     *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by default
>>> if zone has a IPv6 address block allocated/assigned for use for systemvms
>>> (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons
>>> of this?)
>>>>     *
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Rohit Yadav <ro...@shapeblue.com>
>>>> Sent: Thursday, August 19, 2021 15:45
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Hi all,
>>>>
>>>> I've taken feedback from this thread and wrote this design doc:
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
>>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>>
>>>> Kindly review and advise if I missed anything or anything that needs to
>>> be changed/updated. You may comment on the wiki directly too.
>>>>
>>>> Kindly suggest your views on the following (also in the design doc
>>> above):
>>>>
>>>> Outstanding Questions:
>>>>
>>>>     *   Should admin or user be able to specify how VPC super CIDRs are
>>> created/needed; for example a user can ask for VPC with /60 super CIDR? Or
>>> should CloudStack automatically find/allocate a /64 for a new VPC tier from
>>> the root-admin configured /64-/48 block?
>>>>     *   Should we explore FRR and iBGP or other strategies to do dynamic
>>> routing which may not require advance/complex configuration in the VR or
>>> for the users/admin?
>>>>     *   With SLAAC and no dhcpv6, is there a way to support secondary IPv6
>>> addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's
>>> nics?
>>>>     *   Any thoughts on UI/UX for firewall/routing management?
>>>>     *   Any other feature/support for isolated network or VPC feature that
>>> must be explored or supported such as PF, VPN, LB, vpc static routes, vpc
>>> gateway etc.
>>>>     *   For usage - should we have any consideration, or should we assume
>>> that IPv4 and IPv6 address will go together for every nic; so IPv6 usage
>>> for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new
>>> biling/usage needed?
>>>>     *   For smoketests, local dev-test should we explore ULA? Unique Local
>>> Address - in the range fc00::/7. Typically only within the 'local' half
>>> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
>>> This prefix can be randomly generated at first install by CloudStack in a
>>> zone using zoneid etc?
>>>>     *   Should we expand support for VR diagnostic feature to include
>>> support for ipv6, traceroute6?
>>>>     *   Discuss - expand support/ability to allocate a /60, or /56 etc
>>> prefix to an individual VM in shared network (Wido's suggestion)
>>>>
>>>>
>>>> Regards.
>>>>
>>>> ________________________________
>>>> From: Wei ZHOU <us...@gmail.com>
>>>> Sent: Tuesday, August 17, 2021 21:16
>>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>>
>>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>>
>>>> I am fine with not having stateful dhcpv6 and privacy
>>>> extension/temporary address in phase 1. If community decides not to do
>>>> eventually , it is also ok to me.
>>>>
>>>> We could explore how to better use secondary ipv6 addresses as Wido
>>>> advised. It would be great if anyone share some user experience.
>>>>
>>>> -Wei
>>>>
>>>>
>>>> On Tuesday, 17 August 2021, Kristaps Cudars
>>>> <kr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Wei,
>>>>>
>>>>> Published this month's RFC 9099 and explains in different
>>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>>> https://www.rfc-editor.org/rfc/rfc9099.html
>>>>>
>>>>>
>>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
>>>>>> Hi Wido,
>>>>>>
>>>>>> (cc to Rohit and Alex)
>>>>>>
>>>>>> It is a good suggestion to use FRR for ipv6. The configuration is
>>>>>> quite simple and the VMs can get SLAAC, routes, etc.
>>>>>>
>>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>>> https://datatracker.ietf.org/doc/html/rfc4941
>>>>>>
>>>>>> You are right. To use static routing, the admins need to configure
>>>>>> the routes in the upstream router, and add some ipv6 ranges (eg
>>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
>>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
>>>>>> pick up an IPv6 range
>>>>> and
>>>>>> assign it to an isolated network or vpc. @Rohit, correct me if I'm
>>> wrong.
>>>>>>
>>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
>>>>>> router/VR, router/dns etc via RA messages) will be the only option
>>>>>> for now (related
>>>>> to
>>>>>> your pr https://github.com/apache/cloudstack/pull/3077) . Would it
>>>>>> be
>>>>> good
>>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
>>>>>> as an option in cloudstack ? The advantages are
>>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
>>>>>> addresses can be changed
>>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>>> The problem is, stateful dhcpv6 does not support
>>>>>> routers,nameservers,
>>>>> etc.
>>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>>
>>>>>> -Wei
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>>
>   
> 
>> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> See my inline responses:
>>>>>>>
>>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
>>>>>>>> discussing
>>>>> them
>>>>>>> with Alex and came up with a PoC/design which can be implemented
>>>>>>> in the following phases:
>>>>>>>>
>>>>>>>>      *   Phase1: implement ipv6 support in isolated networks and
>>> VPC
>>>>> with
>>>>>>> static routing
>>>>>>>>      *   Phase2: discuss and implement support for dynamic routing
>>>>> (TBD)
>>>>>>>>
>>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>>
>>>>>>>>      *   IPv6 address management:
>>>>>>>>         *   At the zone level root-admin specifies a /64 public
>>> range
>>>>> that
>>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6 range
>>>>>>> for
>>>>> guest
>>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>>>         *   On creation of any IPv6 enabled isolated network or VPC
>>>>> tier,
>>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>>>         *   We assume SLAAC and autoconfiguration, no DHCPv6 in the
>>>>> zone
>>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
>>>>>>> of
>>>>> slaac be
>>>>>>> explored?)
>>>>>>>
>>>>>>> Privacy Extensions are only a concern for client devices which
>>>>>>> roam between different IPv6 networks.
>>>>>>>
>>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>>
>>>>>>> This way a network like Google, Facebook, etc could track your
>>>>>>> device moving from network to network if they only look at the
>>>>>>> last 64-bits of the IPv6 address.
>>>>>>>
>>>>>>> For servers this is not a problem as you already know in which
>>>>>>> network they are.
>>>>>>>
>>>>>>>>      *   Network offerings: root-admin can create new network
>>> offerings
>>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>>>         *   ipv4 only (default, for backward compatibility all
>>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>>>         *   ipv4-and-ipv6
>>>>>>>>         *   ipv6-only (this can be phase 1.b)
>>>>>>>>         *   A new routing option: static (phase1), dynamic (phase2,
>>>>> with
>>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>>
>>>>>>> This means that the network admin will need to statically route
>>>>>>> the
>>>>> IPv6
>>>>>>> subnet to the VR's outside IPv6 address, for example, on a JunOS
>>>>> router:
>>>>>>>
>>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
>>>>>>> next-hop
>>>>>>> 2001:db8:100::50
>>>>>>>
>>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
>>>>>>> the outside (/64) network. In reality this will probably be a
>>>>>>> longer address, but this is for just the example.
>>>>>>>
>>>>>>>>      *   VR changes:
>>>>>>>>         *   VR gets its guest and public nics set to inet6 auto
>>>>>>>>         *   For each /64 allocated to guest network and VPC tiers,
>>>>> radvd
>>>>>>> is configured to do RA
>>>>>>>
>>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
>>>>>>> might already want to look into FRRouting. FRR can also
>>>>>>> advertise RAs while not doing any routing.
>>>>>>>
>>>>>>> interface ens4
>>>>>>>      no ipv6 nd suppress-ra
>>>>>>>      ipv6 nd prefix 2001:db8:500::/64
>>>>>>>      ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>>
>>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
>>>>>>>
>>>>>>>>         *   Firewall: a new ipv6 zone/chain is created for ipv6
>>> where
>>>>> ipv6
>>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
>>>>>>> between
>>>>> VPC
>>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>>
>>>>>>> Please take a look at the existing security_group.py script
>>>>>>> which implements RFC4890
>>>>>>>
>>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
>>>>>>>
>>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should always
>>>>>>> be allowed.
>>>>>>>
>>>>>>>>         *   It is assumed that static routes are created on the
>>>>> core/main
>>>>>>> router by the admin or automated using some scripts/tools; for
>>>>>>> this CloudStack will announce events with details of /64
>>>>>>> networks and VR's public IPv6 address that can be consumed by a
>>>>>>> rabbitmq/message bus
>>>>> client
>>>>>>> (for example), or a custom cron job or script as part of
>>> orchestration.
>>>>>>> (this wouldn't be necessary for dynamic routing bgp with
>>>>>>> phase2)\\
>>>>>>>
>>>>>>> You would only need to announce the /48 or /56 allocated to the
>>>>>>> VR, that's all. You don't need to inform the upstream router
>>>>>>> about the /64 subnets created within that larger subnet.
>>>>>>>
>>>>>>>>      *   Guest Networking: With SLAAC, it's easy for CloudStack to
>>>>>>> calculate allocate and use a /64 and determine the IPv6 address
>>>>>>> of VR
>>>>> nics
>>>>>>> and guest VM nics
>>>>>>>>         *   A user create an isolated network/VPC with an offering
>>>>> that is
>>>>>>> ipv6 enabled
>>>>>>>>         *   A user can manage firewall for the IPv6 address/guest
>>> nics;
>>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>>>         *   A users can run workloads in the guest VMs that listen
>>> on
>>>>>>> publically routable ipv6 addresses
>>>>>>>>         *   Usage/billing etc continue to work, no change needed
>>>>>>>>
>>>>>>>> Network layout:
>>>>>>>>
>>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
>>>>>>>> VLAN] ->
>>>>>>> [guest VMs/nics]
>>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>>
>>>>>>>> Thoughts, feedback?
>>>>>>>
>>>>>>> Looks doable!
>>>>>>>
>>>>>>> Side-note: It would be very cool if you could use parts of this
>>>>>>> implementation to also route /48, /56, or /60 subnets to
>>>>>>> individual VMs in Shared networks.
>>>>>>>
>>>>>>> Why? This allows for running Docker containers with native IPv6
>>>>>>> inside the VM or running a (Open)VPN server inside a VM which
>>>>>>> then assigns public IPv6 addresses to clients connected.
>>>>>>>
>>>>>>> Instead of routing the subnet to a VR we route the subnet to a
>>>>>>> single instance in a shared network.
>>>>>>>
>>>>>>> If we could then also move these subnets between Instances
>>>>>>> easily one can quickly migrate to a different instance while
>>>>>>> keeping the same IPv6 subnet.
>>>>>>>
>>>>>>> Wido
>>>>>>>
>>>>>>>>
>>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>>
>>>>>>>>      *   Created an isolated network and deployed a VM in my home
>>> lab
>>>>>>>> The VR running on KVM has following nics
>>>>>>>> eth0 - guest network
>>>>>>>> eth1 - link local
>>>>>>>> eth2 - public network
>>>>>>>>
>>>>>>>>      *   I setup a custom openwrt router on a RPi4 to serve as a
>>>>> toy-core
>>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker and
>>>>>>> I got
>>>>> a
>>>>>>> /48 allocated. My configuration looks like:
>>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>>
>>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>>> subnet/prefix
>>>>>>> 0:
>>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>>>      *
>>>>>>>>      *
>>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
>>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
>>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
>>>>>>>> net.ipv6.conf.all.autoconf = 1
>>>>>>>>
>>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
>>>>>>>> configured the nics:
>>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
>>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
>>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
>>>>>>>> network without cleanup to have it
>>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>>
>>>>>>>>      *
>>>>>>>> Next, I created a /64 network for the isolated guest network
>>>>>>>> on eth0
>>>>> of
>>>>>>> VR using radvd:
>>>>>>>>
>>>>>>>> # cat /etc/radvd.conf
>>>>>>>> interface eth0
>>>>>>>> {
>>>>>>>>        AdvSendAdvert on;
>>>>>>>>        MinRtrAdvInterval 5;
>>>>>>>>        MaxRtrAdvInterval 15;
>>>>>>>>        prefix 2001:470:ed36:1::/64
>>>>>>>>        {
>>>>>>>>            AdvOnLink on;
>>>>>>>>            AdvAutonomous on;
>>>>>>>>        };
>>>>>>>> };
>>>>>>>> systemctl restart radvd
>>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
>>>>>>>> this
>>>>>>> ...:1::/64 network
>>>>>>>>      *   Finally I added a static route in toy core-router for the
>>> new
>>>>> /64
>>>>>>> IPv6 range in the isolated network
>>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>>> eth2> dev
>>>>>>> <local lan nic>
>>>>>>>>      *
>>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
>>>>>>>> for the
>>>>>>> new /64 network
>>>>>>>>
>>>>>>>> And voila all done! I create a domain AAAA record that points
>>>>>>>> to my
>>>>>>> guest VM IPv6 address a test webserver on
>>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>>>>>>>>
>>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
>>>>>>>> after a
>>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
> 

Re: IPV6 in Isolated/VPC networks

Posted by Abhishek Kumar <Ab...@shapeblue.com>.
Yes. When a public IPv6 is assigned or released, CloudStack will publish event with type NET.IP6ASSIGN, NET.IP6RELEASE.
These event notifications can be tracked. And with improvements in events framework, these event messages will have network uuid as entityuuid and Network as entitytype. Using this network can be queried using to list IPv6 routes that need to be added.

Sample event message,

{"eventDateTime":"2022-05-10 09:32:12 +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":"14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned public IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID: 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":"bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-af19-1e00320001f3","entity":"Network","status":"Completed"}


?Sample API call,
> list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98 filter=id,name,ip6routes
{
  "count": 1,
  "network": [
    {
      "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98",
      "ip6routes": [
        {
          "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4",
          "subnet": "fd23:313a:2f53:3cbf::/64"
        }
      ],
      "name": "routing_test"
    }
  ]
}




________________________________
From: Wido den Hollander <wi...@widodh.nl>
Sent: 10 May 2022 13:59
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>; Abhishek Kumar <Ab...@shapeblue.com>
Subject: Re: IPV6 in Isolated/VPC networks

Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
> Hi all,
>
> IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
> Documentation for it is available at http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier (will be available in the official docs once 4.17.0 version of docs is built).
>

Great work!

I see only static routing is supported. But do we publish something on
the message bus once a new VR/VPC is created?

This way you could pick up these messages and have the network create a
(static) route based on those.

ExaBGP for example could be used to inject such routes.

Wido

> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
>
> Regards,
> Abhishek
>
> ________________________________
> From: Rohit Yadav <ro...@shapeblue.com>
> Sent: 13 September 2021 14:30
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Thanks Alex, Wei. I've updated the docs here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
>
> I'll leave the thread open for futher discussion/ideas/feedback. I think we've completed the phase1 design doc including all feedback comments for adding IPv6 support in CloudStack and some initial poc/work can be started. My colleagues and I will keep everyone posted on this thread and/or on a Github PR as and when we're able to start our work on the same (after 4.16, potentially towards 4.17).
>
>
> Regards.
>
> ________________________________
> From: Wei ZHOU <us...@gmail.com>
> Sent: Friday, September 10, 2021 20:22
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Agree with Alex.
> We only need to know how many /64 are allocated. We do not care how many
> ipv6 addresses are used by VMs.
>
> -Wei
>
> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <Al...@shapeblue.com>
> wrote:
>
>> Hi Rohit,
>>
>> I'd go for option 2, don't see a point tracking anything smaller than a
>> /64 tbh.
>>
>> Cheers
>> Alex
>>
>>
>>
>>
>> -----Original Message-----
>> From: Rohit Yadav <ro...@shapeblue.com>
>> Sent: 09 September 2021 12:44
>> To: dev@cloudstack.apache.org
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>> agreements:
>>
>>    *   Allocate /64 for both isolated network and VPC tiers, no large
>> allocation of prefixes to VPC (cons: more static routing rules for upstream
>> router/admins)
>>    *   All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a
>> dedicated /64 prefix/block for systemvms
>>
>> The only outstanding question now is:
>>
>>    *   How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage
>> for shared network (design, implementation and use-cases?)
>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>> usage we don't track. For public VR/nics/networks, we can simply add the
>> IPv6 details for a related IPv4 address.
>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>> tracking/management and usage for all VMs and systemvms nic (this means
>> account/domain level limits and new billing/records)
>> Option3: other thoughts?
>>
>>
>> Regards.
>>
>> ________________________________
>> From: Alex Mattioli <Al...@shapeblue.com>
>> Sent: Wednesday, September 8, 2021 23:24
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: RE: IPV6 in Isolated/VPC networks
>>
>> Hi Rohit, Kristaps,
>>
>> I'd say option 1 as well,  it does create a bit more overhead with static
>> routes but if that's automated for a VPC it can also be easily automated
>> for several tiers of a VPC.  We also don't constrain the amount of tiers in
>> a  VPC.
>> It has the added advantage to be closer to the desired behaviour with
>> dynamic routing in the future, where a VPC VR can announce several subnets
>> upstream.
>>
>> Cheers
>> Alex
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Rohit Yadav <ro...@shapeblue.com>
>> Sent: 08 September 2021 19:04
>> To: dev@cloudstack.apache.org
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Hi Kristaps,
>>
>> Thanks for sharing, I suppose that means individual tiers should be
>> allocated /64 instead of larger ipv6 blocks to the whole VPC which could
>> cause wastage.
>>
>> Any objection from anybody?
>>
>> Regards.
>> ________________________________
>> From: Kristaps Cudars <kr...@gmail.com>
>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Hello,
>>
>> I asked networking team to comment on "How should the IPv6
>> block/allocation work in VPCs?"
>> Option1: They haven't seen lately devices with limits on how many static
>> routes can be created.
>> Option2: With /60 and /62 assignments and big quantity of routers IPv6
>> assignment from RIPE NNC can be drained fast.
>>
>> /48 contains 64000 /64
>> /60 contains 16 /64
>> 64000 / 16 = 4000 routers
>>
>>
>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
>>> All,
>>>
>>> After another iteration with Alex, I've updated the design doc. Kindly
>> review:
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>
>>>
>>> ... and advise on some outstanding questions:
>>>
>>>    *   How should the IPv6 block/allocation work in VPCs?
>>> Option1: Should this be simply /64 allocation on any new tier, the
>>> cons of this option is one static route/rule per VPC tier. (many
>>> upstream routers may have limit on no. of static routes?)
>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
>> (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks
>> global setting (3 is default). The pros of this option are less no. of
>> static route/rule and easy programming, but potential wastage of multiple
>> /64 prefix blocks for unused/uncreated tiers.
>>>    *   How do we manage IPv6 usage? Can anyone advise how we do IPv6
>> usage for shared network (design, implementation and use-cases?)
>>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>> usage we don't track. For public VR/nics/networks, we can simply add the
>> IPv6 details for a related IPv4 address.
>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>> tracking/management and usage for all VMs and systemvms nic (this means
>> account/domain level limits and new billing/records)
>>>    *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by default
>> if zone has a IPv6 address block allocated/assigned for use for systemvms
>> (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons
>> of this?)
>>>    *
>>>
>>> Regards.
>>>
>>> ________________________________
>>> From: Rohit Yadav <ro...@shapeblue.com>
>>> Sent: Thursday, August 19, 2021 15:45
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Hi all,
>>>
>>> I've taken feedback from this thread and wrote this design doc:
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>
>>> Kindly review and advise if I missed anything or anything that needs to
>> be changed/updated. You may comment on the wiki directly too.
>>>
>>> Kindly suggest your views on the following (also in the design doc
>> above):
>>>
>>> Outstanding Questions:
>>>
>>>    *   Should admin or user be able to specify how VPC super CIDRs are
>> created/needed; for example a user can ask for VPC with /60 super CIDR? Or
>> should CloudStack automatically find/allocate a /64 for a new VPC tier from
>> the root-admin configured /64-/48 block?
>>>    *   Should we explore FRR and iBGP or other strategies to do dynamic
>> routing which may not require advance/complex configuration in the VR or
>> for the users/admin?
>>>    *   With SLAAC and no dhcpv6, is there a way to support secondary IPv6
>> addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's
>> nics?
>>>    *   Any thoughts on UI/UX for firewall/routing management?
>>>    *   Any other feature/support for isolated network or VPC feature that
>> must be explored or supported such as PF, VPN, LB, vpc static routes, vpc
>> gateway etc.
>>>    *   For usage - should we have any consideration, or should we assume
>> that IPv4 and IPv6 address will go together for every nic; so IPv6 usage
>> for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new
>> biling/usage needed?
>>>    *   For smoketests, local dev-test should we explore ULA? Unique Local
>> Address - in the range fc00::/7. Typically only within the 'local' half
>> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
>> This prefix can be randomly generated at first install by CloudStack in a
>> zone using zoneid etc?
>>>    *   Should we expand support for VR diagnostic feature to include
>> support for ipv6, traceroute6?
>>>    *   Discuss - expand support/ability to allocate a /60, or /56 etc
>> prefix to an individual VM in shared network (Wido's suggestion)
>>>
>>>
>>> Regards.
>>>
>>> ________________________________
>>> From: Wei ZHOU <us...@gmail.com>
>>> Sent: Tuesday, August 17, 2021 21:16
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>
>>> I am fine with not having stateful dhcpv6 and privacy
>>> extension/temporary address in phase 1. If community decides not to do
>>> eventually , it is also ok to me.
>>>
>>> We could explore how to better use secondary ipv6 addresses as Wido
>>> advised. It would be great if anyone share some user experience.
>>>
>>> -Wei
>>>
>>>
>>> On Tuesday, 17 August 2021, Kristaps Cudars
>>> <kr...@gmail.com>
>>> wrote:
>>>
>>>> Hi Wei,
>>>>
>>>> Published this month's RFC 9099 and explains in different
>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>> https://www.rfc-editor.org/rfc/rfc9099.html
>>>>
>>>>
>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
>>>>> Hi Wido,
>>>>>
>>>>> (cc to Rohit and Alex)
>>>>>
>>>>> It is a good suggestion to use FRR for ipv6. The configuration is
>>>>> quite simple and the VMs can get SLAAC, routes, etc.
>>>>>
>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>> https://datatracker.ietf.org/doc/html/rfc4941
>>>>>
>>>>> You are right. To use static routing, the admins need to configure
>>>>> the routes in the upstream router, and add some ipv6 ranges (eg
>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
>>>>> pick up an IPv6 range
>>>> and
>>>>> assign it to an isolated network or vpc. @Rohit, correct me if I'm
>> wrong.
>>>>>
>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
>>>>> router/VR, router/dns etc via RA messages) will be the only option
>>>>> for now (related
>>>> to
>>>>> your pr https://github.com/apache/cloudstack/pull/3077) . Would it
>>>>> be
>>>> good
>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
>>>>> as an option in cloudstack ? The advantages are
>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
>>>>> addresses can be changed
>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>> The problem is, stateful dhcpv6 does not support
>>>>> routers,nameservers,
>>>> etc.
>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>
>>>>> -Wei
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>
>
>
>
 

> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> See my inline responses:
>>>>>>
>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
>>>>>>> discussing
>>>> them
>>>>>> with Alex and came up with a PoC/design which can be implemented
>>>>>> in the following phases:
>>>>>>>
>>>>>>>     *   Phase1: implement ipv6 support in isolated networks and
>> VPC
>>>> with
>>>>>> static routing
>>>>>>>     *   Phase2: discuss and implement support for dynamic routing
>>>> (TBD)
>>>>>>>
>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>
>>>>>>>     *   IPv6 address management:
>>>>>>>        *   At the zone level root-admin specifies a /64 public
>> range
>>>> that
>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6 range
>>>>>> for
>>>> guest
>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>>        *   On creation of any IPv6 enabled isolated network or VPC
>>>> tier,
>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>>        *   We assume SLAAC and autoconfiguration, no DHCPv6 in the
>>>> zone
>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
>>>>>> of
>>>> slaac be
>>>>>> explored?)
>>>>>>
>>>>>> Privacy Extensions are only a concern for client devices which
>>>>>> roam between different IPv6 networks.
>>>>>>
>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>
>>>>>> This way a network like Google, Facebook, etc could track your
>>>>>> device moving from network to network if they only look at the
>>>>>> last 64-bits of the IPv6 address.
>>>>>>
>>>>>> For servers this is not a problem as you already know in which
>>>>>> network they are.
>>>>>>
>>>>>>>     *   Network offerings: root-admin can create new network
>> offerings
>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>>        *   ipv4 only (default, for backward compatibility all
>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>>        *   ipv4-and-ipv6
>>>>>>>        *   ipv6-only (this can be phase 1.b)
>>>>>>>        *   A new routing option: static (phase1), dynamic (phase2,
>>>> with
>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>
>>>>>> This means that the network admin will need to statically route
>>>>>> the
>>>> IPv6
>>>>>> subnet to the VR's outside IPv6 address, for example, on a JunOS
>>>> router:
>>>>>>
>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
>>>>>> next-hop
>>>>>> 2001:db8:100::50
>>>>>>
>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
>>>>>> the outside (/64) network. In reality this will probably be a
>>>>>> longer address, but this is for just the example.
>>>>>>
>>>>>>>     *   VR changes:
>>>>>>>        *   VR gets its guest and public nics set to inet6 auto
>>>>>>>        *   For each /64 allocated to guest network and VPC tiers,
>>>> radvd
>>>>>> is configured to do RA
>>>>>>
>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
>>>>>> might already want to look into FRRouting. FRR can also
>>>>>> advertise RAs while not doing any routing.
>>>>>>
>>>>>> interface ens4
>>>>>>     no ipv6 nd suppress-ra
>>>>>>     ipv6 nd prefix 2001:db8:500::/64
>>>>>>     ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>
>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
>>>>>>
>>>>>>>        *   Firewall: a new ipv6 zone/chain is created for ipv6
>> where
>>>> ipv6
>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
>>>>>> between
>>>> VPC
>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>
>>>>>> Please take a look at the existing security_group.py script
>>>>>> which implements RFC4890
>>>>>>
>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
>>>>>>
>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should always
>>>>>> be allowed.
>>>>>>
>>>>>>>        *   It is assumed that static routes are created on the
>>>> core/main
>>>>>> router by the admin or automated using some scripts/tools; for
>>>>>> this CloudStack will announce events with details of /64
>>>>>> networks and VR's public IPv6 address that can be consumed by a
>>>>>> rabbitmq/message bus
>>>> client
>>>>>> (for example), or a custom cron job or script as part of
>> orchestration.
>>>>>> (this wouldn't be necessary for dynamic routing bgp with
>>>>>> phase2)\\
>>>>>>
>>>>>> You would only need to announce the /48 or /56 allocated to the
>>>>>> VR, that's all. You don't need to inform the upstream router
>>>>>> about the /64 subnets created within that larger subnet.
>>>>>>
>>>>>>>     *   Guest Networking: With SLAAC, it's easy for CloudStack to
>>>>>> calculate allocate and use a /64 and determine the IPv6 address
>>>>>> of VR
>>>> nics
>>>>>> and guest VM nics
>>>>>>>        *   A user create an isolated network/VPC with an offering
>>>> that is
>>>>>> ipv6 enabled
>>>>>>>        *   A user can manage firewall for the IPv6 address/guest
>> nics;
>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>>        *   A users can run workloads in the guest VMs that listen
>> on
>>>>>> publically routable ipv6 addresses
>>>>>>>        *   Usage/billing etc continue to work, no change needed
>>>>>>>
>>>>>>> Network layout:
>>>>>>>
>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
>>>>>>> VLAN] ->
>>>>>> [guest VMs/nics]
>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>
>>>>>>> Thoughts, feedback?
>>>>>>
>>>>>> Looks doable!
>>>>>>
>>>>>> Side-note: It would be very cool if you could use parts of this
>>>>>> implementation to also route /48, /56, or /60 subnets to
>>>>>> individual VMs in Shared networks.
>>>>>>
>>>>>> Why? This allows for running Docker containers with native IPv6
>>>>>> inside the VM or running a (Open)VPN server inside a VM which
>>>>>> then assigns public IPv6 addresses to clients connected.
>>>>>>
>>>>>> Instead of routing the subnet to a VR we route the subnet to a
>>>>>> single instance in a shared network.
>>>>>>
>>>>>> If we could then also move these subnets between Instances
>>>>>> easily one can quickly migrate to a different instance while
>>>>>> keeping the same IPv6 subnet.
>>>>>>
>>>>>> Wido
>>>>>>
>>>>>>>
>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>
>>>>>>>     *   Created an isolated network and deployed a VM in my home
>> lab
>>>>>>> The VR running on KVM has following nics
>>>>>>> eth0 - guest network
>>>>>>> eth1 - link local
>>>>>>> eth2 - public network
>>>>>>>
>>>>>>>     *   I setup a custom openwrt router on a RPi4 to serve as a
>>>> toy-core
>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker and
>>>>>> I got
>>>> a
>>>>>> /48 allocated. My configuration looks like:
>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>
>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>> subnet/prefix
>>>>>> 0:
>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>>     *
>>>>>>>     *
>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
>>>>>>> net.ipv6.conf.all.autoconf = 1
>>>>>>>
>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
>>>>>>> configured the nics:
>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
>>>>>>> network without cleanup to have it
>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>
>>>>>>>     *
>>>>>>> Next, I created a /64 network for the isolated guest network
>>>>>>> on eth0
>>>> of
>>>>>> VR using radvd:
>>>>>>>
>>>>>>> # cat /etc/radvd.conf
>>>>>>> interface eth0
>>>>>>> {
>>>>>>>       AdvSendAdvert on;
>>>>>>>       MinRtrAdvInterval 5;
>>>>>>>       MaxRtrAdvInterval 15;
>>>>>>>       prefix 2001:470:ed36:1::/64
>>>>>>>       {
>>>>>>>           AdvOnLink on;
>>>>>>>           AdvAutonomous on;
>>>>>>>       };
>>>>>>> };
>>>>>>> systemctl restart radvd
>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
>>>>>>> this
>>>>>> ...:1::/64 network
>>>>>>>     *   Finally I added a static route in toy core-router for the
>> new
>>>> /64
>>>>>> IPv6 range in the isolated network
>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>> eth2> dev
>>>>>> <local lan nic>
>>>>>>>     *
>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
>>>>>>> for the
>>>>>> new /64 network
>>>>>>>
>>>>>>> And voila all done! I create a domain AAAA record that points
>>>>>>> to my
>>>>>> guest VM IPv6 address a test webserver on
>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>>>>>>>
>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
>>>>>>> after a
>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>
>>>>>>>
>>>>>>> Regards.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>>
>

Re: IPV6 in Isolated/VPC networks

Posted by Wido den Hollander <wi...@widodh.nl>.
Op 10-05-2022 om 10:19 schreef Abhishek Kumar:
> Hi all,
> 
> IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
> Documentation for it is available at http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier (will be available in the official docs once 4.17.0 version of docs is built).
> 

Great work!

I see only static routing is supported. But do we publish something on 
the message bus once a new VR/VPC is created?

This way you could pick up these messages and have the network create a 
(static) route based on those.

ExaBGP for example could be used to inject such routes.

Wido

> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> 
> Regards,
> Abhishek
> 
> ________________________________
> From: Rohit Yadav <ro...@shapeblue.com>
> Sent: 13 September 2021 14:30
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
> 
> Thanks Alex, Wei. I've updated the docs here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> 
> I'll leave the thread open for futher discussion/ideas/feedback. I think we've completed the phase1 design doc including all feedback comments for adding IPv6 support in CloudStack and some initial poc/work can be started. My colleagues and I will keep everyone posted on this thread and/or on a Github PR as and when we're able to start our work on the same (after 4.16, potentially towards 4.17).
> 
> 
> Regards.
> 
> ________________________________
> From: Wei ZHOU <us...@gmail.com>
> Sent: Friday, September 10, 2021 20:22
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
> 
> Agree with Alex.
> We only need to know how many /64 are allocated. We do not care how many
> ipv6 addresses are used by VMs.
> 
> -Wei
> 
> On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <Al...@shapeblue.com>
> wrote:
> 
>> Hi Rohit,
>>
>> I'd go for option 2, don't see a point tracking anything smaller than a
>> /64 tbh.
>>
>> Cheers
>> Alex
>>
>>
>>
>>
>> -----Original Message-----
>> From: Rohit Yadav <ro...@shapeblue.com>
>> Sent: 09 September 2021 12:44
>> To: dev@cloudstack.apache.org
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Thanks Alex, Kristaps. I've updated the design doc to reflect two
>> agreements:
>>
>>    *   Allocate /64 for both isolated network and VPC tiers, no large
>> allocation of prefixes to VPC (cons: more static routing rules for upstream
>> router/admins)
>>    *   All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a
>> dedicated /64 prefix/block for systemvms
>>
>> The only outstanding question now is:
>>
>>    *   How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage
>> for shared network (design, implementation and use-cases?)
>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>> usage we don't track. For public VR/nics/networks, we can simply add the
>> IPv6 details for a related IPv4 address.
>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>> tracking/management and usage for all VMs and systemvms nic (this means
>> account/domain level limits and new billing/records)
>> Option3: other thoughts?
>>
>>
>> Regards.
>>
>> ________________________________
>> From: Alex Mattioli <Al...@shapeblue.com>
>> Sent: Wednesday, September 8, 2021 23:24
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: RE: IPV6 in Isolated/VPC networks
>>
>> Hi Rohit, Kristaps,
>>
>> I'd say option 1 as well,  it does create a bit more overhead with static
>> routes but if that's automated for a VPC it can also be easily automated
>> for several tiers of a VPC.  We also don't constrain the amount of tiers in
>> a  VPC.
>> It has the added advantage to be closer to the desired behaviour with
>> dynamic routing in the future, where a VPC VR can announce several subnets
>> upstream.
>>
>> Cheers
>> Alex
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Rohit Yadav <ro...@shapeblue.com>
>> Sent: 08 September 2021 19:04
>> To: dev@cloudstack.apache.org
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Hi Kristaps,
>>
>> Thanks for sharing, I suppose that means individual tiers should be
>> allocated /64 instead of larger ipv6 blocks to the whole VPC which could
>> cause wastage.
>>
>> Any objection from anybody?
>>
>> Regards.
>> ________________________________
>> From: Kristaps Cudars <kr...@gmail.com>
>> Sent: Wednesday, September 8, 2021 9:24:01 PM
>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>> Subject: Re: IPV6 in Isolated/VPC networks
>>
>> Hello,
>>
>> I asked networking team to comment on "How should the IPv6
>> block/allocation work in VPCs?"
>> Option1: They haven't seen lately devices with limits on how many static
>> routes can be created.
>> Option2: With /60 and /62 assignments and big quantity of routers IPv6
>> assignment from RIPE NNC can be drained fast.
>>
>> /48 contains 64000 /64
>> /60 contains 16 /64
>> 64000 / 16 = 4000 routers
>>
>>
>> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
>>> All,
>>>
>>> After another iteration with Alex, I've updated the design doc. Kindly
>> review:
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>
>>>
>>> ... and advise on some outstanding questions:
>>>
>>>    *   How should the IPv6 block/allocation work in VPCs?
>>> Option1: Should this be simply /64 allocation on any new tier, the
>>> cons of this option is one static route/rule per VPC tier. (many
>>> upstream routers may have limit on no. of static routes?)
>>> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
>> (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks
>> global setting (3 is default). The pros of this option are less no. of
>> static route/rule and easy programming, but potential wastage of multiple
>> /64 prefix blocks for unused/uncreated tiers.
>>>    *   How do we manage IPv6 usage? Can anyone advise how we do IPv6
>> usage for shared network (design, implementation and use-cases?)
>>> Option1: We don't do it, all user VMs nics have ipv4 address which whose
>> usage we don't track. For public VR/nics/networks, we can simply add the
>> IPv6 details for a related IPv4 address.
>>> Option2: Implement a separate, first-class IPv6 address or /64 prefix
>> tracking/management and usage for all VMs and systemvms nic (this means
>> account/domain level limits and new billing/records)
>>>    *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by default
>> if zone has a IPv6 address block allocated/assigned for use for systemvms
>> (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons
>> of this?)
>>>    *
>>>
>>> Regards.
>>>
>>> ________________________________
>>> From: Rohit Yadav <ro...@shapeblue.com>
>>> Sent: Thursday, August 19, 2021 15:45
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Hi all,
>>>
>>> I've taken feedback from this thread and wrote this design doc:
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
>>> +Isolated+Network+and+VPC+with+Static+Routing
>>>
>>> Kindly review and advise if I missed anything or anything that needs to
>> be changed/updated. You may comment on the wiki directly too.
>>>
>>> Kindly suggest your views on the following (also in the design doc
>> above):
>>>
>>> Outstanding Questions:
>>>
>>>    *   Should admin or user be able to specify how VPC super CIDRs are
>> created/needed; for example a user can ask for VPC with /60 super CIDR? Or
>> should CloudStack automatically find/allocate a /64 for a new VPC tier from
>> the root-admin configured /64-/48 block?
>>>    *   Should we explore FRR and iBGP or other strategies to do dynamic
>> routing which may not require advance/complex configuration in the VR or
>> for the users/admin?
>>>    *   With SLAAC and no dhcpv6, is there a way to support secondary IPv6
>> addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's
>> nics?
>>>    *   Any thoughts on UI/UX for firewall/routing management?
>>>    *   Any other feature/support for isolated network or VPC feature that
>> must be explored or supported such as PF, VPN, LB, vpc static routes, vpc
>> gateway etc.
>>>    *   For usage - should we have any consideration, or should we assume
>> that IPv4 and IPv6 address will go together for every nic; so IPv6 usage
>> for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new
>> biling/usage needed?
>>>    *   For smoketests, local dev-test should we explore ULA? Unique Local
>> Address - in the range fc00::/7. Typically only within the 'local' half
>> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
>> This prefix can be randomly generated at first install by CloudStack in a
>> zone using zoneid etc?
>>>    *   Should we expand support for VR diagnostic feature to include
>> support for ipv6, traceroute6?
>>>    *   Discuss - expand support/ability to allocate a /60, or /56 etc
>> prefix to an individual VM in shared network (Wido's suggestion)
>>>
>>>
>>> Regards.
>>>
>>> ________________________________
>>> From: Wei ZHOU <us...@gmail.com>
>>> Sent: Tuesday, August 17, 2021 21:16
>>> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
>>> Subject: Re: IPV6 in Isolated/VPC networks
>>>
>>> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>>>
>>> I am fine with not having stateful dhcpv6 and privacy
>>> extension/temporary address in phase 1. If community decides not to do
>>> eventually , it is also ok to me.
>>>
>>> We could explore how to better use secondary ipv6 addresses as Wido
>>> advised. It would be great if anyone share some user experience.
>>>
>>> -Wei
>>>
>>>
>>> On Tuesday, 17 August 2021, Kristaps Cudars
>>> <kr...@gmail.com>
>>> wrote:
>>>
>>>> Hi Wei,
>>>>
>>>> Published this month's RFC 9099 and explains in different
>>>> words/perspective what has been written by Alex, Rohit and Wido.
>>>> https://www.rfc-editor.org/rfc/rfc9099.html
>>>>
>>>>
>>>> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
>>>>> Hi Wido,
>>>>>
>>>>> (cc to Rohit and Alex)
>>>>>
>>>>> It is a good suggestion to use FRR for ipv6. The configuration is
>>>>> quite simple and the VMs can get SLAAC, routes, etc.
>>>>>
>>>>> Privacy extension looks not the same as what you mentioned. see
>>>>> https://datatracker.ietf.org/doc/html/rfc4941
>>>>>
>>>>> You are right. To use static routing, the admins need to configure
>>>>> the routes in the upstream router, and add some ipv6 ranges (eg
>>>>> /56 for VPCs and /64 for isolated networks) and their next-hop
>>>>> (which will be configured in VRs) in CloudStack. CloudStack will
>>>>> pick up an IPv6 range
>>>> and
>>>>> assign it to an isolated network or vpc. @Rohit, correct me if I'm
>> wrong.
>>>>>
>>>>> I have a question, it looks stateless dhcpv6 (SLAAC from
>>>>> router/VR, router/dns etc via RA messages) will be the only option
>>>>> for now (related
>>>> to
>>>>> your pr https://github.com/apache/cloudstack/pull/3077) . Would it
>>>>> be
>>>> good
>>>>> to provide stateful dhcpv6 (which can be implemented by dnsmasq)
>>>>> as an option in cloudstack ? The advantages are
>>>>> (1) support other ipv6 cidr sizes than /64.
>>>>> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
>>>>> addresses can be changed
>>>>> (4) an Ipv6 addresses can be re-used by multiple vms.
>>>>> The problem is, stateful dhcpv6 does not support
>>>>> routers,nameservers,
>>>> etc.
>>>>> we need to figure it out (probably use radvd/frr and dnsmasq both).
>>>>>
>>>>> -Wei
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
> 
> 
> 
>   
> 
>> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> See my inline responses:
>>>>>>
>>>>>> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Thanks for your feedback and ideas, I've gone ahead with
>>>>>>> discussing
>>>> them
>>>>>> with Alex and came up with a PoC/design which can be implemented
>>>>>> in the following phases:
>>>>>>>
>>>>>>>     *   Phase1: implement ipv6 support in isolated networks and
>> VPC
>>>> with
>>>>>> static routing
>>>>>>>     *   Phase2: discuss and implement support for dynamic routing
>>>> (TBD)
>>>>>>>
>>>>>>> For Phase1 here's the high-level proposal:
>>>>>>>
>>>>>>>     *   IPv6 address management:
>>>>>>>        *   At the zone level root-admin specifies a /64 public
>> range
>>>> that
>>>>>> will be used for VRs, then they can add a /48, or /56 IPv6 range
>>>>>> for
>>>> guest
>>>>>> networks (to be used by isolated networks and VPC tiers)
>>>>>>>        *   On creation of any IPv6 enabled isolated network or VPC
>>>> tier,
>>>>>> from the /48 or /56 block a /64 network is allocated/used
>>>>>>>        *   We assume SLAAC and autoconfiguration, no DHCPv6 in the
>>>> zone
>>>>>> (discuss: is privacy a concern, can privacy extensions rfc4941
>>>>>> of
>>>> slaac be
>>>>>> explored?)
>>>>>>
>>>>>> Privacy Extensions are only a concern for client devices which
>>>>>> roam between different IPv6 networks.
>>>>>>
>>>>>> If you IPv6 address of a client keeps the same suffix (MAC
>>>>>> based) and switches network then only the prefix (/64) will change.
>>>>>>
>>>>>> This way a network like Google, Facebook, etc could track your
>>>>>> device moving from network to network if they only look at the
>>>>>> last 64-bits of the IPv6 address.
>>>>>>
>>>>>> For servers this is not a problem as you already know in which
>>>>>> network they are.
>>>>>>
>>>>>>>     *   Network offerings: root-admin can create new network
>> offerings
>>>>>> (with VPC too) that specifies a network stack option:
>>>>>>>        *   ipv4 only (default, for backward compatibility all
>>>>>> networks/offerings post-upgrade migrate to this option)
>>>>>>>        *   ipv4-and-ipv6
>>>>>>>        *   ipv6-only (this can be phase 1.b)
>>>>>>>        *   A new routing option: static (phase1), dynamic (phase2,
>>>> with
>>>>>> multiple sub-options such as ospf/bgp etc...)
>>>>>>
>>>>>> This means that the network admin will need to statically route
>>>>>> the
>>>> IPv6
>>>>>> subnet to the VR's outside IPv6 address, for example, on a JunOS
>>>> router:
>>>>>>
>>>>>> set routing-options rib inet6.0 static route 2001:db8:500::/48
>>>>>> next-hop
>>>>>> 2001:db8:100::50
>>>>>>
>>>>>> I'm assuming that 2001:db8:100::50 is the address of the VR on
>>>>>> the outside (/64) network. In reality this will probably be a
>>>>>> longer address, but this is for just the example.
>>>>>>
>>>>>>>     *   VR changes:
>>>>>>>        *   VR gets its guest and public nics set to inet6 auto
>>>>>>>        *   For each /64 allocated to guest network and VPC tiers,
>>>> radvd
>>>>>> is configured to do RA
>>>>>>
>>>>>> radvd is fine, but looking at phase 2 with dynamic routing you
>>>>>> might already want to look into FRRouting. FRR can also
>>>>>> advertise RAs while not doing any routing.
>>>>>>
>>>>>> interface ens4
>>>>>>     no ipv6 nd suppress-ra
>>>>>>     ipv6 nd prefix 2001:db8:500::/64
>>>>>>     ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>>>>>>
>>>>>> See: http://docs.frrouting.org/en/latest/ipv6.html
>>>>>>
>>>>>>>        *   Firewall: a new ipv6 zone/chain is created for ipv6
>> where
>>>> ipv6
>>>>>> firewall rules (ACLs, ingress, egress) are implemented; ACLs
>>>>>> between
>>>> VPC
>>>>>> tiers are managed/implemented by ipv6 firewall on VR
>>>>>>
>>>>>> Please take a look at the existing security_group.py script
>>>>>> which implements RFC4890
>>>>>>
>>>>>> https://datatracker.ietf.org/doc/html/rfc4890
>>>>>>
>>>>>> ICMPv6 is a vital part of IPv6 and certain packets should always
>>>>>> be allowed.
>>>>>>
>>>>>>>        *   It is assumed that static routes are created on the
>>>> core/main
>>>>>> router by the admin or automated using some scripts/tools; for
>>>>>> this CloudStack will announce events with details of /64
>>>>>> networks and VR's public IPv6 address that can be consumed by a
>>>>>> rabbitmq/message bus
>>>> client
>>>>>> (for example), or a custom cron job or script as part of
>> orchestration.
>>>>>> (this wouldn't be necessary for dynamic routing bgp with
>>>>>> phase2)\\
>>>>>>
>>>>>> You would only need to announce the /48 or /56 allocated to the
>>>>>> VR, that's all. You don't need to inform the upstream router
>>>>>> about the /64 subnets created within that larger subnet.
>>>>>>
>>>>>>>     *   Guest Networking: With SLAAC, it's easy for CloudStack to
>>>>>> calculate allocate and use a /64 and determine the IPv6 address
>>>>>> of VR
>>>> nics
>>>>>> and guest VM nics
>>>>>>>        *   A user create an isolated network/VPC with an offering
>>>> that is
>>>>>> ipv6 enabled
>>>>>>>        *   A user can manage firewall for the IPv6 address/guest
>> nics;
>>>>>> there'll be no port forward and LB feature though for IPv6
>>>>>>>        *   A users can run workloads in the guest VMs that listen
>> on
>>>>>> publically routable ipv6 addresses
>>>>>>>        *   Usage/billing etc continue to work, no change needed
>>>>>>>
>>>>>>> Network layout:
>>>>>>>
>>>>>>> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
>>>>>>> VLAN] ->
>>>>>> [guest VMs/nics]
>>>>>>> *core/ISP router needs static routes to be added (manually or
>>>>>> automated), assumes a /48 or /56 configured for the zone
>>>>>>>
>>>>>>> Thoughts, feedback?
>>>>>>
>>>>>> Looks doable!
>>>>>>
>>>>>> Side-note: It would be very cool if you could use parts of this
>>>>>> implementation to also route /48, /56, or /60 subnets to
>>>>>> individual VMs in Shared networks.
>>>>>>
>>>>>> Why? This allows for running Docker containers with native IPv6
>>>>>> inside the VM or running a (Open)VPN server inside a VM which
>>>>>> then assigns public IPv6 addresses to clients connected.
>>>>>>
>>>>>> Instead of routing the subnet to a VR we route the subnet to a
>>>>>> single instance in a shared network.
>>>>>>
>>>>>> If we could then also move these subnets between Instances
>>>>>> easily one can quickly migrate to a different instance while
>>>>>> keeping the same IPv6 subnet.
>>>>>>
>>>>>> Wido
>>>>>>
>>>>>>>
>>>>>>> Proof-of-concept commentary: here's what I did to test the idea:
>>>>>>>
>>>>>>>     *   Created an isolated network and deployed a VM in my home
>> lab
>>>>>>> The VR running on KVM has following nics
>>>>>>> eth0 - guest network
>>>>>>> eth1 - link local
>>>>>>> eth2 - public network
>>>>>>>
>>>>>>>     *   I setup a custom openwrt router on a RPi4 to serve as a
>>>> toy-core
>>>>>> router where I create a wan6 IPv6 tunnel using tunnel broker and
>>>>>> I got
>>>> a
>>>>>> /48 allocated. My configuration looks like:
>>>>>>> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
>>>>>>> /64 - 2001:470:36:3e2::/64 (default allocated by)
>>>>>>>
>>>>>>> I create a LAN ipv6 (public network for CloudStack VR): at
>>>> subnet/prefix
>>>>>> 0:
>>>>>>> LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
>>>>>>> SLAAC+stateless DHCP (no dhcpv6)
>>>>>>>     *
>>>>>>>     *
>>>>>>> In the isolated VR, I enabled ipv6 as:
>>>>>>> net.ipv6.conf.all.disable_ipv6 = 0
>>>>>>> net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
>>>>>>> 1 net.ipv6.conf.all.accept_redirects = 1
>>>>>>> net.ipv6.conf.all.autoconf = 1
>>>>>>>
>>>>>>> Set up a IPv6 nameserver/dns in /etc/resolve.conf And
>>>>>>> configured the nics:
>>>>>>> echo iface eth0 inet6 auto >> /etc/network/interfaces echo
>>>>>>> iface eth2 inet6 auto >> /etc/network/interfaces
>>>>>>> /etc/init.d/networking restart Next, restart ACS isolated
>>>>>>> network without cleanup to have it
>>>>>> reconfigure IPv4 nics, firewall, NAT etc
>>>>>>>
>>>>>>>     *
>>>>>>> Next, I created a /64 network for the isolated guest network
>>>>>>> on eth0
>>>> of
>>>>>> VR using radvd:
>>>>>>>
>>>>>>> # cat /etc/radvd.conf
>>>>>>> interface eth0
>>>>>>> {
>>>>>>>       AdvSendAdvert on;
>>>>>>>       MinRtrAdvInterval 5;
>>>>>>>       MaxRtrAdvInterval 15;
>>>>>>>       prefix 2001:470:ed36:1::/64
>>>>>>>       {
>>>>>>>           AdvOnLink on;
>>>>>>>           AdvAutonomous on;
>>>>>>>       };
>>>>>>> };
>>>>>>> systemctl restart radvd
>>>>>>> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
>>>>>>> this
>>>>>> ...:1::/64 network
>>>>>>>     *   Finally I added a static route in toy core-router for the
>> new
>>>> /64
>>>>>> IPv6 range in the isolated network
>>>>>>> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
>>>>>>> eth2> dev
>>>>>> <local lan nic>
>>>>>>>     *
>>>>>>> ... and I enabled firewall rules to allow any traffic to pass
>>>>>>> for the
>>>>>> new /64 network
>>>>>>>
>>>>>>> And voila all done! I create a domain AAAA record that points
>>>>>>> to my
>>>>>> guest VM IPv6 address a test webserver on
>>>>>>> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>>>>>>>
>>>>>>> (Note: I'll get rid of the tunnel and request a new /48 block
>>>>>>> after a
>>>>>> few days, sharing this solely for testing purposes)
>>>>>>>
>>>>>>>
>>>>>>> Regards.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>>
> 

Re: IPV6 in Isolated/VPC networks

Posted by Abhishek Kumar <Ab...@shapeblue.com>.
Hi all,

IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ?
Documentation for it is available at http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier (will be available in the official docs once 4.17.0 version of docs is built).

[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing

Regards,
Abhishek

________________________________
From: Rohit Yadav <ro...@shapeblue.com>
Sent: 13 September 2021 14:30
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks

Thanks Alex, Wei. I've updated the docs here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing

I'll leave the thread open for futher discussion/ideas/feedback. I think we've completed the phase1 design doc including all feedback comments for adding IPv6 support in CloudStack and some initial poc/work can be started. My colleagues and I will keep everyone posted on this thread and/or on a Github PR as and when we're able to start our work on the same (after 4.16, potentially towards 4.17).


Regards.

________________________________
From: Wei ZHOU <us...@gmail.com>
Sent: Friday, September 10, 2021 20:22
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks

Agree with Alex.
We only need to know how many /64 are allocated. We do not care how many
ipv6 addresses are used by VMs.

-Wei

On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <Al...@shapeblue.com>
wrote:

> Hi Rohit,
>
> I'd go for option 2, don't see a point tracking anything smaller than a
> /64 tbh.
>
> Cheers
> Alex
>
>
>
>
> -----Original Message-----
> From: Rohit Yadav <ro...@shapeblue.com>
> Sent: 09 September 2021 12:44
> To: dev@cloudstack.apache.org
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Thanks Alex, Kristaps. I've updated the design doc to reflect two
> agreements:
>
>   *   Allocate /64 for both isolated network and VPC tiers, no large
> allocation of prefixes to VPC (cons: more static routing rules for upstream
> router/admins)
>   *   All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a
> dedicated /64 prefix/block for systemvms
>
> The only outstanding question now is:
>
>   *   How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage
> for shared network (design, implementation and use-cases?)
> Option1: We don't do it, all user VMs nics have ipv4 address which whose
> usage we don't track. For public VR/nics/networks, we can simply add the
> IPv6 details for a related IPv4 address.
> Option2: Implement a separate, first-class IPv6 address or /64 prefix
> tracking/management and usage for all VMs and systemvms nic (this means
> account/domain level limits and new billing/records)
> Option3: other thoughts?
>
>
> Regards.
>
> ________________________________
> From: Alex Mattioli <Al...@shapeblue.com>
> Sent: Wednesday, September 8, 2021 23:24
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: RE: IPV6 in Isolated/VPC networks
>
> Hi Rohit, Kristaps,
>
> I'd say option 1 as well,  it does create a bit more overhead with static
> routes but if that's automated for a VPC it can also be easily automated
> for several tiers of a VPC.  We also don't constrain the amount of tiers in
> a  VPC.
> It has the added advantage to be closer to the desired behaviour with
> dynamic routing in the future, where a VPC VR can announce several subnets
> upstream.
>
> Cheers
> Alex
>
>
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: Rohit Yadav <ro...@shapeblue.com>
> Sent: 08 September 2021 19:04
> To: dev@cloudstack.apache.org
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Hi Kristaps,
>
> Thanks for sharing, I suppose that means individual tiers should be
> allocated /64 instead of larger ipv6 blocks to the whole VPC which could
> cause wastage.
>
> Any objection from anybody?
>
> Regards.
> ________________________________
> From: Kristaps Cudars <kr...@gmail.com>
> Sent: Wednesday, September 8, 2021 9:24:01 PM
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Hello,
>
> I asked networking team to comment on "How should the IPv6
> block/allocation work in VPCs?"
> Option1: They haven't seen lately devices with limits on how many static
> routes can be created.
> Option2: With /60 and /62 assignments and big quantity of routers IPv6
> assignment from RIPE NNC can be drained fast.
>
> /48 contains 64000 /64
> /60 contains 16 /64
> 64000 / 16 = 4000 routers
>
>
> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
> > All,
> >
> > After another iteration with Alex, I've updated the design doc. Kindly
> review:
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> > +Isolated+Network+and+VPC+with+Static+Routing
> >
> >
> > ... and advise on some outstanding questions:
> >
> >   *   How should the IPv6 block/allocation work in VPCs?
> > Option1: Should this be simply /64 allocation on any new tier, the
> > cons of this option is one static route/rule per VPC tier. (many
> > upstream routers may have limit on no. of static routes?)
> > Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
> (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks
> global setting (3 is default). The pros of this option are less no. of
> static route/rule and easy programming, but potential wastage of multiple
> /64 prefix blocks for unused/uncreated tiers.
> >   *   How do we manage IPv6 usage? Can anyone advise how we do IPv6
> usage for shared network (design, implementation and use-cases?)
> > Option1: We don't do it, all user VMs nics have ipv4 address which whose
> usage we don't track. For public VR/nics/networks, we can simply add the
> IPv6 details for a related IPv4 address.
> > Option2: Implement a separate, first-class IPv6 address or /64 prefix
> tracking/management and usage for all VMs and systemvms nic (this means
> account/domain level limits and new billing/records)
> >   *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by default
> if zone has a IPv6 address block allocated/assigned for use for systemvms
> (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons
> of this?)
> >   *
> >
> > Regards.
> >
> > ________________________________
> > From: Rohit Yadav <ro...@shapeblue.com>
> > Sent: Thursday, August 19, 2021 15:45
> > To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> > Subject: Re: IPV6 in Isolated/VPC networks
> >
> > Hi all,
> >
> > I've taken feedback from this thread and wrote this design doc:
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> > +Isolated+Network+and+VPC+with+Static+Routing
> >
> > Kindly review and advise if I missed anything or anything that needs to
> be changed/updated. You may comment on the wiki directly too.
> >
> > Kindly suggest your views on the following (also in the design doc
> above):
> >
> > Outstanding Questions:
> >
> >   *   Should admin or user be able to specify how VPC super CIDRs are
> created/needed; for example a user can ask for VPC with /60 super CIDR? Or
> should CloudStack automatically find/allocate a /64 for a new VPC tier from
> the root-admin configured /64-/48 block?
> >   *   Should we explore FRR and iBGP or other strategies to do dynamic
> routing which may not require advance/complex configuration in the VR or
> for the users/admin?
> >   *   With SLAAC and no dhcpv6, is there a way to support secondary IPv6
> addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's
> nics?
> >   *   Any thoughts on UI/UX for firewall/routing management?
> >   *   Any other feature/support for isolated network or VPC feature that
> must be explored or supported such as PF, VPN, LB, vpc static routes, vpc
> gateway etc.
> >   *   For usage - should we have any consideration, or should we assume
> that IPv4 and IPv6 address will go together for every nic; so IPv6 usage
> for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new
> biling/usage needed?
> >   *   For smoketests, local dev-test should we explore ULA? Unique Local
> Address - in the range fc00::/7. Typically only within the 'local' half
> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
> This prefix can be randomly generated at first install by CloudStack in a
> zone using zoneid etc?
> >   *   Should we expand support for VR diagnostic feature to include
> support for ipv6, traceroute6?
> >   *   Discuss - expand support/ability to allocate a /60, or /56 etc
> prefix to an individual VM in shared network (Wido's suggestion)
> >
> >
> > Regards.
> >
> > ________________________________
> > From: Wei ZHOU <us...@gmail.com>
> > Sent: Tuesday, August 17, 2021 21:16
> > To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> > Subject: Re: IPV6 in Isolated/VPC networks
> >
> > Thanks Kristaps, Wido, Rohit and Alex for your replies.
> >
> > I am fine with not having stateful dhcpv6 and privacy
> > extension/temporary address in phase 1. If community decides not to do
> > eventually , it is also ok to me.
> >
> > We could explore how to better use secondary ipv6 addresses as Wido
> > advised. It would be great if anyone share some user experience.
> >
> > -Wei
> >
> >
> > On Tuesday, 17 August 2021, Kristaps Cudars
> > <kr...@gmail.com>
> > wrote:
> >
> > > Hi Wei,
> > >
> > > Published this month's RFC 9099 and explains in different
> > > words/perspective what has been written by Alex, Rohit and Wido.
> > > https://www.rfc-editor.org/rfc/rfc9099.html
> > >
> > >
> > > On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
> > > > Hi Wido,
> > > >
> > > > (cc to Rohit and Alex)
> > > >
> > > > It is a good suggestion to use FRR for ipv6. The configuration is
> > > > quite simple and the VMs can get SLAAC, routes, etc.
> > > >
> > > > Privacy extension looks not the same as what you mentioned. see
> > > > https://datatracker.ietf.org/doc/html/rfc4941
> > > >
> > > > You are right. To use static routing, the admins need to configure
> > > > the routes in the upstream router, and add some ipv6 ranges (eg
> > > > /56 for VPCs and /64 for isolated networks) and their next-hop
> > > > (which will be configured in VRs) in CloudStack. CloudStack will
> > > > pick up an IPv6 range
> > > and
> > > > assign it to an isolated network or vpc. @Rohit, correct me if I'm
> wrong.
> > > >
> > > > I have a question, it looks stateless dhcpv6 (SLAAC from
> > > > router/VR, router/dns etc via RA messages) will be the only option
> > > > for now (related
> > > to
> > > > your pr https://github.com/apache/cloudstack/pull/3077) . Would it
> > > > be
> > > good
> > > > to provide stateful dhcpv6 (which can be implemented by dnsmasq)
> > > > as an option in cloudstack ? The advantages are
> > > > (1) support other ipv6 cidr sizes than /64.
> > > > (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
> > > > addresses can be changed
> > > > (4) an Ipv6 addresses can be re-used by multiple vms.
> > > > The problem is, stateful dhcpv6 does not support
> > > > routers,nameservers,
> > > etc.
> > > > we need to figure it out (probably use radvd/frr and dnsmasq both).
> > > >
> > > > -Wei
> > > >
> > > >
> > >
> >
> >
> >
> >
> >
> >
>
>
>



 

> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > See my inline responses:
> > > > >
> > > > > Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> > > > > > Hi all,
> > > > > >
> > > > > > Thanks for your feedback and ideas, I've gone ahead with
> > > > > > discussing
> > > them
> > > > > with Alex and came up with a PoC/design which can be implemented
> > > > > in the following phases:
> > > > > >
> > > > > >    *   Phase1: implement ipv6 support in isolated networks and
> VPC
> > > with
> > > > > static routing
> > > > > >    *   Phase2: discuss and implement support for dynamic routing
> > > (TBD)
> > > > > >
> > > > > > For Phase1 here's the high-level proposal:
> > > > > >
> > > > > >    *   IPv6 address management:
> > > > > >       *   At the zone level root-admin specifies a /64 public
> range
> > > that
> > > > > will be used for VRs, then they can add a /48, or /56 IPv6 range
> > > > > for
> > > guest
> > > > > networks (to be used by isolated networks and VPC tiers)
> > > > > >       *   On creation of any IPv6 enabled isolated network or VPC
> > > tier,
> > > > > from the /48 or /56 block a /64 network is allocated/used
> > > > > >       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the
> > > zone
> > > > > (discuss: is privacy a concern, can privacy extensions rfc4941
> > > > > of
> > > slaac be
> > > > > explored?)
> > > > >
> > > > > Privacy Extensions are only a concern for client devices which
> > > > > roam between different IPv6 networks.
> > > > >
> > > > > If you IPv6 address of a client keeps the same suffix (MAC
> > > > > based) and switches network then only the prefix (/64) will change.
> > > > >
> > > > > This way a network like Google, Facebook, etc could track your
> > > > > device moving from network to network if they only look at the
> > > > > last 64-bits of the IPv6 address.
> > > > >
> > > > > For servers this is not a problem as you already know in which
> > > > > network they are.
> > > > >
> > > > > >    *   Network offerings: root-admin can create new network
> offerings
> > > > > (with VPC too) that specifies a network stack option:
> > > > > >       *   ipv4 only (default, for backward compatibility all
> > > > > networks/offerings post-upgrade migrate to this option)
> > > > > >       *   ipv4-and-ipv6
> > > > > >       *   ipv6-only (this can be phase 1.b)
> > > > > >       *   A new routing option: static (phase1), dynamic (phase2,
> > > with
> > > > > multiple sub-options such as ospf/bgp etc...)
> > > > >
> > > > > This means that the network admin will need to statically route
> > > > > the
> > > IPv6
> > > > > subnet to the VR's outside IPv6 address, for example, on a JunOS
> > > router:
> > > > >
> > > > > set routing-options rib inet6.0 static route 2001:db8:500::/48
> > > > > next-hop
> > > > > 2001:db8:100::50
> > > > >
> > > > > I'm assuming that 2001:db8:100::50 is the address of the VR on
> > > > > the outside (/64) network. In reality this will probably be a
> > > > > longer address, but this is for just the example.
> > > > >
> > > > > >    *   VR changes:
> > > > > >       *   VR gets its guest and public nics set to inet6 auto
> > > > > >       *   For each /64 allocated to guest network and VPC tiers,
> > > radvd
> > > > > is configured to do RA
> > > > >
> > > > > radvd is fine, but looking at phase 2 with dynamic routing you
> > > > > might already want to look into FRRouting. FRR can also
> > > > > advertise RAs while not doing any routing.
> > > > >
> > > > > interface ens4
> > > > >    no ipv6 nd suppress-ra
> > > > >    ipv6 nd prefix 2001:db8:500::/64
> > > > >    ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> > > > >
> > > > > See: http://docs.frrouting.org/en/latest/ipv6.html
> > > > >
> > > > > >       *   Firewall: a new ipv6 zone/chain is created for ipv6
> where
> > > ipv6
> > > > > firewall rules (ACLs, ingress, egress) are implemented; ACLs
> > > > > between
> > > VPC
> > > > > tiers are managed/implemented by ipv6 firewall on VR
> > > > >
> > > > > Please take a look at the existing security_group.py script
> > > > > which implements RFC4890
> > > > >
> > > > > https://datatracker.ietf.org/doc/html/rfc4890
> > > > >
> > > > > ICMPv6 is a vital part of IPv6 and certain packets should always
> > > > > be allowed.
> > > > >
> > > > > >       *   It is assumed that static routes are created on the
> > > core/main
> > > > > router by the admin or automated using some scripts/tools; for
> > > > > this CloudStack will announce events with details of /64
> > > > > networks and VR's public IPv6 address that can be consumed by a
> > > > > rabbitmq/message bus
> > > client
> > > > > (for example), or a custom cron job or script as part of
> orchestration.
> > > > > (this wouldn't be necessary for dynamic routing bgp with
> > > > > phase2)\\
> > > > >
> > > > > You would only need to announce the /48 or /56 allocated to the
> > > > > VR, that's all. You don't need to inform the upstream router
> > > > > about the /64 subnets created within that larger subnet.
> > > > >
> > > > > >    *   Guest Networking: With SLAAC, it's easy for CloudStack to
> > > > > calculate allocate and use a /64 and determine the IPv6 address
> > > > > of VR
> > > nics
> > > > > and guest VM nics
> > > > > >       *   A user create an isolated network/VPC with an offering
> > > that is
> > > > > ipv6 enabled
> > > > > >       *   A user can manage firewall for the IPv6 address/guest
> nics;
> > > > > there'll be no port forward and LB feature though for IPv6
> > > > > >       *   A users can run workloads in the guest VMs that listen
> on
> > > > > publically routable ipv6 addresses
> > > > > >       *   Usage/billing etc continue to work, no change needed
> > > > > >
> > > > > > Network layout:
> > > > > >
> > > > > > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
> > > > > > VLAN] ->
> > > > > [guest VMs/nics]
> > > > > > *core/ISP router needs static routes to be added (manually or
> > > > > automated), assumes a /48 or /56 configured for the zone
> > > > > >
> > > > > > Thoughts, feedback?
> > > > >
> > > > > Looks doable!
> > > > >
> > > > > Side-note: It would be very cool if you could use parts of this
> > > > > implementation to also route /48, /56, or /60 subnets to
> > > > > individual VMs in Shared networks.
> > > > >
> > > > > Why? This allows for running Docker containers with native IPv6
> > > > > inside the VM or running a (Open)VPN server inside a VM which
> > > > > then assigns public IPv6 addresses to clients connected.
> > > > >
> > > > > Instead of routing the subnet to a VR we route the subnet to a
> > > > > single instance in a shared network.
> > > > >
> > > > > If we could then also move these subnets between Instances
> > > > > easily one can quickly migrate to a different instance while
> > > > > keeping the same IPv6 subnet.
> > > > >
> > > > > Wido
> > > > >
> > > > > >
> > > > > > Proof-of-concept commentary: here's what I did to test the idea:
> > > > > >
> > > > > >    *   Created an isolated network and deployed a VM in my home
> lab
> > > > > > The VR running on KVM has following nics
> > > > > > eth0 - guest network
> > > > > > eth1 - link local
> > > > > > eth2 - public network
> > > > > >
> > > > > >    *   I setup a custom openwrt router on a RPi4 to serve as a
> > > toy-core
> > > > > router where I create a wan6 IPv6 tunnel using tunnel broker and
> > > > > I got
> > > a
> > > > > /48 allocated. My configuration looks like:
> > > > > > /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> > > > > > /64 - 2001:470:36:3e2::/64 (default allocated by)
> > > > > >
> > > > > > I create a LAN ipv6 (public network for CloudStack VR): at
> > > subnet/prefix
> > > > > 0:
> > > > > > LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
> > > > > > SLAAC+stateless DHCP (no dhcpv6)
> > > > > >    *
> > > > > >    *
> > > > > > In the isolated VR, I enabled ipv6 as:
> > > > > > net.ipv6.conf.all.disable_ipv6 = 0
> > > > > > net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
> > > > > > 1 net.ipv6.conf.all.accept_redirects = 1
> > > > > > net.ipv6.conf.all.autoconf = 1
> > > > > >
> > > > > > Set up a IPv6 nameserver/dns in /etc/resolve.conf And
> > > > > > configured the nics:
> > > > > > echo iface eth0 inet6 auto >> /etc/network/interfaces echo
> > > > > > iface eth2 inet6 auto >> /etc/network/interfaces
> > > > > > /etc/init.d/networking restart Next, restart ACS isolated
> > > > > > network without cleanup to have it
> > > > > reconfigure IPv4 nics, firewall, NAT etc
> > > > > >
> > > > > >    *
> > > > > > Next, I created a /64 network for the isolated guest network
> > > > > > on eth0
> > > of
> > > > > VR using radvd:
> > > > > >
> > > > > > # cat /etc/radvd.conf
> > > > > > interface eth0
> > > > > > {
> > > > > >      AdvSendAdvert on;
> > > > > >      MinRtrAdvInterval 5;
> > > > > >      MaxRtrAdvInterval 15;
> > > > > >      prefix 2001:470:ed36:1::/64
> > > > > >      {
> > > > > >          AdvOnLink on;
> > > > > >          AdvAutonomous on;
> > > > > >      };
> > > > > > };
> > > > > > systemctl restart radvd
> > > > > > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
> > > > > > this
> > > > > ...:1::/64 network
> > > > > >    *   Finally I added a static route in toy core-router for the
> new
> > > /64
> > > > > IPv6 range in the isolated network
> > > > > > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
> > > > > > eth2> dev
> > > > > <local lan nic>
> > > > > >    *
> > > > > > ... and I enabled firewall rules to allow any traffic to pass
> > > > > > for the
> > > > > new /64 network
> > > > > >
> > > > > > And voila all done! I create a domain AAAA record that points
> > > > > > to my
> > > > > guest VM IPv6 address a test webserver on
> > > > > > http://ipv6-isolated-ntwk-demo.yadav.cloud/
> > > > > >
> > > > > > (Note: I'll get rid of the tunnel and request a new /48 block
> > > > > > after a
> > > > > few days, sharing this solely for testing purposes)
> > > > > >
> > > > > >
> > > > > > Regards.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>

Re: IPV6 in Isolated/VPC networks

Posted by Rohit Yadav <ro...@shapeblue.com>.
Thanks Alex, Wei. I've updated the docs here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing

I'll leave the thread open for futher discussion/ideas/feedback. I think we've completed the phase1 design doc including all feedback comments for adding IPv6 support in CloudStack and some initial poc/work can be started. My colleagues and I will keep everyone posted on this thread and/or on a Github PR as and when we're able to start our work on the same (after 4.16, potentially towards 4.17).


Regards.

________________________________
From: Wei ZHOU <us...@gmail.com>
Sent: Friday, September 10, 2021 20:22
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks

Agree with Alex.
We only need to know how many /64 are allocated. We do not care how many
ipv6 addresses are used by VMs.

-Wei

On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <Al...@shapeblue.com>
wrote:

> Hi Rohit,
>
> I'd go for option 2, don't see a point tracking anything smaller than a
> /64 tbh.
>
> Cheers
> Alex
>
>
>
>
> -----Original Message-----
> From: Rohit Yadav <ro...@shapeblue.com>
> Sent: 09 September 2021 12:44
> To: dev@cloudstack.apache.org
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Thanks Alex, Kristaps. I've updated the design doc to reflect two
> agreements:
>
>   *   Allocate /64 for both isolated network and VPC tiers, no large
> allocation of prefixes to VPC (cons: more static routing rules for upstream
> router/admins)
>   *   All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a
> dedicated /64 prefix/block for systemvms
>
> The only outstanding question now is:
>
>   *   How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage
> for shared network (design, implementation and use-cases?)
> Option1: We don't do it, all user VMs nics have ipv4 address which whose
> usage we don't track. For public VR/nics/networks, we can simply add the
> IPv6 details for a related IPv4 address.
> Option2: Implement a separate, first-class IPv6 address or /64 prefix
> tracking/management and usage for all VMs and systemvms nic (this means
> account/domain level limits and new billing/records)
> Option3: other thoughts?
>
>
> Regards.
>
> ________________________________
> From: Alex Mattioli <Al...@shapeblue.com>
> Sent: Wednesday, September 8, 2021 23:24
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: RE: IPV6 in Isolated/VPC networks
>
> Hi Rohit, Kristaps,
>
> I'd say option 1 as well,  it does create a bit more overhead with static
> routes but if that's automated for a VPC it can also be easily automated
> for several tiers of a VPC.  We also don't constrain the amount of tiers in
> a  VPC.
> It has the added advantage to be closer to the desired behaviour with
> dynamic routing in the future, where a VPC VR can announce several subnets
> upstream.
>
> Cheers
> Alex
>
>
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: Rohit Yadav <ro...@shapeblue.com>
> Sent: 08 September 2021 19:04
> To: dev@cloudstack.apache.org
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Hi Kristaps,
>
> Thanks for sharing, I suppose that means individual tiers should be
> allocated /64 instead of larger ipv6 blocks to the whole VPC which could
> cause wastage.
>
> Any objection from anybody?
>
> Regards.
> ________________________________
> From: Kristaps Cudars <kr...@gmail.com>
> Sent: Wednesday, September 8, 2021 9:24:01 PM
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Hello,
>
> I asked networking team to comment on "How should the IPv6
> block/allocation work in VPCs?"
> Option1: They haven't seen lately devices with limits on how many static
> routes can be created.
> Option2: With /60 and /62 assignments and big quantity of routers IPv6
> assignment from RIPE NNC can be drained fast.
>
> /48 contains 64000 /64
> /60 contains 16 /64
> 64000 / 16 = 4000 routers
>
>
> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
> > All,
> >
> > After another iteration with Alex, I've updated the design doc. Kindly
> review:
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> > +Isolated+Network+and+VPC+with+Static+Routing
> >
> >
> > ... and advise on some outstanding questions:
> >
> >   *   How should the IPv6 block/allocation work in VPCs?
> > Option1: Should this be simply /64 allocation on any new tier, the
> > cons of this option is one static route/rule per VPC tier. (many
> > upstream routers may have limit on no. of static routes?)
> > Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
> (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks
> global setting (3 is default). The pros of this option are less no. of
> static route/rule and easy programming, but potential wastage of multiple
> /64 prefix blocks for unused/uncreated tiers.
> >   *   How do we manage IPv6 usage? Can anyone advise how we do IPv6
> usage for shared network (design, implementation and use-cases?)
> > Option1: We don't do it, all user VMs nics have ipv4 address which whose
> usage we don't track. For public VR/nics/networks, we can simply add the
> IPv6 details for a related IPv4 address.
> > Option2: Implement a separate, first-class IPv6 address or /64 prefix
> tracking/management and usage for all VMs and systemvms nic (this means
> account/domain level limits and new billing/records)
> >   *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by default
> if zone has a IPv6 address block allocated/assigned for use for systemvms
> (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons
> of this?)
> >   *
> >
> > Regards.
> >
> > ________________________________
> > From: Rohit Yadav <ro...@shapeblue.com>
> > Sent: Thursday, August 19, 2021 15:45
> > To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> > Subject: Re: IPV6 in Isolated/VPC networks
> >
> > Hi all,
> >
> > I've taken feedback from this thread and wrote this design doc:
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> > +Isolated+Network+and+VPC+with+Static+Routing
> >
> > Kindly review and advise if I missed anything or anything that needs to
> be changed/updated. You may comment on the wiki directly too.
> >
> > Kindly suggest your views on the following (also in the design doc
> above):
> >
> > Outstanding Questions:
> >
> >   *   Should admin or user be able to specify how VPC super CIDRs are
> created/needed; for example a user can ask for VPC with /60 super CIDR? Or
> should CloudStack automatically find/allocate a /64 for a new VPC tier from
> the root-admin configured /64-/48 block?
> >   *   Should we explore FRR and iBGP or other strategies to do dynamic
> routing which may not require advance/complex configuration in the VR or
> for the users/admin?
> >   *   With SLAAC and no dhcpv6, is there a way to support secondary IPv6
> addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's
> nics?
> >   *   Any thoughts on UI/UX for firewall/routing management?
> >   *   Any other feature/support for isolated network or VPC feature that
> must be explored or supported such as PF, VPN, LB, vpc static routes, vpc
> gateway etc.
> >   *   For usage - should we have any consideration, or should we assume
> that IPv4 and IPv6 address will go together for every nic; so IPv6 usage
> for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new
> biling/usage needed?
> >   *   For smoketests, local dev-test should we explore ULA? Unique Local
> Address - in the range fc00::/7. Typically only within the 'local' half
> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
> This prefix can be randomly generated at first install by CloudStack in a
> zone using zoneid etc?
> >   *   Should we expand support for VR diagnostic feature to include
> support for ipv6, traceroute6?
> >   *   Discuss - expand support/ability to allocate a /60, or /56 etc
> prefix to an individual VM in shared network (Wido's suggestion)
> >
> >
> > Regards.
> >
> > ________________________________
> > From: Wei ZHOU <us...@gmail.com>
> > Sent: Tuesday, August 17, 2021 21:16
> > To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> > Subject: Re: IPV6 in Isolated/VPC networks
> >
> > Thanks Kristaps, Wido, Rohit and Alex for your replies.
> >
> > I am fine with not having stateful dhcpv6 and privacy
> > extension/temporary address in phase 1. If community decides not to do
> > eventually , it is also ok to me.
> >
> > We could explore how to better use secondary ipv6 addresses as Wido
> > advised. It would be great if anyone share some user experience.
> >
> > -Wei
> >
> >
> > On Tuesday, 17 August 2021, Kristaps Cudars
> > <kr...@gmail.com>
> > wrote:
> >
> > > Hi Wei,
> > >
> > > Published this month's RFC 9099 and explains in different
> > > words/perspective what has been written by Alex, Rohit and Wido.
> > > https://www.rfc-editor.org/rfc/rfc9099.html
> > >
> > >
> > > On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
> > > > Hi Wido,
> > > >
> > > > (cc to Rohit and Alex)
> > > >
> > > > It is a good suggestion to use FRR for ipv6. The configuration is
> > > > quite simple and the VMs can get SLAAC, routes, etc.
> > > >
> > > > Privacy extension looks not the same as what you mentioned. see
> > > > https://datatracker.ietf.org/doc/html/rfc4941
> > > >
> > > > You are right. To use static routing, the admins need to configure
> > > > the routes in the upstream router, and add some ipv6 ranges (eg
> > > > /56 for VPCs and /64 for isolated networks) and their next-hop
> > > > (which will be configured in VRs) in CloudStack. CloudStack will
> > > > pick up an IPv6 range
> > > and
> > > > assign it to an isolated network or vpc. @Rohit, correct me if I'm
> wrong.
> > > >
> > > > I have a question, it looks stateless dhcpv6 (SLAAC from
> > > > router/VR, router/dns etc via RA messages) will be the only option
> > > > for now (related
> > > to
> > > > your pr https://github.com/apache/cloudstack/pull/3077) . Would it
> > > > be
> > > good
> > > > to provide stateful dhcpv6 (which can be implemented by dnsmasq)
> > > > as an option in cloudstack ? The advantages are
> > > > (1) support other ipv6 cidr sizes than /64.
> > > > (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
> > > > addresses can be changed
> > > > (4) an Ipv6 addresses can be re-used by multiple vms.
> > > > The problem is, stateful dhcpv6 does not support
> > > > routers,nameservers,
> > > etc.
> > > > we need to figure it out (probably use radvd/frr and dnsmasq both).
> > > >
> > > > -Wei
> > > >
> > > >
> > >
> >
> >
> >
> >
> >
> >
>
>
> 
 

> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > See my inline responses:
> > > > >
> > > > > Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> > > > > > Hi all,
> > > > > >
> > > > > > Thanks for your feedback and ideas, I've gone ahead with
> > > > > > discussing
> > > them
> > > > > with Alex and came up with a PoC/design which can be implemented
> > > > > in the following phases:
> > > > > >
> > > > > >    *   Phase1: implement ipv6 support in isolated networks and
> VPC
> > > with
> > > > > static routing
> > > > > >    *   Phase2: discuss and implement support for dynamic routing
> > > (TBD)
> > > > > >
> > > > > > For Phase1 here's the high-level proposal:
> > > > > >
> > > > > >    *   IPv6 address management:
> > > > > >       *   At the zone level root-admin specifies a /64 public
> range
> > > that
> > > > > will be used for VRs, then they can add a /48, or /56 IPv6 range
> > > > > for
> > > guest
> > > > > networks (to be used by isolated networks and VPC tiers)
> > > > > >       *   On creation of any IPv6 enabled isolated network or VPC
> > > tier,
> > > > > from the /48 or /56 block a /64 network is allocated/used
> > > > > >       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the
> > > zone
> > > > > (discuss: is privacy a concern, can privacy extensions rfc4941
> > > > > of
> > > slaac be
> > > > > explored?)
> > > > >
> > > > > Privacy Extensions are only a concern for client devices which
> > > > > roam between different IPv6 networks.
> > > > >
> > > > > If you IPv6 address of a client keeps the same suffix (MAC
> > > > > based) and switches network then only the prefix (/64) will change.
> > > > >
> > > > > This way a network like Google, Facebook, etc could track your
> > > > > device moving from network to network if they only look at the
> > > > > last 64-bits of the IPv6 address.
> > > > >
> > > > > For servers this is not a problem as you already know in which
> > > > > network they are.
> > > > >
> > > > > >    *   Network offerings: root-admin can create new network
> offerings
> > > > > (with VPC too) that specifies a network stack option:
> > > > > >       *   ipv4 only (default, for backward compatibility all
> > > > > networks/offerings post-upgrade migrate to this option)
> > > > > >       *   ipv4-and-ipv6
> > > > > >       *   ipv6-only (this can be phase 1.b)
> > > > > >       *   A new routing option: static (phase1), dynamic (phase2,
> > > with
> > > > > multiple sub-options such as ospf/bgp etc...)
> > > > >
> > > > > This means that the network admin will need to statically route
> > > > > the
> > > IPv6
> > > > > subnet to the VR's outside IPv6 address, for example, on a JunOS
> > > router:
> > > > >
> > > > > set routing-options rib inet6.0 static route 2001:db8:500::/48
> > > > > next-hop
> > > > > 2001:db8:100::50
> > > > >
> > > > > I'm assuming that 2001:db8:100::50 is the address of the VR on
> > > > > the outside (/64) network. In reality this will probably be a
> > > > > longer address, but this is for just the example.
> > > > >
> > > > > >    *   VR changes:
> > > > > >       *   VR gets its guest and public nics set to inet6 auto
> > > > > >       *   For each /64 allocated to guest network and VPC tiers,
> > > radvd
> > > > > is configured to do RA
> > > > >
> > > > > radvd is fine, but looking at phase 2 with dynamic routing you
> > > > > might already want to look into FRRouting. FRR can also
> > > > > advertise RAs while not doing any routing.
> > > > >
> > > > > interface ens4
> > > > >    no ipv6 nd suppress-ra
> > > > >    ipv6 nd prefix 2001:db8:500::/64
> > > > >    ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> > > > >
> > > > > See: http://docs.frrouting.org/en/latest/ipv6.html
> > > > >
> > > > > >       *   Firewall: a new ipv6 zone/chain is created for ipv6
> where
> > > ipv6
> > > > > firewall rules (ACLs, ingress, egress) are implemented; ACLs
> > > > > between
> > > VPC
> > > > > tiers are managed/implemented by ipv6 firewall on VR
> > > > >
> > > > > Please take a look at the existing security_group.py script
> > > > > which implements RFC4890
> > > > >
> > > > > https://datatracker.ietf.org/doc/html/rfc4890
> > > > >
> > > > > ICMPv6 is a vital part of IPv6 and certain packets should always
> > > > > be allowed.
> > > > >
> > > > > >       *   It is assumed that static routes are created on the
> > > core/main
> > > > > router by the admin or automated using some scripts/tools; for
> > > > > this CloudStack will announce events with details of /64
> > > > > networks and VR's public IPv6 address that can be consumed by a
> > > > > rabbitmq/message bus
> > > client
> > > > > (for example), or a custom cron job or script as part of
> orchestration.
> > > > > (this wouldn't be necessary for dynamic routing bgp with
> > > > > phase2)\\
> > > > >
> > > > > You would only need to announce the /48 or /56 allocated to the
> > > > > VR, that's all. You don't need to inform the upstream router
> > > > > about the /64 subnets created within that larger subnet.
> > > > >
> > > > > >    *   Guest Networking: With SLAAC, it's easy for CloudStack to
> > > > > calculate allocate and use a /64 and determine the IPv6 address
> > > > > of VR
> > > nics
> > > > > and guest VM nics
> > > > > >       *   A user create an isolated network/VPC with an offering
> > > that is
> > > > > ipv6 enabled
> > > > > >       *   A user can manage firewall for the IPv6 address/guest
> nics;
> > > > > there'll be no port forward and LB feature though for IPv6
> > > > > >       *   A users can run workloads in the guest VMs that listen
> on
> > > > > publically routable ipv6 addresses
> > > > > >       *   Usage/billing etc continue to work, no change needed
> > > > > >
> > > > > > Network layout:
> > > > > >
> > > > > > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
> > > > > > VLAN] ->
> > > > > [guest VMs/nics]
> > > > > > *core/ISP router needs static routes to be added (manually or
> > > > > automated), assumes a /48 or /56 configured for the zone
> > > > > >
> > > > > > Thoughts, feedback?
> > > > >
> > > > > Looks doable!
> > > > >
> > > > > Side-note: It would be very cool if you could use parts of this
> > > > > implementation to also route /48, /56, or /60 subnets to
> > > > > individual VMs in Shared networks.
> > > > >
> > > > > Why? This allows for running Docker containers with native IPv6
> > > > > inside the VM or running a (Open)VPN server inside a VM which
> > > > > then assigns public IPv6 addresses to clients connected.
> > > > >
> > > > > Instead of routing the subnet to a VR we route the subnet to a
> > > > > single instance in a shared network.
> > > > >
> > > > > If we could then also move these subnets between Instances
> > > > > easily one can quickly migrate to a different instance while
> > > > > keeping the same IPv6 subnet.
> > > > >
> > > > > Wido
> > > > >
> > > > > >
> > > > > > Proof-of-concept commentary: here's what I did to test the idea:
> > > > > >
> > > > > >    *   Created an isolated network and deployed a VM in my home
> lab
> > > > > > The VR running on KVM has following nics
> > > > > > eth0 - guest network
> > > > > > eth1 - link local
> > > > > > eth2 - public network
> > > > > >
> > > > > >    *   I setup a custom openwrt router on a RPi4 to serve as a
> > > toy-core
> > > > > router where I create a wan6 IPv6 tunnel using tunnel broker and
> > > > > I got
> > > a
> > > > > /48 allocated. My configuration looks like:
> > > > > > /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> > > > > > /64 - 2001:470:36:3e2::/64 (default allocated by)
> > > > > >
> > > > > > I create a LAN ipv6 (public network for CloudStack VR): at
> > > subnet/prefix
> > > > > 0:
> > > > > > LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
> > > > > > SLAAC+stateless DHCP (no dhcpv6)
> > > > > >    *
> > > > > >    *
> > > > > > In the isolated VR, I enabled ipv6 as:
> > > > > > net.ipv6.conf.all.disable_ipv6 = 0
> > > > > > net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
> > > > > > 1 net.ipv6.conf.all.accept_redirects = 1
> > > > > > net.ipv6.conf.all.autoconf = 1
> > > > > >
> > > > > > Set up a IPv6 nameserver/dns in /etc/resolve.conf And
> > > > > > configured the nics:
> > > > > > echo iface eth0 inet6 auto >> /etc/network/interfaces echo
> > > > > > iface eth2 inet6 auto >> /etc/network/interfaces
> > > > > > /etc/init.d/networking restart Next, restart ACS isolated
> > > > > > network without cleanup to have it
> > > > > reconfigure IPv4 nics, firewall, NAT etc
> > > > > >
> > > > > >    *
> > > > > > Next, I created a /64 network for the isolated guest network
> > > > > > on eth0
> > > of
> > > > > VR using radvd:
> > > > > >
> > > > > > # cat /etc/radvd.conf
> > > > > > interface eth0
> > > > > > {
> > > > > >      AdvSendAdvert on;
> > > > > >      MinRtrAdvInterval 5;
> > > > > >      MaxRtrAdvInterval 15;
> > > > > >      prefix 2001:470:ed36:1::/64
> > > > > >      {
> > > > > >          AdvOnLink on;
> > > > > >          AdvAutonomous on;
> > > > > >      };
> > > > > > };
> > > > > > systemctl restart radvd
> > > > > > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
> > > > > > this
> > > > > ...:1::/64 network
> > > > > >    *   Finally I added a static route in toy core-router for the
> new
> > > /64
> > > > > IPv6 range in the isolated network
> > > > > > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
> > > > > > eth2> dev
> > > > > <local lan nic>
> > > > > >    *
> > > > > > ... and I enabled firewall rules to allow any traffic to pass
> > > > > > for the
> > > > > new /64 network
> > > > > >
> > > > > > And voila all done! I create a domain AAAA record that points
> > > > > > to my
> > > > > guest VM IPv6 address a test webserver on
> > > > > > http://ipv6-isolated-ntwk-demo.yadav.cloud/
> > > > > >
> > > > > > (Note: I'll get rid of the tunnel and request a new /48 block
> > > > > > after a
> > > > > few days, sharing this solely for testing purposes)
> > > > > >
> > > > > >
> > > > > > Regards.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>

Re: IPV6 in Isolated/VPC networks

Posted by Wei ZHOU <us...@gmail.com>.
Agree with Alex.
We only need to know how many /64 are allocated. We do not care how many
ipv6 addresses are used by VMs.

-Wei

On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <Al...@shapeblue.com>
wrote:

> Hi Rohit,
>
> I'd go for option 2, don't see a point tracking anything smaller than a
> /64 tbh.
>
> Cheers
> Alex
>
>
>
>
> -----Original Message-----
> From: Rohit Yadav <ro...@shapeblue.com>
> Sent: 09 September 2021 12:44
> To: dev@cloudstack.apache.org
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Thanks Alex, Kristaps. I've updated the design doc to reflect two
> agreements:
>
>   *   Allocate /64 for both isolated network and VPC tiers, no large
> allocation of prefixes to VPC (cons: more static routing rules for upstream
> router/admins)
>   *   All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a
> dedicated /64 prefix/block for systemvms
>
> The only outstanding question now is:
>
>   *   How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage
> for shared network (design, implementation and use-cases?)
> Option1: We don't do it, all user VMs nics have ipv4 address which whose
> usage we don't track. For public VR/nics/networks, we can simply add the
> IPv6 details for a related IPv4 address.
> Option2: Implement a separate, first-class IPv6 address or /64 prefix
> tracking/management and usage for all VMs and systemvms nic (this means
> account/domain level limits and new billing/records)
> Option3: other thoughts?
>
>
> Regards.
>
> ________________________________
> From: Alex Mattioli <Al...@shapeblue.com>
> Sent: Wednesday, September 8, 2021 23:24
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: RE: IPV6 in Isolated/VPC networks
>
> Hi Rohit, Kristaps,
>
> I'd say option 1 as well,  it does create a bit more overhead with static
> routes but if that's automated for a VPC it can also be easily automated
> for several tiers of a VPC.  We also don't constrain the amount of tiers in
> a  VPC.
> It has the added advantage to be closer to the desired behaviour with
> dynamic routing in the future, where a VPC VR can announce several subnets
> upstream.
>
> Cheers
> Alex
>
>
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: Rohit Yadav <ro...@shapeblue.com>
> Sent: 08 September 2021 19:04
> To: dev@cloudstack.apache.org
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Hi Kristaps,
>
> Thanks for sharing, I suppose that means individual tiers should be
> allocated /64 instead of larger ipv6 blocks to the whole VPC which could
> cause wastage.
>
> Any objection from anybody?
>
> Regards.
> ________________________________
> From: Kristaps Cudars <kr...@gmail.com>
> Sent: Wednesday, September 8, 2021 9:24:01 PM
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Hello,
>
> I asked networking team to comment on "How should the IPv6
> block/allocation work in VPCs?"
> Option1: They haven't seen lately devices with limits on how many static
> routes can be created.
> Option2: With /60 and /62 assignments and big quantity of routers IPv6
> assignment from RIPE NNC can be drained fast.
>
> /48 contains 64000 /64
> /60 contains 16 /64
> 64000 / 16 = 4000 routers
>
>
> On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
> > All,
> >
> > After another iteration with Alex, I've updated the design doc. Kindly
> review:
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> > +Isolated+Network+and+VPC+with+Static+Routing
> >
> >
> > ... and advise on some outstanding questions:
> >
> >   *   How should the IPv6 block/allocation work in VPCs?
> > Option1: Should this be simply /64 allocation on any new tier, the
> > cons of this option is one static route/rule per VPC tier. (many
> > upstream routers may have limit on no. of static routes?)
> > Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62
> (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks
> global setting (3 is default). The pros of this option are less no. of
> static route/rule and easy programming, but potential wastage of multiple
> /64 prefix blocks for unused/uncreated tiers.
> >   *   How do we manage IPv6 usage? Can anyone advise how we do IPv6
> usage for shared network (design, implementation and use-cases?)
> > Option1: We don't do it, all user VMs nics have ipv4 address which whose
> usage we don't track. For public VR/nics/networks, we can simply add the
> IPv6 details for a related IPv4 address.
> > Option2: Implement a separate, first-class IPv6 address or /64 prefix
> tracking/management and usage for all VMs and systemvms nic (this means
> account/domain level limits and new billing/records)
> >   *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by default
> if zone has a IPv6 address block allocated/assigned for use for systemvms
> (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons
> of this?)
> >   *
> >
> > Regards.
> >
> > ________________________________
> > From: Rohit Yadav <ro...@shapeblue.com>
> > Sent: Thursday, August 19, 2021 15:45
> > To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> > Subject: Re: IPV6 in Isolated/VPC networks
> >
> > Hi all,
> >
> > I've taken feedback from this thread and wrote this design doc:
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> > +Isolated+Network+and+VPC+with+Static+Routing
> >
> > Kindly review and advise if I missed anything or anything that needs to
> be changed/updated. You may comment on the wiki directly too.
> >
> > Kindly suggest your views on the following (also in the design doc
> above):
> >
> > Outstanding Questions:
> >
> >   *   Should admin or user be able to specify how VPC super CIDRs are
> created/needed; for example a user can ask for VPC with /60 super CIDR? Or
> should CloudStack automatically find/allocate a /64 for a new VPC tier from
> the root-admin configured /64-/48 block?
> >   *   Should we explore FRR and iBGP or other strategies to do dynamic
> routing which may not require advance/complex configuration in the VR or
> for the users/admin?
> >   *   With SLAAC and no dhcpv6, is there a way to support secondary IPv6
> addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's
> nics?
> >   *   Any thoughts on UI/UX for firewall/routing management?
> >   *   Any other feature/support for isolated network or VPC feature that
> must be explored or supported such as PF, VPN, LB, vpc static routes, vpc
> gateway etc.
> >   *   For usage - should we have any consideration, or should we assume
> that IPv4 and IPv6 address will go together for every nic; so IPv6 usage
> for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new
> biling/usage needed?
> >   *   For smoketests, local dev-test should we explore ULA? Unique Local
> Address - in the range fc00::/7. Typically only within the 'local' half
> fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing.
> This prefix can be randomly generated at first install by CloudStack in a
> zone using zoneid etc?
> >   *   Should we expand support for VR diagnostic feature to include
> support for ipv6, traceroute6?
> >   *   Discuss - expand support/ability to allocate a /60, or /56 etc
> prefix to an individual VM in shared network (Wido's suggestion)
> >
> >
> > Regards.
> >
> > ________________________________
> > From: Wei ZHOU <us...@gmail.com>
> > Sent: Tuesday, August 17, 2021 21:16
> > To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> > Subject: Re: IPV6 in Isolated/VPC networks
> >
> > Thanks Kristaps, Wido, Rohit and Alex for your replies.
> >
> > I am fine with not having stateful dhcpv6 and privacy
> > extension/temporary address in phase 1. If community decides not to do
> > eventually , it is also ok to me.
> >
> > We could explore how to better use secondary ipv6 addresses as Wido
> > advised. It would be great if anyone share some user experience.
> >
> > -Wei
> >
> >
> > On Tuesday, 17 August 2021, Kristaps Cudars
> > <kr...@gmail.com>
> > wrote:
> >
> > > Hi Wei,
> > >
> > > Published this month's RFC 9099 and explains in different
> > > words/perspective what has been written by Alex, Rohit and Wido.
> > > https://www.rfc-editor.org/rfc/rfc9099.html
> > >
> > >
> > > On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
> > > > Hi Wido,
> > > >
> > > > (cc to Rohit and Alex)
> > > >
> > > > It is a good suggestion to use FRR for ipv6. The configuration is
> > > > quite simple and the VMs can get SLAAC, routes, etc.
> > > >
> > > > Privacy extension looks not the same as what you mentioned. see
> > > > https://datatracker.ietf.org/doc/html/rfc4941
> > > >
> > > > You are right. To use static routing, the admins need to configure
> > > > the routes in the upstream router, and add some ipv6 ranges (eg
> > > > /56 for VPCs and /64 for isolated networks) and their next-hop
> > > > (which will be configured in VRs) in CloudStack. CloudStack will
> > > > pick up an IPv6 range
> > > and
> > > > assign it to an isolated network or vpc. @Rohit, correct me if I'm
> wrong.
> > > >
> > > > I have a question, it looks stateless dhcpv6 (SLAAC from
> > > > router/VR, router/dns etc via RA messages) will be the only option
> > > > for now (related
> > > to
> > > > your pr https://github.com/apache/cloudstack/pull/3077) . Would it
> > > > be
> > > good
> > > > to provide stateful dhcpv6 (which can be implemented by dnsmasq)
> > > > as an option in cloudstack ? The advantages are
> > > > (1) support other ipv6 cidr sizes than /64.
> > > > (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
> > > > addresses can be changed
> > > > (4) an Ipv6 addresses can be re-used by multiple vms.
> > > > The problem is, stateful dhcpv6 does not support
> > > > routers,nameservers,
> > > etc.
> > > > we need to figure it out (probably use radvd/frr and dnsmasq both).
> > > >
> > > > -Wei
> > > >
> > > >
> > >
> >
> >
> >
> >
> >
> >
>
>
> > On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > See my inline responses:
> > > > >
> > > > > Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> > > > > > Hi all,
> > > > > >
> > > > > > Thanks for your feedback and ideas, I've gone ahead with
> > > > > > discussing
> > > them
> > > > > with Alex and came up with a PoC/design which can be implemented
> > > > > in the following phases:
> > > > > >
> > > > > >    *   Phase1: implement ipv6 support in isolated networks and
> VPC
> > > with
> > > > > static routing
> > > > > >    *   Phase2: discuss and implement support for dynamic routing
> > > (TBD)
> > > > > >
> > > > > > For Phase1 here's the high-level proposal:
> > > > > >
> > > > > >    *   IPv6 address management:
> > > > > >       *   At the zone level root-admin specifies a /64 public
> range
> > > that
> > > > > will be used for VRs, then they can add a /48, or /56 IPv6 range
> > > > > for
> > > guest
> > > > > networks (to be used by isolated networks and VPC tiers)
> > > > > >       *   On creation of any IPv6 enabled isolated network or VPC
> > > tier,
> > > > > from the /48 or /56 block a /64 network is allocated/used
> > > > > >       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the
> > > zone
> > > > > (discuss: is privacy a concern, can privacy extensions rfc4941
> > > > > of
> > > slaac be
> > > > > explored?)
> > > > >
> > > > > Privacy Extensions are only a concern for client devices which
> > > > > roam between different IPv6 networks.
> > > > >
> > > > > If you IPv6 address of a client keeps the same suffix (MAC
> > > > > based) and switches network then only the prefix (/64) will change.
> > > > >
> > > > > This way a network like Google, Facebook, etc could track your
> > > > > device moving from network to network if they only look at the
> > > > > last 64-bits of the IPv6 address.
> > > > >
> > > > > For servers this is not a problem as you already know in which
> > > > > network they are.
> > > > >
> > > > > >    *   Network offerings: root-admin can create new network
> offerings
> > > > > (with VPC too) that specifies a network stack option:
> > > > > >       *   ipv4 only (default, for backward compatibility all
> > > > > networks/offerings post-upgrade migrate to this option)
> > > > > >       *   ipv4-and-ipv6
> > > > > >       *   ipv6-only (this can be phase 1.b)
> > > > > >       *   A new routing option: static (phase1), dynamic (phase2,
> > > with
> > > > > multiple sub-options such as ospf/bgp etc...)
> > > > >
> > > > > This means that the network admin will need to statically route
> > > > > the
> > > IPv6
> > > > > subnet to the VR's outside IPv6 address, for example, on a JunOS
> > > router:
> > > > >
> > > > > set routing-options rib inet6.0 static route 2001:db8:500::/48
> > > > > next-hop
> > > > > 2001:db8:100::50
> > > > >
> > > > > I'm assuming that 2001:db8:100::50 is the address of the VR on
> > > > > the outside (/64) network. In reality this will probably be a
> > > > > longer address, but this is for just the example.
> > > > >
> > > > > >    *   VR changes:
> > > > > >       *   VR gets its guest and public nics set to inet6 auto
> > > > > >       *   For each /64 allocated to guest network and VPC tiers,
> > > radvd
> > > > > is configured to do RA
> > > > >
> > > > > radvd is fine, but looking at phase 2 with dynamic routing you
> > > > > might already want to look into FRRouting. FRR can also
> > > > > advertise RAs while not doing any routing.
> > > > >
> > > > > interface ens4
> > > > >    no ipv6 nd suppress-ra
> > > > >    ipv6 nd prefix 2001:db8:500::/64
> > > > >    ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> > > > >
> > > > > See: http://docs.frrouting.org/en/latest/ipv6.html
> > > > >
> > > > > >       *   Firewall: a new ipv6 zone/chain is created for ipv6
> where
> > > ipv6
> > > > > firewall rules (ACLs, ingress, egress) are implemented; ACLs
> > > > > between
> > > VPC
> > > > > tiers are managed/implemented by ipv6 firewall on VR
> > > > >
> > > > > Please take a look at the existing security_group.py script
> > > > > which implements RFC4890
> > > > >
> > > > > https://datatracker.ietf.org/doc/html/rfc4890
> > > > >
> > > > > ICMPv6 is a vital part of IPv6 and certain packets should always
> > > > > be allowed.
> > > > >
> > > > > >       *   It is assumed that static routes are created on the
> > > core/main
> > > > > router by the admin or automated using some scripts/tools; for
> > > > > this CloudStack will announce events with details of /64
> > > > > networks and VR's public IPv6 address that can be consumed by a
> > > > > rabbitmq/message bus
> > > client
> > > > > (for example), or a custom cron job or script as part of
> orchestration.
> > > > > (this wouldn't be necessary for dynamic routing bgp with
> > > > > phase2)\\
> > > > >
> > > > > You would only need to announce the /48 or /56 allocated to the
> > > > > VR, that's all. You don't need to inform the upstream router
> > > > > about the /64 subnets created within that larger subnet.
> > > > >
> > > > > >    *   Guest Networking: With SLAAC, it's easy for CloudStack to
> > > > > calculate allocate and use a /64 and determine the IPv6 address
> > > > > of VR
> > > nics
> > > > > and guest VM nics
> > > > > >       *   A user create an isolated network/VPC with an offering
> > > that is
> > > > > ipv6 enabled
> > > > > >       *   A user can manage firewall for the IPv6 address/guest
> nics;
> > > > > there'll be no port forward and LB feature though for IPv6
> > > > > >       *   A users can run workloads in the guest VMs that listen
> on
> > > > > publically routable ipv6 addresses
> > > > > >       *   Usage/billing etc continue to work, no change needed
> > > > > >
> > > > > > Network layout:
> > > > > >
> > > > > > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
> > > > > > VLAN] ->
> > > > > [guest VMs/nics]
> > > > > > *core/ISP router needs static routes to be added (manually or
> > > > > automated), assumes a /48 or /56 configured for the zone
> > > > > >
> > > > > > Thoughts, feedback?
> > > > >
> > > > > Looks doable!
> > > > >
> > > > > Side-note: It would be very cool if you could use parts of this
> > > > > implementation to also route /48, /56, or /60 subnets to
> > > > > individual VMs in Shared networks.
> > > > >
> > > > > Why? This allows for running Docker containers with native IPv6
> > > > > inside the VM or running a (Open)VPN server inside a VM which
> > > > > then assigns public IPv6 addresses to clients connected.
> > > > >
> > > > > Instead of routing the subnet to a VR we route the subnet to a
> > > > > single instance in a shared network.
> > > > >
> > > > > If we could then also move these subnets between Instances
> > > > > easily one can quickly migrate to a different instance while
> > > > > keeping the same IPv6 subnet.
> > > > >
> > > > > Wido
> > > > >
> > > > > >
> > > > > > Proof-of-concept commentary: here's what I did to test the idea:
> > > > > >
> > > > > >    *   Created an isolated network and deployed a VM in my home
> lab
> > > > > > The VR running on KVM has following nics
> > > > > > eth0 - guest network
> > > > > > eth1 - link local
> > > > > > eth2 - public network
> > > > > >
> > > > > >    *   I setup a custom openwrt router on a RPi4 to serve as a
> > > toy-core
> > > > > router where I create a wan6 IPv6 tunnel using tunnel broker and
> > > > > I got
> > > a
> > > > > /48 allocated. My configuration looks like:
> > > > > > /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> > > > > > /64 - 2001:470:36:3e2::/64 (default allocated by)
> > > > > >
> > > > > > I create a LAN ipv6 (public network for CloudStack VR): at
> > > subnet/prefix
> > > > > 0:
> > > > > > LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
> > > > > > SLAAC+stateless DHCP (no dhcpv6)
> > > > > >    *
> > > > > >    *
> > > > > > In the isolated VR, I enabled ipv6 as:
> > > > > > net.ipv6.conf.all.disable_ipv6 = 0
> > > > > > net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
> > > > > > 1 net.ipv6.conf.all.accept_redirects = 1
> > > > > > net.ipv6.conf.all.autoconf = 1
> > > > > >
> > > > > > Set up a IPv6 nameserver/dns in /etc/resolve.conf And
> > > > > > configured the nics:
> > > > > > echo iface eth0 inet6 auto >> /etc/network/interfaces echo
> > > > > > iface eth2 inet6 auto >> /etc/network/interfaces
> > > > > > /etc/init.d/networking restart Next, restart ACS isolated
> > > > > > network without cleanup to have it
> > > > > reconfigure IPv4 nics, firewall, NAT etc
> > > > > >
> > > > > >    *
> > > > > > Next, I created a /64 network for the isolated guest network
> > > > > > on eth0
> > > of
> > > > > VR using radvd:
> > > > > >
> > > > > > # cat /etc/radvd.conf
> > > > > > interface eth0
> > > > > > {
> > > > > >      AdvSendAdvert on;
> > > > > >      MinRtrAdvInterval 5;
> > > > > >      MaxRtrAdvInterval 15;
> > > > > >      prefix 2001:470:ed36:1::/64
> > > > > >      {
> > > > > >          AdvOnLink on;
> > > > > >          AdvAutonomous on;
> > > > > >      };
> > > > > > };
> > > > > > systemctl restart radvd
> > > > > > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
> > > > > > this
> > > > > ...:1::/64 network
> > > > > >    *   Finally I added a static route in toy core-router for the
> new
> > > /64
> > > > > IPv6 range in the isolated network
> > > > > > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
> > > > > > eth2> dev
> > > > > <local lan nic>
> > > > > >    *
> > > > > > ... and I enabled firewall rules to allow any traffic to pass
> > > > > > for the
> > > > > new /64 network
> > > > > >
> > > > > > And voila all done! I create a domain AAAA record that points
> > > > > > to my
> > > > > guest VM IPv6 address a test webserver on
> > > > > > http://ipv6-isolated-ntwk-demo.yadav.cloud/
> > > > > >
> > > > > > (Note: I'll get rid of the tunnel and request a new /48 block
> > > > > > after a
> > > > > few days, sharing this solely for testing purposes)
> > > > > >
> > > > > >
> > > > > > Regards.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>

RE: IPV6 in Isolated/VPC networks

Posted by Alex Mattioli <Al...@shapeblue.com>.
Hi Rohit,

I'd go for option 2, don't see a point tracking anything smaller than a /64 tbh.

Cheers
Alex

 


-----Original Message-----
From: Rohit Yadav <ro...@shapeblue.com> 
Sent: 09 September 2021 12:44
To: dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks

Thanks Alex, Kristaps. I've updated the design doc to reflect two agreements:

  *   Allocate /64 for both isolated network and VPC tiers, no large allocation of prefixes to VPC (cons: more static routing rules for upstream router/admins)
  *   All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a dedicated /64 prefix/block for systemvms

The only outstanding question now is:

  *   How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage for shared network (design, implementation and use-cases?)
Option1: We don't do it, all user VMs nics have ipv4 address which whose usage we don't track. For public VR/nics/networks, we can simply add the IPv6 details for a related IPv4 address.
Option2: Implement a separate, first-class IPv6 address or /64 prefix tracking/management and usage for all VMs and systemvms nic (this means account/domain level limits and new billing/records)
Option3: other thoughts?


Regards.

________________________________
From: Alex Mattioli <Al...@shapeblue.com>
Sent: Wednesday, September 8, 2021 23:24
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: RE: IPV6 in Isolated/VPC networks

Hi Rohit, Kristaps,

I'd say option 1 as well,  it does create a bit more overhead with static routes but if that's automated for a VPC it can also be easily automated for several tiers of a VPC.  We also don't constrain the amount of tiers in a  VPC.
It has the added advantage to be closer to the desired behaviour with dynamic routing in the future, where a VPC VR can announce several subnets upstream.

Cheers
Alex







 


-----Original Message-----
From: Rohit Yadav <ro...@shapeblue.com>
Sent: 08 September 2021 19:04
To: dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks

Hi Kristaps,

Thanks for sharing, I suppose that means individual tiers should be allocated /64 instead of larger ipv6 blocks to the whole VPC which could cause wastage.

Any objection from anybody?

Regards.
________________________________
From: Kristaps Cudars <kr...@gmail.com>
Sent: Wednesday, September 8, 2021 9:24:01 PM
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks

Hello,

I asked networking team to comment on "How should the IPv6 block/allocation work in VPCs?"
Option1: They haven't seen lately devices with limits on how many static routes can be created.
Option2: With /60 and /62 assignments and big quantity of routers IPv6 assignment from RIPE NNC can be drained fast.

/48 contains 64000 /64
/60 contains 16 /64
64000 / 16 = 4000 routers


On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
> All,
>
> After another iteration with Alex, I've updated the design doc. Kindly review:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> +Isolated+Network+and+VPC+with+Static+Routing
>
>
> ... and advise on some outstanding questions:
>
>   *   How should the IPv6 block/allocation work in VPCs?
> Option1: Should this be simply /64 allocation on any new tier, the 
> cons of this option is one static route/rule per VPC tier. (many 
> upstream routers may have limit on no. of static routes?)
> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62 (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks global setting (3 is default). The pros of this option are less no. of static route/rule and easy programming, but potential wastage of multiple /64 prefix blocks for unused/uncreated tiers.
>   *   How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage for shared network (design, implementation and use-cases?)
> Option1: We don't do it, all user VMs nics have ipv4 address which whose usage we don't track. For public VR/nics/networks, we can simply add the IPv6 details for a related IPv4 address.
> Option2: Implement a separate, first-class IPv6 address or /64 prefix tracking/management and usage for all VMs and systemvms nic (this means account/domain level limits and new billing/records)
>   *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by default if zone has a IPv6 address block allocated/assigned for use for systemvms (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons of this?)
>   *
>
> Regards.
>
> ________________________________
> From: Rohit Yadav <ro...@shapeblue.com>
> Sent: Thursday, August 19, 2021 15:45
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Hi all,
>
> I've taken feedback from this thread and wrote this design doc:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> +Isolated+Network+and+VPC+with+Static+Routing
>
> Kindly review and advise if I missed anything or anything that needs to be changed/updated. You may comment on the wiki directly too.
>
> Kindly suggest your views on the following (also in the design doc above):
>
> Outstanding Questions:
>
>   *   Should admin or user be able to specify how VPC super CIDRs are created/needed; for example a user can ask for VPC with /60 super CIDR? Or should CloudStack automatically find/allocate a /64 for a new VPC tier from the root-admin configured /64-/48 block?
>   *   Should we explore FRR and iBGP or other strategies to do dynamic routing which may not require advance/complex configuration in the VR or for the users/admin?
>   *   With SLAAC and no dhcpv6, is there a way to support secondary IPv6 addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's nics?
>   *   Any thoughts on UI/UX for firewall/routing management?
>   *   Any other feature/support for isolated network or VPC feature that must be explored or supported such as PF, VPN, LB, vpc static routes, vpc gateway etc.
>   *   For usage - should we have any consideration, or should we assume that IPv4 and IPv6 address will go together for every nic; so IPv6 usage for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new biling/usage needed?
>   *   For smoketests, local dev-test should we explore ULA? Unique Local Address - in the range fc00::/7. Typically only within the 'local' half fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing. This prefix can be randomly generated at first install by CloudStack in a zone using zoneid etc?
>   *   Should we expand support for VR diagnostic feature to include support for ipv6, traceroute6?
>   *   Discuss - expand support/ability to allocate a /60, or /56 etc prefix to an individual VM in shared network (Wido's suggestion)
>
>
> Regards.
>
> ________________________________
> From: Wei ZHOU <us...@gmail.com>
> Sent: Tuesday, August 17, 2021 21:16
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>
> I am fine with not having stateful dhcpv6 and privacy 
> extension/temporary address in phase 1. If community decides not to do 
> eventually , it is also ok to me.
>
> We could explore how to better use secondary ipv6 addresses as Wido 
> advised. It would be great if anyone share some user experience.
>
> -Wei
>
>
> On Tuesday, 17 August 2021, Kristaps Cudars 
> <kr...@gmail.com>
> wrote:
>
> > Hi Wei,
> >
> > Published this month's RFC 9099 and explains in different 
> > words/perspective what has been written by Alex, Rohit and Wido.
> > https://www.rfc-editor.org/rfc/rfc9099.html
> >
> >
> > On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
> > > Hi Wido,
> > >
> > > (cc to Rohit and Alex)
> > >
> > > It is a good suggestion to use FRR for ipv6. The configuration is 
> > > quite simple and the VMs can get SLAAC, routes, etc.
> > >
> > > Privacy extension looks not the same as what you mentioned. see
> > > https://datatracker.ietf.org/doc/html/rfc4941
> > >
> > > You are right. To use static routing, the admins need to configure 
> > > the routes in the upstream router, and add some ipv6 ranges (eg
> > > /56 for VPCs and /64 for isolated networks) and their next-hop 
> > > (which will be configured in VRs) in CloudStack. CloudStack will 
> > > pick up an IPv6 range
> > and
> > > assign it to an isolated network or vpc. @Rohit, correct me if I'm wrong.
> > >
> > > I have a question, it looks stateless dhcpv6 (SLAAC from 
> > > router/VR, router/dns etc via RA messages) will be the only option 
> > > for now (related
> > to
> > > your pr https://github.com/apache/cloudstack/pull/3077) . Would it 
> > > be
> > good
> > > to provide stateful dhcpv6 (which can be implemented by dnsmasq) 
> > > as an option in cloudstack ? The advantages are
> > > (1) support other ipv6 cidr sizes than /64.
> > > (2) we can assign a specified Ipv6 address to a vm. vm Ipv6 
> > > addresses can be changed
> > > (4) an Ipv6 addresses can be re-used by multiple vms.
> > > The problem is, stateful dhcpv6 does not support 
> > > routers,nameservers,
> > etc.
> > > we need to figure it out (probably use radvd/frr and dnsmasq both).
> > >
> > > -Wei
> > >
> > >
> >
>
>
>
>
>
>


> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
> > >
> > > > Hi,
> > > >
> > > > See my inline responses:
> > > >
> > > > Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> > > > > Hi all,
> > > > >
> > > > > Thanks for your feedback and ideas, I've gone ahead with 
> > > > > discussing
> > them
> > > > with Alex and came up with a PoC/design which can be implemented 
> > > > in the following phases:
> > > > >
> > > > >    *   Phase1: implement ipv6 support in isolated networks and VPC
> > with
> > > > static routing
> > > > >    *   Phase2: discuss and implement support for dynamic routing
> > (TBD)
> > > > >
> > > > > For Phase1 here's the high-level proposal:
> > > > >
> > > > >    *   IPv6 address management:
> > > > >       *   At the zone level root-admin specifies a /64 public range
> > that
> > > > will be used for VRs, then they can add a /48, or /56 IPv6 range 
> > > > for
> > guest
> > > > networks (to be used by isolated networks and VPC tiers)
> > > > >       *   On creation of any IPv6 enabled isolated network or VPC
> > tier,
> > > > from the /48 or /56 block a /64 network is allocated/used
> > > > >       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the
> > zone
> > > > (discuss: is privacy a concern, can privacy extensions rfc4941 
> > > > of
> > slaac be
> > > > explored?)
> > > >
> > > > Privacy Extensions are only a concern for client devices which 
> > > > roam between different IPv6 networks.
> > > >
> > > > If you IPv6 address of a client keeps the same suffix (MAC
> > > > based) and switches network then only the prefix (/64) will change.
> > > >
> > > > This way a network like Google, Facebook, etc could track your 
> > > > device moving from network to network if they only look at the 
> > > > last 64-bits of the IPv6 address.
> > > >
> > > > For servers this is not a problem as you already know in which 
> > > > network they are.
> > > >
> > > > >    *   Network offerings: root-admin can create new network offerings
> > > > (with VPC too) that specifies a network stack option:
> > > > >       *   ipv4 only (default, for backward compatibility all
> > > > networks/offerings post-upgrade migrate to this option)
> > > > >       *   ipv4-and-ipv6
> > > > >       *   ipv6-only (this can be phase 1.b)
> > > > >       *   A new routing option: static (phase1), dynamic (phase2,
> > with
> > > > multiple sub-options such as ospf/bgp etc...)
> > > >
> > > > This means that the network admin will need to statically route 
> > > > the
> > IPv6
> > > > subnet to the VR's outside IPv6 address, for example, on a JunOS
> > router:
> > > >
> > > > set routing-options rib inet6.0 static route 2001:db8:500::/48 
> > > > next-hop
> > > > 2001:db8:100::50
> > > >
> > > > I'm assuming that 2001:db8:100::50 is the address of the VR on 
> > > > the outside (/64) network. In reality this will probably be a 
> > > > longer address, but this is for just the example.
> > > >
> > > > >    *   VR changes:
> > > > >       *   VR gets its guest and public nics set to inet6 auto
> > > > >       *   For each /64 allocated to guest network and VPC tiers,
> > radvd
> > > > is configured to do RA
> > > >
> > > > radvd is fine, but looking at phase 2 with dynamic routing you 
> > > > might already want to look into FRRouting. FRR can also 
> > > > advertise RAs while not doing any routing.
> > > >
> > > > interface ens4
> > > >    no ipv6 nd suppress-ra
> > > >    ipv6 nd prefix 2001:db8:500::/64
> > > >    ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> > > >
> > > > See: http://docs.frrouting.org/en/latest/ipv6.html
> > > >
> > > > >       *   Firewall: a new ipv6 zone/chain is created for ipv6 where
> > ipv6
> > > > firewall rules (ACLs, ingress, egress) are implemented; ACLs 
> > > > between
> > VPC
> > > > tiers are managed/implemented by ipv6 firewall on VR
> > > >
> > > > Please take a look at the existing security_group.py script 
> > > > which implements RFC4890
> > > >
> > > > https://datatracker.ietf.org/doc/html/rfc4890
> > > >
> > > > ICMPv6 is a vital part of IPv6 and certain packets should always 
> > > > be allowed.
> > > >
> > > > >       *   It is assumed that static routes are created on the
> > core/main
> > > > router by the admin or automated using some scripts/tools; for 
> > > > this CloudStack will announce events with details of /64 
> > > > networks and VR's public IPv6 address that can be consumed by a 
> > > > rabbitmq/message bus
> > client
> > > > (for example), or a custom cron job or script as part of orchestration.
> > > > (this wouldn't be necessary for dynamic routing bgp with 
> > > > phase2)\\
> > > >
> > > > You would only need to announce the /48 or /56 allocated to the 
> > > > VR, that's all. You don't need to inform the upstream router 
> > > > about the /64 subnets created within that larger subnet.
> > > >
> > > > >    *   Guest Networking: With SLAAC, it's easy for CloudStack to
> > > > calculate allocate and use a /64 and determine the IPv6 address 
> > > > of VR
> > nics
> > > > and guest VM nics
> > > > >       *   A user create an isolated network/VPC with an offering
> > that is
> > > > ipv6 enabled
> > > > >       *   A user can manage firewall for the IPv6 address/guest nics;
> > > > there'll be no port forward and LB feature though for IPv6
> > > > >       *   A users can run workloads in the guest VMs that listen on
> > > > publically routable ipv6 addresses
> > > > >       *   Usage/billing etc continue to work, no change needed
> > > > >
> > > > > Network layout:
> > > > >
> > > > > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a 
> > > > > VLAN] ->
> > > > [guest VMs/nics]
> > > > > *core/ISP router needs static routes to be added (manually or
> > > > automated), assumes a /48 or /56 configured for the zone
> > > > >
> > > > > Thoughts, feedback?
> > > >
> > > > Looks doable!
> > > >
> > > > Side-note: It would be very cool if you could use parts of this 
> > > > implementation to also route /48, /56, or /60 subnets to 
> > > > individual VMs in Shared networks.
> > > >
> > > > Why? This allows for running Docker containers with native IPv6 
> > > > inside the VM or running a (Open)VPN server inside a VM which 
> > > > then assigns public IPv6 addresses to clients connected.
> > > >
> > > > Instead of routing the subnet to a VR we route the subnet to a 
> > > > single instance in a shared network.
> > > >
> > > > If we could then also move these subnets between Instances 
> > > > easily one can quickly migrate to a different instance while 
> > > > keeping the same IPv6 subnet.
> > > >
> > > > Wido
> > > >
> > > > >
> > > > > Proof-of-concept commentary: here's what I did to test the idea:
> > > > >
> > > > >    *   Created an isolated network and deployed a VM in my home lab
> > > > > The VR running on KVM has following nics
> > > > > eth0 - guest network
> > > > > eth1 - link local
> > > > > eth2 - public network
> > > > >
> > > > >    *   I setup a custom openwrt router on a RPi4 to serve as a
> > toy-core
> > > > router where I create a wan6 IPv6 tunnel using tunnel broker and 
> > > > I got
> > a
> > > > /48 allocated. My configuration looks like:
> > > > > /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> > > > > /64 - 2001:470:36:3e2::/64 (default allocated by)
> > > > >
> > > > > I create a LAN ipv6 (public network for CloudStack VR): at
> > subnet/prefix
> > > > 0:
> > > > > LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
> > > > > SLAAC+stateless DHCP (no dhcpv6)
> > > > >    *
> > > > >    *
> > > > > In the isolated VR, I enabled ipv6 as:
> > > > > net.ipv6.conf.all.disable_ipv6 = 0 
> > > > > net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
> > > > > 1 net.ipv6.conf.all.accept_redirects = 1 
> > > > > net.ipv6.conf.all.autoconf = 1
> > > > >
> > > > > Set up a IPv6 nameserver/dns in /etc/resolve.conf And 
> > > > > configured the nics:
> > > > > echo iface eth0 inet6 auto >> /etc/network/interfaces echo 
> > > > > iface eth2 inet6 auto >> /etc/network/interfaces 
> > > > > /etc/init.d/networking restart Next, restart ACS isolated 
> > > > > network without cleanup to have it
> > > > reconfigure IPv4 nics, firewall, NAT etc
> > > > >
> > > > >    *
> > > > > Next, I created a /64 network for the isolated guest network 
> > > > > on eth0
> > of
> > > > VR using radvd:
> > > > >
> > > > > # cat /etc/radvd.conf
> > > > > interface eth0
> > > > > {
> > > > >      AdvSendAdvert on;
> > > > >      MinRtrAdvInterval 5;
> > > > >      MaxRtrAdvInterval 15;
> > > > >      prefix 2001:470:ed36:1::/64
> > > > >      {
> > > > >          AdvOnLink on;
> > > > >          AdvAutonomous on;
> > > > >      };
> > > > > };
> > > > > systemctl restart radvd
> > > > > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in 
> > > > > this
> > > > ...:1::/64 network
> > > > >    *   Finally I added a static route in toy core-router for the new
> > /64
> > > > IPv6 range in the isolated network
> > > > > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
> > > > > eth2> dev
> > > > <local lan nic>
> > > > >    *
> > > > > ... and I enabled firewall rules to allow any traffic to pass 
> > > > > for the
> > > > new /64 network
> > > > >
> > > > > And voila all done! I create a domain AAAA record that points 
> > > > > to my
> > > > guest VM IPv6 address a test webserver on
> > > > > http://ipv6-isolated-ntwk-demo.yadav.cloud/
> > > > >
> > > > > (Note: I'll get rid of the tunnel and request a new /48 block 
> > > > > after a
> > > > few days, sharing this solely for testing purposes)
> > > > >
> > > > >
> > > > > Regards.
> > > > >
> > > > >
> > > >
> > >
> >
>



Re: IPV6 in Isolated/VPC networks

Posted by Rohit Yadav <ro...@shapeblue.com>.
Thanks Alex, Kristaps. I've updated the design doc to reflect two agreements:

  *   Allocate /64 for both isolated network and VPC tiers, no large allocation of prefixes to VPC (cons: more static routing rules for upstream router/admins)
  *   All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a dedicated /64 prefix/block for systemvms

The only outstanding question now is:

  *   How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage for shared network (design, implementation and use-cases?)
Option1: We don't do it, all user VMs nics have ipv4 address which whose usage we don't track. For public VR/nics/networks, we can simply add the IPv6 details for a related IPv4 address.
Option2: Implement a separate, first-class IPv6 address or /64 prefix tracking/management and usage for all VMs and systemvms nic (this means account/domain level limits and new billing/records)
Option3: other thoughts?


Regards.

________________________________
From: Alex Mattioli <Al...@shapeblue.com>
Sent: Wednesday, September 8, 2021 23:24
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: RE: IPV6 in Isolated/VPC networks

Hi Rohit, Kristaps,

I'd say option 1 as well,  it does create a bit more overhead with static routes but if that's automated for a VPC it can also be easily automated for several tiers of a VPC.  We also don't constrain the amount of tiers in a  VPC.
It has the added advantage to be closer to the desired behaviour with dynamic routing in the future, where a VPC VR can announce several subnets upstream.

Cheers
Alex







 


-----Original Message-----
From: Rohit Yadav <ro...@shapeblue.com>
Sent: 08 September 2021 19:04
To: dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks

Hi Kristaps,

Thanks for sharing, I suppose that means individual tiers should be allocated /64 instead of larger ipv6 blocks to the whole VPC which could cause wastage.

Any objection from anybody?

Regards.
________________________________
From: Kristaps Cudars <kr...@gmail.com>
Sent: Wednesday, September 8, 2021 9:24:01 PM
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks

Hello,

I asked networking team to comment on "How should the IPv6 block/allocation work in VPCs?"
Option1: They haven't seen lately devices with limits on how many static routes can be created.
Option2: With /60 and /62 assignments and big quantity of routers IPv6 assignment from RIPE NNC can be drained fast.

/48 contains 64000 /64
/60 contains 16 /64
64000 / 16 = 4000 routers


On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
> All,
>
> After another iteration with Alex, I've updated the design doc. Kindly review:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> +Isolated+Network+and+VPC+with+Static+Routing
>
>
> ... and advise on some outstanding questions:
>
>   *   How should the IPv6 block/allocation work in VPCs?
> Option1: Should this be simply /64 allocation on any new tier, the
> cons of this option is one static route/rule per VPC tier. (many
> upstream routers may have limit on no. of static routes?)
> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62 (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks global setting (3 is default). The pros of this option are less no. of static route/rule and easy programming, but potential wastage of multiple /64 prefix blocks for unused/uncreated tiers.
>   *   How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage for shared network (design, implementation and use-cases?)
> Option1: We don't do it, all user VMs nics have ipv4 address which whose usage we don't track. For public VR/nics/networks, we can simply add the IPv6 details for a related IPv4 address.
> Option2: Implement a separate, first-class IPv6 address or /64 prefix tracking/management and usage for all VMs and systemvms nic (this means account/domain level limits and new billing/records)
>   *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by default if zone has a IPv6 address block allocated/assigned for use for systemvms (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons of this?)
>   *
>
> Regards.
>
> ________________________________
> From: Rohit Yadav <ro...@shapeblue.com>
> Sent: Thursday, August 19, 2021 15:45
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Hi all,
>
> I've taken feedback from this thread and wrote this design doc:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> +Isolated+Network+and+VPC+with+Static+Routing
>
> Kindly review and advise if I missed anything or anything that needs to be changed/updated. You may comment on the wiki directly too.
>
> Kindly suggest your views on the following (also in the design doc above):
>
> Outstanding Questions:
>
>   *   Should admin or user be able to specify how VPC super CIDRs are created/needed; for example a user can ask for VPC with /60 super CIDR? Or should CloudStack automatically find/allocate a /64 for a new VPC tier from the root-admin configured /64-/48 block?
>   *   Should we explore FRR and iBGP or other strategies to do dynamic routing which may not require advance/complex configuration in the VR or for the users/admin?
>   *   With SLAAC and no dhcpv6, is there a way to support secondary IPv6 addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's nics?
>   *   Any thoughts on UI/UX for firewall/routing management?
>   *   Any other feature/support for isolated network or VPC feature that must be explored or supported such as PF, VPN, LB, vpc static routes, vpc gateway etc.
>   *   For usage - should we have any consideration, or should we assume that IPv4 and IPv6 address will go together for every nic; so IPv6 usage for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new biling/usage needed?
>   *   For smoketests, local dev-test should we explore ULA? Unique Local Address - in the range fc00::/7. Typically only within the 'local' half fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing. This prefix can be randomly generated at first install by CloudStack in a zone using zoneid etc?
>   *   Should we expand support for VR diagnostic feature to include support for ipv6, traceroute6?
>   *   Discuss - expand support/ability to allocate a /60, or /56 etc prefix to an individual VM in shared network (Wido's suggestion)
>
>
> Regards.
>
> ________________________________
> From: Wei ZHOU <us...@gmail.com>
> Sent: Tuesday, August 17, 2021 21:16
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>
> I am fine with not having stateful dhcpv6 and privacy
> extension/temporary address in phase 1. If community decides not to do
> eventually , it is also ok to me.
>
> We could explore how to better use secondary ipv6 addresses as Wido
> advised. It would be great if anyone share some user experience.
>
> -Wei
>
>
> On Tuesday, 17 August 2021, Kristaps Cudars
> <kr...@gmail.com>
> wrote:
>
> > Hi Wei,
> >
> > Published this month's RFC 9099 and explains in different
> > words/perspective what has been written by Alex, Rohit and Wido.
> > https://www.rfc-editor.org/rfc/rfc9099.html
> >
> >
> > On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
> > > Hi Wido,
> > >
> > > (cc to Rohit and Alex)
> > >
> > > It is a good suggestion to use FRR for ipv6. The configuration is
> > > quite simple and the VMs can get SLAAC, routes, etc.
> > >
> > > Privacy extension looks not the same as what you mentioned. see
> > > https://datatracker.ietf.org/doc/html/rfc4941
> > >
> > > You are right. To use static routing, the admins need to configure
> > > the routes in the upstream router, and add some ipv6 ranges (eg
> > > /56 for VPCs and /64 for isolated networks) and their next-hop
> > > (which will be configured in VRs) in CloudStack. CloudStack will
> > > pick up an IPv6 range
> > and
> > > assign it to an isolated network or vpc. @Rohit, correct me if I'm wrong.
> > >
> > > I have a question, it looks stateless dhcpv6 (SLAAC from
> > > router/VR, router/dns etc via RA messages) will be the only option
> > > for now (related
> > to
> > > your pr https://github.com/apache/cloudstack/pull/3077) . Would it
> > > be
> > good
> > > to provide stateful dhcpv6 (which can be implemented by dnsmasq)
> > > as an option in cloudstack ? The advantages are
> > > (1) support other ipv6 cidr sizes than /64.
> > > (2) we can assign a specified Ipv6 address to a vm. vm Ipv6
> > > addresses can be changed
> > > (4) an Ipv6 addresses can be re-used by multiple vms.
> > > The problem is, stateful dhcpv6 does not support
> > > routers,nameservers,
> > etc.
> > > we need to figure it out (probably use radvd/frr and dnsmasq both).
> > >
> > > -Wei
> > >
> > >
> >
>
>
>
>
>
>


> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
> > >
> > > > Hi,
> > > >
> > > > See my inline responses:
> > > >
> > > > Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> > > > > Hi all,
> > > > >
> > > > > Thanks for your feedback and ideas, I've gone ahead with
> > > > > discussing
> > them
> > > > with Alex and came up with a PoC/design which can be implemented
> > > > in the following phases:
> > > > >
> > > > >    *   Phase1: implement ipv6 support in isolated networks and VPC
> > with
> > > > static routing
> > > > >    *   Phase2: discuss and implement support for dynamic routing
> > (TBD)
> > > > >
> > > > > For Phase1 here's the high-level proposal:
> > > > >
> > > > >    *   IPv6 address management:
> > > > >       *   At the zone level root-admin specifies a /64 public range
> > that
> > > > will be used for VRs, then they can add a /48, or /56 IPv6 range
> > > > for
> > guest
> > > > networks (to be used by isolated networks and VPC tiers)
> > > > >       *   On creation of any IPv6 enabled isolated network or VPC
> > tier,
> > > > from the /48 or /56 block a /64 network is allocated/used
> > > > >       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the
> > zone
> > > > (discuss: is privacy a concern, can privacy extensions rfc4941
> > > > of
> > slaac be
> > > > explored?)
> > > >
> > > > Privacy Extensions are only a concern for client devices which
> > > > roam between different IPv6 networks.
> > > >
> > > > If you IPv6 address of a client keeps the same suffix (MAC
> > > > based) and switches network then only the prefix (/64) will change.
> > > >
> > > > This way a network like Google, Facebook, etc could track your
> > > > device moving from network to network if they only look at the
> > > > last 64-bits of the IPv6 address.
> > > >
> > > > For servers this is not a problem as you already know in which
> > > > network they are.
> > > >
> > > > >    *   Network offerings: root-admin can create new network offerings
> > > > (with VPC too) that specifies a network stack option:
> > > > >       *   ipv4 only (default, for backward compatibility all
> > > > networks/offerings post-upgrade migrate to this option)
> > > > >       *   ipv4-and-ipv6
> > > > >       *   ipv6-only (this can be phase 1.b)
> > > > >       *   A new routing option: static (phase1), dynamic (phase2,
> > with
> > > > multiple sub-options such as ospf/bgp etc...)
> > > >
> > > > This means that the network admin will need to statically route
> > > > the
> > IPv6
> > > > subnet to the VR's outside IPv6 address, for example, on a JunOS
> > router:
> > > >
> > > > set routing-options rib inet6.0 static route 2001:db8:500::/48
> > > > next-hop
> > > > 2001:db8:100::50
> > > >
> > > > I'm assuming that 2001:db8:100::50 is the address of the VR on
> > > > the outside (/64) network. In reality this will probably be a
> > > > longer address, but this is for just the example.
> > > >
> > > > >    *   VR changes:
> > > > >       *   VR gets its guest and public nics set to inet6 auto
> > > > >       *   For each /64 allocated to guest network and VPC tiers,
> > radvd
> > > > is configured to do RA
> > > >
> > > > radvd is fine, but looking at phase 2 with dynamic routing you
> > > > might already want to look into FRRouting. FRR can also
> > > > advertise RAs while not doing any routing.
> > > >
> > > > interface ens4
> > > >    no ipv6 nd suppress-ra
> > > >    ipv6 nd prefix 2001:db8:500::/64
> > > >    ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> > > >
> > > > See: http://docs.frrouting.org/en/latest/ipv6.html
> > > >
> > > > >       *   Firewall: a new ipv6 zone/chain is created for ipv6 where
> > ipv6
> > > > firewall rules (ACLs, ingress, egress) are implemented; ACLs
> > > > between
> > VPC
> > > > tiers are managed/implemented by ipv6 firewall on VR
> > > >
> > > > Please take a look at the existing security_group.py script
> > > > which implements RFC4890
> > > >
> > > > https://datatracker.ietf.org/doc/html/rfc4890
> > > >
> > > > ICMPv6 is a vital part of IPv6 and certain packets should always
> > > > be allowed.
> > > >
> > > > >       *   It is assumed that static routes are created on the
> > core/main
> > > > router by the admin or automated using some scripts/tools; for
> > > > this CloudStack will announce events with details of /64
> > > > networks and VR's public IPv6 address that can be consumed by a
> > > > rabbitmq/message bus
> > client
> > > > (for example), or a custom cron job or script as part of orchestration.
> > > > (this wouldn't be necessary for dynamic routing bgp with
> > > > phase2)\\
> > > >
> > > > You would only need to announce the /48 or /56 allocated to the
> > > > VR, that's all. You don't need to inform the upstream router
> > > > about the /64 subnets created within that larger subnet.
> > > >
> > > > >    *   Guest Networking: With SLAAC, it's easy for CloudStack to
> > > > calculate allocate and use a /64 and determine the IPv6 address
> > > > of VR
> > nics
> > > > and guest VM nics
> > > > >       *   A user create an isolated network/VPC with an offering
> > that is
> > > > ipv6 enabled
> > > > >       *   A user can manage firewall for the IPv6 address/guest nics;
> > > > there'll be no port forward and LB feature though for IPv6
> > > > >       *   A users can run workloads in the guest VMs that listen on
> > > > publically routable ipv6 addresses
> > > > >       *   Usage/billing etc continue to work, no change needed
> > > > >
> > > > > Network layout:
> > > > >
> > > > > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a
> > > > > VLAN] ->
> > > > [guest VMs/nics]
> > > > > *core/ISP router needs static routes to be added (manually or
> > > > automated), assumes a /48 or /56 configured for the zone
> > > > >
> > > > > Thoughts, feedback?
> > > >
> > > > Looks doable!
> > > >
> > > > Side-note: It would be very cool if you could use parts of this
> > > > implementation to also route /48, /56, or /60 subnets to
> > > > individual VMs in Shared networks.
> > > >
> > > > Why? This allows for running Docker containers with native IPv6
> > > > inside the VM or running a (Open)VPN server inside a VM which
> > > > then assigns public IPv6 addresses to clients connected.
> > > >
> > > > Instead of routing the subnet to a VR we route the subnet to a
> > > > single instance in a shared network.
> > > >
> > > > If we could then also move these subnets between Instances
> > > > easily one can quickly migrate to a different instance while
> > > > keeping the same IPv6 subnet.
> > > >
> > > > Wido
> > > >
> > > > >
> > > > > Proof-of-concept commentary: here's what I did to test the idea:
> > > > >
> > > > >    *   Created an isolated network and deployed a VM in my home lab
> > > > > The VR running on KVM has following nics
> > > > > eth0 - guest network
> > > > > eth1 - link local
> > > > > eth2 - public network
> > > > >
> > > > >    *   I setup a custom openwrt router on a RPi4 to serve as a
> > toy-core
> > > > router where I create a wan6 IPv6 tunnel using tunnel broker and
> > > > I got
> > a
> > > > /48 allocated. My configuration looks like:
> > > > > /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> > > > > /64 - 2001:470:36:3e2::/64 (default allocated by)
> > > > >
> > > > > I create a LAN ipv6 (public network for CloudStack VR): at
> > subnet/prefix
> > > > 0:
> > > > > LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
> > > > > SLAAC+stateless DHCP (no dhcpv6)
> > > > >    *
> > > > >    *
> > > > > In the isolated VR, I enabled ipv6 as:
> > > > > net.ipv6.conf.all.disable_ipv6 = 0
> > > > > net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra =
> > > > > 1 net.ipv6.conf.all.accept_redirects = 1
> > > > > net.ipv6.conf.all.autoconf = 1
> > > > >
> > > > > Set up a IPv6 nameserver/dns in /etc/resolve.conf And
> > > > > configured the nics:
> > > > > echo iface eth0 inet6 auto >> /etc/network/interfaces echo
> > > > > iface eth2 inet6 auto >> /etc/network/interfaces
> > > > > /etc/init.d/networking restart Next, restart ACS isolated
> > > > > network without cleanup to have it
> > > > reconfigure IPv4 nics, firewall, NAT etc
> > > > >
> > > > >    *
> > > > > Next, I created a /64 network for the isolated guest network
> > > > > on eth0
> > of
> > > > VR using radvd:
> > > > >
> > > > > # cat /etc/radvd.conf
> > > > > interface eth0
> > > > > {
> > > > >      AdvSendAdvert on;
> > > > >      MinRtrAdvInterval 5;
> > > > >      MaxRtrAdvInterval 15;
> > > > >      prefix 2001:470:ed36:1::/64
> > > > >      {
> > > > >          AdvOnLink on;
> > > > >          AdvAutonomous on;
> > > > >      };
> > > > > };
> > > > > systemctl restart radvd
> > > > > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in
> > > > > this
> > > > ...:1::/64 network
> > > > >    *   Finally I added a static route in toy core-router for the new
> > /64
> > > > IPv6 range in the isolated network
> > > > > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on
> > > > > eth2> dev
> > > > <local lan nic>
> > > > >    *
> > > > > ... and I enabled firewall rules to allow any traffic to pass
> > > > > for the
> > > > new /64 network
> > > > >
> > > > > And voila all done! I create a domain AAAA record that points
> > > > > to my
> > > > guest VM IPv6 address a test webserver on
> > > > > http://ipv6-isolated-ntwk-demo.yadav.cloud/
> > > > >
> > > > > (Note: I'll get rid of the tunnel and request a new /48 block
> > > > > after a
> > > > few days, sharing this solely for testing purposes)
> > > > >
> > > > >
> > > > > Regards.
> > > > >
> > > > >
> > > >
> > >
> >
>


RE: IPV6 in Isolated/VPC networks

Posted by Alex Mattioli <Al...@shapeblue.com>.
Hi Rohit, Kristaps,

I'd say option 1 as well,  it does create a bit more overhead with static routes but if that's automated for a VPC it can also be easily automated for several tiers of a VPC.  We also don't constrain the amount of tiers in a  VPC.
It has the added advantage to be closer to the desired behaviour with dynamic routing in the future, where a VPC VR can announce several subnets upstream.

Cheers
Alex




 


-----Original Message-----
From: Rohit Yadav <ro...@shapeblue.com> 
Sent: 08 September 2021 19:04
To: dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks

Hi Kristaps,

Thanks for sharing, I suppose that means individual tiers should be allocated /64 instead of larger ipv6 blocks to the whole VPC which could cause wastage.

Any objection from anybody?

Regards.
________________________________
From: Kristaps Cudars <kr...@gmail.com>
Sent: Wednesday, September 8, 2021 9:24:01 PM
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks

Hello,

I asked networking team to comment on "How should the IPv6 block/allocation work in VPCs?"
Option1: They haven't seen lately devices with limits on how many static routes can be created.
Option2: With /60 and /62 assignments and big quantity of routers IPv6 assignment from RIPE NNC can be drained fast.

/48 contains 64000 /64
/60 contains 16 /64
64000 / 16 = 4000 routers


On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
> All,
>
> After another iteration with Alex, I've updated the design doc. Kindly review:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> +Isolated+Network+and+VPC+with+Static+Routing
>
>
> ... and advise on some outstanding questions:
>
>   *   How should the IPv6 block/allocation work in VPCs?
> Option1: Should this be simply /64 allocation on any new tier, the 
> cons of this option is one static route/rule per VPC tier. (many 
> upstream routers may have limit on no. of static routes?)
> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62 (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks global setting (3 is default). The pros of this option are less no. of static route/rule and easy programming, but potential wastage of multiple /64 prefix blocks for unused/uncreated tiers.
>   *   How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage for shared network (design, implementation and use-cases?)
> Option1: We don't do it, all user VMs nics have ipv4 address which whose usage we don't track. For public VR/nics/networks, we can simply add the IPv6 details for a related IPv4 address.
> Option2: Implement a separate, first-class IPv6 address or /64 prefix tracking/management and usage for all VMs and systemvms nic (this means account/domain level limits and new billing/records)
>   *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by default if zone has a IPv6 address block allocated/assigned for use for systemvms (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons of this?)
>   *
>
> Regards.
>
> ________________________________
> From: Rohit Yadav <ro...@shapeblue.com>
> Sent: Thursday, August 19, 2021 15:45
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Hi all,
>
> I've taken feedback from this thread and wrote this design doc:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in
> +Isolated+Network+and+VPC+with+Static+Routing
>
> Kindly review and advise if I missed anything or anything that needs to be changed/updated. You may comment on the wiki directly too.
>
> Kindly suggest your views on the following (also in the design doc above):
>
> Outstanding Questions:
>
>   *   Should admin or user be able to specify how VPC super CIDRs are created/needed; for example a user can ask for VPC with /60 super CIDR? Or should CloudStack automatically find/allocate a /64 for a new VPC tier from the root-admin configured /64-/48 block?
>   *   Should we explore FRR and iBGP or other strategies to do dynamic routing which may not require advance/complex configuration in the VR or for the users/admin?
>   *   With SLAAC and no dhcpv6, is there a way to support secondary IPv6 addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's nics?
>   *   Any thoughts on UI/UX for firewall/routing management?
>   *   Any other feature/support for isolated network or VPC feature that must be explored or supported such as PF, VPN, LB, vpc static routes, vpc gateway etc.
>   *   For usage - should we have any consideration, or should we assume that IPv4 and IPv6 address will go together for every nic; so IPv6 usage for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new biling/usage needed?
>   *   For smoketests, local dev-test should we explore ULA? Unique Local Address - in the range fc00::/7. Typically only within the 'local' half fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing. This prefix can be randomly generated at first install by CloudStack in a zone using zoneid etc?
>   *   Should we expand support for VR diagnostic feature to include support for ipv6, traceroute6?
>   *   Discuss - expand support/ability to allocate a /60, or /56 etc prefix to an individual VM in shared network (Wido's suggestion)
>
>
> Regards.
>
> ________________________________
> From: Wei ZHOU <us...@gmail.com>
> Sent: Tuesday, August 17, 2021 21:16
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>
> I am fine with not having stateful dhcpv6 and privacy 
> extension/temporary address in phase 1. If community decides not to do 
> eventually , it is also ok to me.
>
> We could explore how to better use secondary ipv6 addresses as Wido 
> advised. It would be great if anyone share some user experience.
>
> -Wei
>
>
> On Tuesday, 17 August 2021, Kristaps Cudars 
> <kr...@gmail.com>
> wrote:
>
> > Hi Wei,
> >
> > Published this month's RFC 9099 and explains in different 
> > words/perspective what has been written by Alex, Rohit and Wido.
> > https://www.rfc-editor.org/rfc/rfc9099.html
> >
> >
> > On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
> > > Hi Wido,
> > >
> > > (cc to Rohit and Alex)
> > >
> > > It is a good suggestion to use FRR for ipv6. The configuration is 
> > > quite simple and the VMs can get SLAAC, routes, etc.
> > >
> > > Privacy extension looks not the same as what you mentioned. see
> > > https://datatracker.ietf.org/doc/html/rfc4941
> > >
> > > You are right. To use static routing, the admins need to configure 
> > > the routes in the upstream router, and add some ipv6 ranges (eg 
> > > /56 for VPCs and /64 for isolated networks) and their next-hop  
> > > (which will be configured in VRs) in CloudStack. CloudStack will 
> > > pick up an IPv6 range
> > and
> > > assign it to an isolated network or vpc. @Rohit, correct me if I'm wrong.
> > >
> > > I have a question, it looks stateless dhcpv6 (SLAAC from 
> > > router/VR, router/dns etc via RA messages) will be the only option 
> > > for now (related
> > to
> > > your pr https://github.com/apache/cloudstack/pull/3077) . Would it 
> > > be
> > good
> > > to provide stateful dhcpv6 (which can be implemented by dnsmasq) 
> > > as an option in cloudstack ? The advantages are
> > > (1) support other ipv6 cidr sizes than /64.
> > > (2) we can assign a specified Ipv6 address to a vm. vm Ipv6 
> > > addresses can be changed
> > > (4) an Ipv6 addresses can be re-used by multiple vms.
> > > The problem is, stateful dhcpv6 does not support 
> > > routers,nameservers,
> > etc.
> > > we need to figure it out (probably use radvd/frr and dnsmasq both).
> > >
> > > -Wei
> > >
> > >
> >
>
>
>
>
>
> 
 

> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
> > >
> > > > Hi,
> > > >
> > > > See my inline responses:
> > > >
> > > > Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> > > > > Hi all,
> > > > >
> > > > > Thanks for your feedback and ideas, I've gone ahead with 
> > > > > discussing
> > them
> > > > with Alex and came up with a PoC/design which can be implemented 
> > > > in the following phases:
> > > > >
> > > > >    *   Phase1: implement ipv6 support in isolated networks and VPC
> > with
> > > > static routing
> > > > >    *   Phase2: discuss and implement support for dynamic routing
> > (TBD)
> > > > >
> > > > > For Phase1 here's the high-level proposal:
> > > > >
> > > > >    *   IPv6 address management:
> > > > >       *   At the zone level root-admin specifies a /64 public range
> > that
> > > > will be used for VRs, then they can add a /48, or /56 IPv6 range 
> > > > for
> > guest
> > > > networks (to be used by isolated networks and VPC tiers)
> > > > >       *   On creation of any IPv6 enabled isolated network or VPC
> > tier,
> > > > from the /48 or /56 block a /64 network is allocated/used
> > > > >       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the
> > zone
> > > > (discuss: is privacy a concern, can privacy extensions rfc4941 
> > > > of
> > slaac be
> > > > explored?)
> > > >
> > > > Privacy Extensions are only a concern for client devices which 
> > > > roam between different IPv6 networks.
> > > >
> > > > If you IPv6 address of a client keeps the same suffix (MAC 
> > > > based) and switches network then only the prefix (/64) will change.
> > > >
> > > > This way a network like Google, Facebook, etc could track your 
> > > > device moving from network to network if they only look at the 
> > > > last 64-bits of the IPv6 address.
> > > >
> > > > For servers this is not a problem as you already know in which 
> > > > network they are.
> > > >
> > > > >    *   Network offerings: root-admin can create new network offerings
> > > > (with VPC too) that specifies a network stack option:
> > > > >       *   ipv4 only (default, for backward compatibility all
> > > > networks/offerings post-upgrade migrate to this option)
> > > > >       *   ipv4-and-ipv6
> > > > >       *   ipv6-only (this can be phase 1.b)
> > > > >       *   A new routing option: static (phase1), dynamic (phase2,
> > with
> > > > multiple sub-options such as ospf/bgp etc...)
> > > >
> > > > This means that the network admin will need to statically route 
> > > > the
> > IPv6
> > > > subnet to the VR's outside IPv6 address, for example, on a JunOS
> > router:
> > > >
> > > > set routing-options rib inet6.0 static route 2001:db8:500::/48 
> > > > next-hop
> > > > 2001:db8:100::50
> > > >
> > > > I'm assuming that 2001:db8:100::50 is the address of the VR on 
> > > > the outside (/64) network. In reality this will probably be a 
> > > > longer address, but this is for just the example.
> > > >
> > > > >    *   VR changes:
> > > > >       *   VR gets its guest and public nics set to inet6 auto
> > > > >       *   For each /64 allocated to guest network and VPC tiers,
> > radvd
> > > > is configured to do RA
> > > >
> > > > radvd is fine, but looking at phase 2 with dynamic routing you 
> > > > might already want to look into FRRouting. FRR can also 
> > > > advertise RAs while not doing any routing.
> > > >
> > > > interface ens4
> > > >    no ipv6 nd suppress-ra
> > > >    ipv6 nd prefix 2001:db8:500::/64
> > > >    ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> > > >
> > > > See: http://docs.frrouting.org/en/latest/ipv6.html
> > > >
> > > > >       *   Firewall: a new ipv6 zone/chain is created for ipv6 where
> > ipv6
> > > > firewall rules (ACLs, ingress, egress) are implemented; ACLs 
> > > > between
> > VPC
> > > > tiers are managed/implemented by ipv6 firewall on VR
> > > >
> > > > Please take a look at the existing security_group.py script 
> > > > which implements RFC4890
> > > >
> > > > https://datatracker.ietf.org/doc/html/rfc4890
> > > >
> > > > ICMPv6 is a vital part of IPv6 and certain packets should always 
> > > > be allowed.
> > > >
> > > > >       *   It is assumed that static routes are created on the
> > core/main
> > > > router by the admin or automated using some scripts/tools; for 
> > > > this CloudStack will announce events with details of /64 
> > > > networks and VR's public IPv6 address that can be consumed by a 
> > > > rabbitmq/message bus
> > client
> > > > (for example), or a custom cron job or script as part of orchestration.
> > > > (this wouldn't be necessary for dynamic routing bgp with 
> > > > phase2)\\
> > > >
> > > > You would only need to announce the /48 or /56 allocated to the 
> > > > VR, that's all. You don't need to inform the upstream router 
> > > > about the /64 subnets created within that larger subnet.
> > > >
> > > > >    *   Guest Networking: With SLAAC, it's easy for CloudStack to
> > > > calculate allocate and use a /64 and determine the IPv6 address 
> > > > of VR
> > nics
> > > > and guest VM nics
> > > > >       *   A user create an isolated network/VPC with an offering
> > that is
> > > > ipv6 enabled
> > > > >       *   A user can manage firewall for the IPv6 address/guest nics;
> > > > there'll be no port forward and LB feature though for IPv6
> > > > >       *   A users can run workloads in the guest VMs that listen on
> > > > publically routable ipv6 addresses
> > > > >       *   Usage/billing etc continue to work, no change needed
> > > > >
> > > > > Network layout:
> > > > >
> > > > > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a 
> > > > > VLAN] ->
> > > > [guest VMs/nics]
> > > > > *core/ISP router needs static routes to be added (manually or
> > > > automated), assumes a /48 or /56 configured for the zone
> > > > >
> > > > > Thoughts, feedback?
> > > >
> > > > Looks doable!
> > > >
> > > > Side-note: It would be very cool if you could use parts of this 
> > > > implementation to also route /48, /56, or /60 subnets to 
> > > > individual VMs in Shared networks.
> > > >
> > > > Why? This allows for running Docker containers with native IPv6 
> > > > inside the VM or running a (Open)VPN server inside a VM which 
> > > > then assigns public IPv6 addresses to clients connected.
> > > >
> > > > Instead of routing the subnet to a VR we route the subnet to a 
> > > > single instance in a shared network.
> > > >
> > > > If we could then also move these subnets between Instances 
> > > > easily one can quickly migrate to a different instance while 
> > > > keeping the same IPv6 subnet.
> > > >
> > > > Wido
> > > >
> > > > >
> > > > > Proof-of-concept commentary: here's what I did to test the idea:
> > > > >
> > > > >    *   Created an isolated network and deployed a VM in my home lab
> > > > > The VR running on KVM has following nics
> > > > > eth0 - guest network
> > > > > eth1 - link local
> > > > > eth2 - public network
> > > > >
> > > > >    *   I setup a custom openwrt router on a RPi4 to serve as a
> > toy-core
> > > > router where I create a wan6 IPv6 tunnel using tunnel broker and 
> > > > I got
> > a
> > > > /48 allocated. My configuration looks like:
> > > > > /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> > > > > /64 - 2001:470:36:3e2::/64 (default allocated by)
> > > > >
> > > > > I create a LAN ipv6 (public network for CloudStack VR): at
> > subnet/prefix
> > > > 0:
> > > > > LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode: 
> > > > > SLAAC+stateless DHCP (no dhcpv6)
> > > > >    *
> > > > >    *
> > > > > In the isolated VR, I enabled ipv6 as:
> > > > > net.ipv6.conf.all.disable_ipv6 = 0 
> > > > > net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra = 
> > > > > 1 net.ipv6.conf.all.accept_redirects = 1 
> > > > > net.ipv6.conf.all.autoconf = 1
> > > > >
> > > > > Set up a IPv6 nameserver/dns in /etc/resolve.conf And 
> > > > > configured the nics:
> > > > > echo iface eth0 inet6 auto >> /etc/network/interfaces echo 
> > > > > iface eth2 inet6 auto >> /etc/network/interfaces 
> > > > > /etc/init.d/networking restart Next, restart ACS isolated 
> > > > > network without cleanup to have it
> > > > reconfigure IPv4 nics, firewall, NAT etc
> > > > >
> > > > >    *
> > > > > Next, I created a /64 network for the isolated guest network 
> > > > > on eth0
> > of
> > > > VR using radvd:
> > > > >
> > > > > # cat /etc/radvd.conf
> > > > > interface eth0
> > > > > {
> > > > >      AdvSendAdvert on;
> > > > >      MinRtrAdvInterval 5;
> > > > >      MaxRtrAdvInterval 15;
> > > > >      prefix 2001:470:ed36:1::/64
> > > > >      {
> > > > >          AdvOnLink on;
> > > > >          AdvAutonomous on;
> > > > >      };
> > > > > };
> > > > > systemctl restart radvd
> > > > > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in 
> > > > > this
> > > > ...:1::/64 network
> > > > >    *   Finally I added a static route in toy core-router for the new
> > /64
> > > > IPv6 range in the isolated network
> > > > > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on 
> > > > > eth2> dev
> > > > <local lan nic>
> > > > >    *
> > > > > ... and I enabled firewall rules to allow any traffic to pass 
> > > > > for the
> > > > new /64 network
> > > > >
> > > > > And voila all done! I create a domain AAAA record that points 
> > > > > to my
> > > > guest VM IPv6 address a test webserver on
> > > > > http://ipv6-isolated-ntwk-demo.yadav.cloud/
> > > > >
> > > > > (Note: I'll get rid of the tunnel and request a new /48 block 
> > > > > after a
> > > > few days, sharing this solely for testing purposes)
> > > > >
> > > > >
> > > > > Regards.
> > > > >
> > > > >
> > > >
> > >
> >
>


Re: IPV6 in Isolated/VPC networks

Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi Kristaps,

Thanks for sharing, I suppose that means individual tiers should be allocated /64 instead of larger ipv6 blocks to the whole VPC which could cause wastage.

Any objection from anybody?

Regards.
________________________________
From: Kristaps Cudars <kr...@gmail.com>
Sent: Wednesday, September 8, 2021 9:24:01 PM
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks

Hello,

I asked networking team to comment on “How should the IPv6 block/allocation work in VPCs?”
Option1: They haven’t seen lately devices with limits on how many static routes can be created.
Option2: With /60 and /62 assignments and big quantity of routers IPv6 assignment from RIPE NNC can be drained fast.

/48 contains 64000 /64
/60 contains 16 /64
64000 / 16 = 4000 routers


On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote:
> All,
>
> After another iteration with Alex, I've updated the design doc. Kindly review:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
>
>
> ... and advise on some outstanding questions:
>
>   *   How should the IPv6 block/allocation work in VPCs?
> Option1: Should this be simply /64 allocation on any new tier, the cons of this option is one static route/rule per VPC tier. (many upstream routers may have limit on no. of static routes?)
> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62 (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks global setting (3 is default). The pros of this option are less no. of static route/rule and easy programming, but potential wastage of multiple /64 prefix blocks for unused/uncreated tiers.
>   *   How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage for shared network (design, implementation and use-cases?)
> Option1: We don't do it, all user VMs nics have ipv4 address which whose usage we don't track. For public VR/nics/networks, we can simply add the IPv6 details for a related IPv4 address.
> Option2: Implement a separate, first-class IPv6 address or /64 prefix tracking/management and usage for all VMs and systemvms nic (this means account/domain level limits and new billing/records)
>   *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by default if zone has a IPv6 address block allocated/assigned for use for systemvms (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons of this?)
>   *
>
> Regards.
>
> ________________________________
> From: Rohit Yadav <ro...@shapeblue.com>
> Sent: Thursday, August 19, 2021 15:45
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Hi all,
>
> I've taken feedback from this thread and wrote this design doc:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
>
> Kindly review and advise if I missed anything or anything that needs to be changed/updated. You may comment on the wiki directly too.
>
> Kindly suggest your views on the following (also in the design doc above):
>
> Outstanding Questions:
>
>   *   Should admin or user be able to specify how VPC super CIDRs are created/needed; for example a user can ask for VPC with /60 super CIDR? Or should CloudStack automatically find/allocate a /64 for a new VPC tier from the root-admin configured /64-/48 block?
>   *   Should we explore FRR and iBGP or other strategies to do dynamic routing which may not require advance/complex configuration in the VR or for the users/admin?
>   *   With SLAAC and no dhcpv6, is there a way to support secondary IPv6 addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's nics?
>   *   Any thoughts on UI/UX for firewall/routing management?
>   *   Any other feature/support for isolated network or VPC feature that must be explored or supported such as PF, VPN, LB, vpc static routes, vpc gateway etc.
>   *   For usage - should we have any consideration, or should we assume that IPv4 and IPv6 address will go together for every nic; so IPv6 usage for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new biling/usage needed?
>   *   For smoketests, local dev-test should we explore ULA? Unique Local Address - in the range fc00::/7. Typically only within the ‘local’ half fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing. This prefix can be randomly generated at first install by CloudStack in a zone using zoneid etc?
>   *   Should we expand support for VR diagnostic feature to include support for ipv6, traceroute6?
>   *   Discuss - expand support/ability to allocate a /60, or /56 etc prefix to an individual VM in shared network (Wido's suggestion)
>
>
> Regards.
>
> ________________________________
> From: Wei ZHOU <us...@gmail.com>
> Sent: Tuesday, August 17, 2021 21:16
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
> Thanks Kristaps, Wido, Rohit and Alex for your replies.
>
> I am fine with not having stateful dhcpv6 and privacy extension/temporary
> address in phase 1. If community decides not to do eventually , it is also
> ok to me.
>
> We could explore how to better use secondary ipv6 addresses as Wido
> advised. It would be great if anyone share some user experience.
>
> -Wei
>
>
> On Tuesday, 17 August 2021, Kristaps Cudars <kr...@gmail.com>
> wrote:
>
> > Hi Wei,
> >
> > Published this month’s RFC 9099 and explains in different
> > words/perspective what has been written by Alex, Rohit and Wido.
> > https://www.rfc-editor.org/rfc/rfc9099.html
> >
> >
> > On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
> > > Hi Wido,
> > >
> > > (cc to Rohit and Alex)
> > >
> > > It is a good suggestion to use FRR for ipv6. The configuration is quite
> > > simple and the VMs can get SLAAC, routes, etc.
> > >
> > > Privacy extension looks not the same as what you mentioned. see
> > > https://datatracker.ietf.org/doc/html/rfc4941
> > >
> > > You are right. To use static routing, the admins need to configure the
> > > routes in the upstream router, and add some ipv6 ranges (eg /56 for VPCs
> > > and /64 for isolated networks) and their next-hop  (which will be
> > > configured in VRs) in CloudStack. CloudStack will pick up an IPv6 range
> > and
> > > assign it to an isolated network or vpc. @Rohit, correct me if I'm wrong.
> > >
> > > I have a question, it looks stateless dhcpv6 (SLAAC from router/VR,
> > > router/dns etc via RA messages) will be the only option for now (related
> > to
> > > your pr https://github.com/apache/cloudstack/pull/3077) . Would it be
> > good
> > > to provide stateful dhcpv6 (which can be implemented by dnsmasq) as an
> > > option in cloudstack ? The advantages are
> > > (1) support other ipv6 cidr sizes than /64.
> > > (2) we can assign a specified Ipv6 address to a vm. vm Ipv6 addresses can
> > > be changed
> > > (4) an Ipv6 addresses can be re-used by multiple vms.
> > > The problem is, stateful dhcpv6 does not support routers,nameservers,
> > etc.
> > > we need to figure it out (probably use radvd/frr and dnsmasq both).
> > >
> > > -Wei
> > >
> > >
> >
>
>
>
>
>
> 
 

> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
> > >
> > > > Hi,
> > > >
> > > > See my inline responses:
> > > >
> > > > Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> > > > > Hi all,
> > > > >
> > > > > Thanks for your feedback and ideas, I've gone ahead with discussing
> > them
> > > > with Alex and came up with a PoC/design which can be implemented in the
> > > > following phases:
> > > > >
> > > > >    *   Phase1: implement ipv6 support in isolated networks and VPC
> > with
> > > > static routing
> > > > >    *   Phase2: discuss and implement support for dynamic routing
> > (TBD)
> > > > >
> > > > > For Phase1 here's the high-level proposal:
> > > > >
> > > > >    *   IPv6 address management:
> > > > >       *   At the zone level root-admin specifies a /64 public range
> > that
> > > > will be used for VRs, then they can add a /48, or /56 IPv6 range for
> > guest
> > > > networks (to be used by isolated networks and VPC tiers)
> > > > >       *   On creation of any IPv6 enabled isolated network or VPC
> > tier,
> > > > from the /48 or /56 block a /64 network is allocated/used
> > > > >       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the
> > zone
> > > > (discuss: is privacy a concern, can privacy extensions rfc4941 of
> > slaac be
> > > > explored?)
> > > >
> > > > Privacy Extensions are only a concern for client devices which roam
> > > > between different IPv6 networks.
> > > >
> > > > If you IPv6 address of a client keeps the same suffix (MAC based) and
> > > > switches network then only the prefix (/64) will change.
> > > >
> > > > This way a network like Google, Facebook, etc could track your device
> > > > moving from network to network if they only look at the last 64-bits of
> > > > the IPv6 address.
> > > >
> > > > For servers this is not a problem as you already know in which network
> > > > they are.
> > > >
> > > > >    *   Network offerings: root-admin can create new network offerings
> > > > (with VPC too) that specifies a network stack option:
> > > > >       *   ipv4 only (default, for backward compatibility all
> > > > networks/offerings post-upgrade migrate to this option)
> > > > >       *   ipv4-and-ipv6
> > > > >       *   ipv6-only (this can be phase 1.b)
> > > > >       *   A new routing option: static (phase1), dynamic (phase2,
> > with
> > > > multiple sub-options such as ospf/bgp etc...)
> > > >
> > > > This means that the network admin will need to statically route the
> > IPv6
> > > > subnet to the VR's outside IPv6 address, for example, on a JunOS
> > router:
> > > >
> > > > set routing-options rib inet6.0 static route 2001:db8:500::/48 next-hop
> > > > 2001:db8:100::50
> > > >
> > > > I'm assuming that 2001:db8:100::50 is the address of the VR on the
> > > > outside (/64) network. In reality this will probably be a longer
> > > > address, but this is for just the example.
> > > >
> > > > >    *   VR changes:
> > > > >       *   VR gets its guest and public nics set to inet6 auto
> > > > >       *   For each /64 allocated to guest network and VPC tiers,
> > radvd
> > > > is configured to do RA
> > > >
> > > > radvd is fine, but looking at phase 2 with dynamic routing you might
> > > > already want to look into FRRouting. FRR can also advertise RAs while
> > > > not doing any routing.
> > > >
> > > > interface ens4
> > > >    no ipv6 nd suppress-ra
> > > >    ipv6 nd prefix 2001:db8:500::/64
> > > >    ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> > > >
> > > > See: http://docs.frrouting.org/en/latest/ipv6.html
> > > >
> > > > >       *   Firewall: a new ipv6 zone/chain is created for ipv6 where
> > ipv6
> > > > firewall rules (ACLs, ingress, egress) are implemented; ACLs between
> > VPC
> > > > tiers are managed/implemented by ipv6 firewall on VR
> > > >
> > > > Please take a look at the existing security_group.py script which
> > > > implements RFC4890
> > > >
> > > > https://datatracker.ietf.org/doc/html/rfc4890
> > > >
> > > > ICMPv6 is a vital part of IPv6 and certain packets should always be
> > > > allowed.
> > > >
> > > > >       *   It is assumed that static routes are created on the
> > core/main
> > > > router by the admin or automated using some scripts/tools; for this
> > > > CloudStack will announce events with details of /64 networks and VR's
> > > > public IPv6 address that can be consumed by a rabbitmq/message bus
> > client
> > > > (for example), or a custom cron job or script as part of orchestration.
> > > > (this wouldn't be necessary for dynamic routing bgp with phase2)\\
> > > >
> > > > You would only need to announce the /48 or /56 allocated to the VR,
> > > > that's all. You don't need to inform the upstream router about the /64
> > > > subnets created within that larger subnet.
> > > >
> > > > >    *   Guest Networking: With SLAAC, it's easy for CloudStack to
> > > > calculate allocate and use a /64 and determine the IPv6 address of VR
> > nics
> > > > and guest VM nics
> > > > >       *   A user create an isolated network/VPC with an offering
> > that is
> > > > ipv6 enabled
> > > > >       *   A user can manage firewall for the IPv6 address/guest nics;
> > > > there'll be no port forward and LB feature though for IPv6
> > > > >       *   A users can run workloads in the guest VMs that listen on
> > > > publically routable ipv6 addresses
> > > > >       *   Usage/billing etc continue to work, no change needed
> > > > >
> > > > > Network layout:
> > > > >
> > > > > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] ->
> > > > [guest VMs/nics]
> > > > > *core/ISP router needs static routes to be added (manually or
> > > > automated), assumes a /48 or /56 configured for the zone
> > > > >
> > > > > Thoughts, feedback?
> > > >
> > > > Looks doable!
> > > >
> > > > Side-note: It would be very cool if you could use parts of this
> > > > implementation to also route /48, /56, or /60 subnets to individual VMs
> > > > in Shared networks.
> > > >
> > > > Why? This allows for running Docker containers with native IPv6 inside
> > > > the VM or running a (Open)VPN server inside a VM which then assigns
> > > > public IPv6 addresses to clients connected.
> > > >
> > > > Instead of routing the subnet to a VR we route the subnet to a single
> > > > instance in a shared network.
> > > >
> > > > If we could then also move these subnets between Instances easily one
> > > > can quickly migrate to a different instance while keeping the same IPv6
> > > > subnet.
> > > >
> > > > Wido
> > > >
> > > > >
> > > > > Proof-of-concept commentary: here's what I did to test the idea:
> > > > >
> > > > >    *   Created an isolated network and deployed a VM in my home lab
> > > > > The VR running on KVM has following nics
> > > > > eth0 - guest network
> > > > > eth1 - link local
> > > > > eth2 - public network
> > > > >
> > > > >    *   I setup a custom openwrt router on a RPi4 to serve as a
> > toy-core
> > > > router where I create a wan6 IPv6 tunnel using tunnel broker and I got
> > a
> > > > /48 allocated. My configuration looks like:
> > > > > /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> > > > > /64 - 2001:470:36:3e2::/64 (default allocated by)
> > > > >
> > > > > I create a LAN ipv6 (public network for CloudStack VR): at
> > subnet/prefix
> > > > 0:
> > > > > LAN IPv6 address: 2001:470:ed36:0::1/64
> > > > > Address mode: SLAAC+stateless DHCP (no dhcpv6)
> > > > >    *
> > > > >    *
> > > > > In the isolated VR, I enabled ipv6 as:
> > > > > net.ipv6.conf.all.disable_ipv6 = 0
> > > > > net.ipv6.conf.all.forwarding = 1
> > > > > net.ipv6.conf.all.accept_ra = 1
> > > > > net.ipv6.conf.all.accept_redirects = 1
> > > > > net.ipv6.conf.all.autoconf = 1
> > > > >
> > > > > Set up a IPv6 nameserver/dns in /etc/resolve.conf
> > > > > And configured the nics:
> > > > > echo iface eth0 inet6 auto >> /etc/network/interfaces
> > > > > echo iface eth2 inet6 auto >> /etc/network/interfaces
> > > > > /etc/init.d/networking restart
> > > > > Next, restart ACS isolated network without cleanup to have it
> > > > reconfigure IPv4 nics, firewall, NAT etc
> > > > >
> > > > >    *
> > > > > Next, I created a /64 network for the isolated guest network on eth0
> > of
> > > > VR using radvd:
> > > > >
> > > > > # cat /etc/radvd.conf
> > > > > interface eth0
> > > > > {
> > > > >      AdvSendAdvert on;
> > > > >      MinRtrAdvInterval 5;
> > > > >      MaxRtrAdvInterval 15;
> > > > >      prefix 2001:470:ed36:1::/64
> > > > >      {
> > > > >          AdvOnLink on;
> > > > >          AdvAutonomous on;
> > > > >      };
> > > > > };
> > > > > systemctl restart radvd
> > > > > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this
> > > > ...:1::/64 network
> > > > >    *   Finally I added a static route in toy core-router for the new
> > /64
> > > > IPv6 range in the isolated network
> > > > > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev
> > > > <local lan nic>
> > > > >    *
> > > > > ... and I enabled firewall rules to allow any traffic to pass for the
> > > > new /64 network
> > > > >
> > > > > And voila all done! I create a domain AAAA record that points to my
> > > > guest VM IPv6 address a test webserver on
> > > > > http://ipv6-isolated-ntwk-demo.yadav.cloud/
> > > > >
> > > > > (Note: I'll get rid of the tunnel and request a new /48 block after a
> > > > few days, sharing this solely for testing purposes)
> > > > >
> > > > >
> > > > > Regards.
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: IPV6 in Isolated/VPC networks

Posted by Kristaps Cudars <kr...@gmail.com>.
Hello,

I asked networking team to comment on “How should the IPv6 block/allocation work in VPCs?”
Option1: They haven’t seen lately devices with limits on how many static routes can be created.
Option2: With /60 and /62 assignments and big quantity of routers IPv6 assignment from RIPE NNC can be drained fast.

/48 contains 64000 /64
/60 contains 16 /64
64000 / 16 = 4000 routers


On 2021/09/07 11:59:09, Rohit Yadav <ro...@shapeblue.com> wrote: 
> All,
> 
> After another iteration with Alex, I've updated the design doc. Kindly review:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> 
> 
> ... and advise on some outstanding questions:
> 
>   *   How should the IPv6 block/allocation work in VPCs?
> Option1: Should this be simply /64 allocation on any new tier, the cons of this option is one static route/rule per VPC tier. (many upstream routers may have limit on no. of static routes?)
> Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62 (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks global setting (3 is default). The pros of this option are less no. of static route/rule and easy programming, but potential wastage of multiple /64 prefix blocks for unused/uncreated tiers.
>   *   How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage for shared network (design, implementation and use-cases?)
> Option1: We don't do it, all user VMs nics have ipv4 address which whose usage we don't track. For public VR/nics/networks, we can simply add the IPv6 details for a related IPv4 address.
> Option2: Implement a separate, first-class IPv6 address or /64 prefix tracking/management and usage for all VMs and systemvms nic (this means account/domain level limits and new billing/records)
>   *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by default if zone has a IPv6 address block allocated/assigned for use for systemvms (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons of this?)
>   *
> 
> Regards.
> 
> ________________________________
> From: Rohit Yadav <ro...@shapeblue.com>
> Sent: Thursday, August 19, 2021 15:45
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
> 
> Hi all,
> 
> I've taken feedback from this thread and wrote this design doc:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing
> 
> Kindly review and advise if I missed anything or anything that needs to be changed/updated. You may comment on the wiki directly too.
> 
> Kindly suggest your views on the following (also in the design doc above):
> 
> Outstanding Questions:
> 
>   *   Should admin or user be able to specify how VPC super CIDRs are created/needed; for example a user can ask for VPC with /60 super CIDR? Or should CloudStack automatically find/allocate a /64 for a new VPC tier from the root-admin configured /64-/48 block?
>   *   Should we explore FRR and iBGP or other strategies to do dynamic routing which may not require advance/complex configuration in the VR or for the users/admin?
>   *   With SLAAC and no dhcpv6, is there a way to support secondary IPv6 addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's nics?
>   *   Any thoughts on UI/UX for firewall/routing management?
>   *   Any other feature/support for isolated network or VPC feature that must be explored or supported such as PF, VPN, LB, vpc static routes, vpc gateway etc.
>   *   For usage - should we have any consideration, or should we assume that IPv4 and IPv6 address will go together for every nic; so IPv6 usage for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new biling/usage needed?
>   *   For smoketests, local dev-test should we explore ULA? Unique Local Address - in the range fc00::/7. Typically only within the ‘local’ half fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing. This prefix can be randomly generated at first install by CloudStack in a zone using zoneid etc?
>   *   Should we expand support for VR diagnostic feature to include support for ipv6, traceroute6?
>   *   Discuss - expand support/ability to allocate a /60, or /56 etc prefix to an individual VM in shared network (Wido's suggestion)
> 
> 
> Regards.
> 
> ________________________________
> From: Wei ZHOU <us...@gmail.com>
> Sent: Tuesday, August 17, 2021 21:16
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
> 
> Thanks Kristaps, Wido, Rohit and Alex for your replies.
> 
> I am fine with not having stateful dhcpv6 and privacy extension/temporary
> address in phase 1. If community decides not to do eventually , it is also
> ok to me.
> 
> We could explore how to better use secondary ipv6 addresses as Wido
> advised. It would be great if anyone share some user experience.
> 
> -Wei
> 
> 
> On Tuesday, 17 August 2021, Kristaps Cudars <kr...@gmail.com>
> wrote:
> 
> > Hi Wei,
> >
> > Published this month’s RFC 9099 and explains in different
> > words/perspective what has been written by Alex, Rohit and Wido.
> > https://www.rfc-editor.org/rfc/rfc9099.html
> >
> >
> > On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
> > > Hi Wido,
> > >
> > > (cc to Rohit and Alex)
> > >
> > > It is a good suggestion to use FRR for ipv6. The configuration is quite
> > > simple and the VMs can get SLAAC, routes, etc.
> > >
> > > Privacy extension looks not the same as what you mentioned. see
> > > https://datatracker.ietf.org/doc/html/rfc4941
> > >
> > > You are right. To use static routing, the admins need to configure the
> > > routes in the upstream router, and add some ipv6 ranges (eg /56 for VPCs
> > > and /64 for isolated networks) and their next-hop  (which will be
> > > configured in VRs) in CloudStack. CloudStack will pick up an IPv6 range
> > and
> > > assign it to an isolated network or vpc. @Rohit, correct me if I'm wrong.
> > >
> > > I have a question, it looks stateless dhcpv6 (SLAAC from router/VR,
> > > router/dns etc via RA messages) will be the only option for now (related
> > to
> > > your pr https://github.com/apache/cloudstack/pull/3077) . Would it be
> > good
> > > to provide stateful dhcpv6 (which can be implemented by dnsmasq) as an
> > > option in cloudstack ? The advantages are
> > > (1) support other ipv6 cidr sizes than /64.
> > > (2) we can assign a specified Ipv6 address to a vm. vm Ipv6 addresses can
> > > be changed
> > > (4) an Ipv6 addresses can be re-used by multiple vms.
> > > The problem is, stateful dhcpv6 does not support routers,nameservers,
> > etc.
> > > we need to figure it out (probably use radvd/frr and dnsmasq both).
> > >
> > > -Wei
> > >
> > >
> >
> 
> 
> 
>  
> 
> > On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
> > >
> > > > Hi,
> > > >
> > > > See my inline responses:
> > > >
> > > > Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> > > > > Hi all,
> > > > >
> > > > > Thanks for your feedback and ideas, I've gone ahead with discussing
> > them
> > > > with Alex and came up with a PoC/design which can be implemented in the
> > > > following phases:
> > > > >
> > > > >    *   Phase1: implement ipv6 support in isolated networks and VPC
> > with
> > > > static routing
> > > > >    *   Phase2: discuss and implement support for dynamic routing
> > (TBD)
> > > > >
> > > > > For Phase1 here's the high-level proposal:
> > > > >
> > > > >    *   IPv6 address management:
> > > > >       *   At the zone level root-admin specifies a /64 public range
> > that
> > > > will be used for VRs, then they can add a /48, or /56 IPv6 range for
> > guest
> > > > networks (to be used by isolated networks and VPC tiers)
> > > > >       *   On creation of any IPv6 enabled isolated network or VPC
> > tier,
> > > > from the /48 or /56 block a /64 network is allocated/used
> > > > >       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the
> > zone
> > > > (discuss: is privacy a concern, can privacy extensions rfc4941 of
> > slaac be
> > > > explored?)
> > > >
> > > > Privacy Extensions are only a concern for client devices which roam
> > > > between different IPv6 networks.
> > > >
> > > > If you IPv6 address of a client keeps the same suffix (MAC based) and
> > > > switches network then only the prefix (/64) will change.
> > > >
> > > > This way a network like Google, Facebook, etc could track your device
> > > > moving from network to network if they only look at the last 64-bits of
> > > > the IPv6 address.
> > > >
> > > > For servers this is not a problem as you already know in which network
> > > > they are.
> > > >
> > > > >    *   Network offerings: root-admin can create new network offerings
> > > > (with VPC too) that specifies a network stack option:
> > > > >       *   ipv4 only (default, for backward compatibility all
> > > > networks/offerings post-upgrade migrate to this option)
> > > > >       *   ipv4-and-ipv6
> > > > >       *   ipv6-only (this can be phase 1.b)
> > > > >       *   A new routing option: static (phase1), dynamic (phase2,
> > with
> > > > multiple sub-options such as ospf/bgp etc...)
> > > >
> > > > This means that the network admin will need to statically route the
> > IPv6
> > > > subnet to the VR's outside IPv6 address, for example, on a JunOS
> > router:
> > > >
> > > > set routing-options rib inet6.0 static route 2001:db8:500::/48 next-hop
> > > > 2001:db8:100::50
> > > >
> > > > I'm assuming that 2001:db8:100::50 is the address of the VR on the
> > > > outside (/64) network. In reality this will probably be a longer
> > > > address, but this is for just the example.
> > > >
> > > > >    *   VR changes:
> > > > >       *   VR gets its guest and public nics set to inet6 auto
> > > > >       *   For each /64 allocated to guest network and VPC tiers,
> > radvd
> > > > is configured to do RA
> > > >
> > > > radvd is fine, but looking at phase 2 with dynamic routing you might
> > > > already want to look into FRRouting. FRR can also advertise RAs while
> > > > not doing any routing.
> > > >
> > > > interface ens4
> > > >    no ipv6 nd suppress-ra
> > > >    ipv6 nd prefix 2001:db8:500::/64
> > > >    ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> > > >
> > > > See: http://docs.frrouting.org/en/latest/ipv6.html
> > > >
> > > > >       *   Firewall: a new ipv6 zone/chain is created for ipv6 where
> > ipv6
> > > > firewall rules (ACLs, ingress, egress) are implemented; ACLs between
> > VPC
> > > > tiers are managed/implemented by ipv6 firewall on VR
> > > >
> > > > Please take a look at the existing security_group.py script which
> > > > implements RFC4890
> > > >
> > > > https://datatracker.ietf.org/doc/html/rfc4890
> > > >
> > > > ICMPv6 is a vital part of IPv6 and certain packets should always be
> > > > allowed.
> > > >
> > > > >       *   It is assumed that static routes are created on the
> > core/main
> > > > router by the admin or automated using some scripts/tools; for this
> > > > CloudStack will announce events with details of /64 networks and VR's
> > > > public IPv6 address that can be consumed by a rabbitmq/message bus
> > client
> > > > (for example), or a custom cron job or script as part of orchestration.
> > > > (this wouldn't be necessary for dynamic routing bgp with phase2)\\
> > > >
> > > > You would only need to announce the /48 or /56 allocated to the VR,
> > > > that's all. You don't need to inform the upstream router about the /64
> > > > subnets created within that larger subnet.
> > > >
> > > > >    *   Guest Networking: With SLAAC, it's easy for CloudStack to
> > > > calculate allocate and use a /64 and determine the IPv6 address of VR
> > nics
> > > > and guest VM nics
> > > > >       *   A user create an isolated network/VPC with an offering
> > that is
> > > > ipv6 enabled
> > > > >       *   A user can manage firewall for the IPv6 address/guest nics;
> > > > there'll be no port forward and LB feature though for IPv6
> > > > >       *   A users can run workloads in the guest VMs that listen on
> > > > publically routable ipv6 addresses
> > > > >       *   Usage/billing etc continue to work, no change needed
> > > > >
> > > > > Network layout:
> > > > >
> > > > > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] ->
> > > > [guest VMs/nics]
> > > > > *core/ISP router needs static routes to be added (manually or
> > > > automated), assumes a /48 or /56 configured for the zone
> > > > >
> > > > > Thoughts, feedback?
> > > >
> > > > Looks doable!
> > > >
> > > > Side-note: It would be very cool if you could use parts of this
> > > > implementation to also route /48, /56, or /60 subnets to individual VMs
> > > > in Shared networks.
> > > >
> > > > Why? This allows for running Docker containers with native IPv6 inside
> > > > the VM or running a (Open)VPN server inside a VM which then assigns
> > > > public IPv6 addresses to clients connected.
> > > >
> > > > Instead of routing the subnet to a VR we route the subnet to a single
> > > > instance in a shared network.
> > > >
> > > > If we could then also move these subnets between Instances easily one
> > > > can quickly migrate to a different instance while keeping the same IPv6
> > > > subnet.
> > > >
> > > > Wido
> > > >
> > > > >
> > > > > Proof-of-concept commentary: here's what I did to test the idea:
> > > > >
> > > > >    *   Created an isolated network and deployed a VM in my home lab
> > > > > The VR running on KVM has following nics
> > > > > eth0 - guest network
> > > > > eth1 - link local
> > > > > eth2 - public network
> > > > >
> > > > >    *   I setup a custom openwrt router on a RPi4 to serve as a
> > toy-core
> > > > router where I create a wan6 IPv6 tunnel using tunnel broker and I got
> > a
> > > > /48 allocated. My configuration looks like:
> > > > > /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> > > > > /64 - 2001:470:36:3e2::/64 (default allocated by)
> > > > >
> > > > > I create a LAN ipv6 (public network for CloudStack VR): at
> > subnet/prefix
> > > > 0:
> > > > > LAN IPv6 address: 2001:470:ed36:0::1/64
> > > > > Address mode: SLAAC+stateless DHCP (no dhcpv6)
> > > > >    *
> > > > >    *
> > > > > In the isolated VR, I enabled ipv6 as:
> > > > > net.ipv6.conf.all.disable_ipv6 = 0
> > > > > net.ipv6.conf.all.forwarding = 1
> > > > > net.ipv6.conf.all.accept_ra = 1
> > > > > net.ipv6.conf.all.accept_redirects = 1
> > > > > net.ipv6.conf.all.autoconf = 1
> > > > >
> > > > > Set up a IPv6 nameserver/dns in /etc/resolve.conf
> > > > > And configured the nics:
> > > > > echo iface eth0 inet6 auto >> /etc/network/interfaces
> > > > > echo iface eth2 inet6 auto >> /etc/network/interfaces
> > > > > /etc/init.d/networking restart
> > > > > Next, restart ACS isolated network without cleanup to have it
> > > > reconfigure IPv4 nics, firewall, NAT etc
> > > > >
> > > > >    *
> > > > > Next, I created a /64 network for the isolated guest network on eth0
> > of
> > > > VR using radvd:
> > > > >
> > > > > # cat /etc/radvd.conf
> > > > > interface eth0
> > > > > {
> > > > >      AdvSendAdvert on;
> > > > >      MinRtrAdvInterval 5;
> > > > >      MaxRtrAdvInterval 15;
> > > > >      prefix 2001:470:ed36:1::/64
> > > > >      {
> > > > >          AdvOnLink on;
> > > > >          AdvAutonomous on;
> > > > >      };
> > > > > };
> > > > > systemctl restart radvd
> > > > > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this
> > > > ...:1::/64 network
> > > > >    *   Finally I added a static route in toy core-router for the new
> > /64
> > > > IPv6 range in the isolated network
> > > > > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev
> > > > <local lan nic>
> > > > >    *
> > > > > ... and I enabled firewall rules to allow any traffic to pass for the
> > > > new /64 network
> > > > >
> > > > > And voila all done! I create a domain AAAA record that points to my
> > > > guest VM IPv6 address a test webserver on
> > > > > http://ipv6-isolated-ntwk-demo.yadav.cloud/
> > > > >
> > > > > (Note: I'll get rid of the tunnel and request a new /48 block after a
> > > > few days, sharing this solely for testing purposes)
> > > > >
> > > > >
> > > > > Regards.
> > > > >
> > > > >
> > > >
> > >
> >
> 

Re: IPV6 in Isolated/VPC networks

Posted by Rohit Yadav <ro...@shapeblue.com>.
All,

After another iteration with Alex, I've updated the design doc. Kindly review:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing


... and advise on some outstanding questions:

  *   How should the IPv6 block/allocation work in VPCs?
Option1: Should this be simply /64 allocation on any new tier, the cons of this option is one static route/rule per VPC tier. (many upstream routers may have limit on no. of static routes?)
Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62 (4 tiers) for the VPC, this can be filtered based on the vpc.max.networks global setting (3 is default). The pros of this option are less no. of static route/rule and easy programming, but potential wastage of multiple /64 prefix blocks for unused/uncreated tiers.
  *   How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage for shared network (design, implementation and use-cases?)
Option1: We don't do it, all user VMs nics have ipv4 address which whose usage we don't track. For public VR/nics/networks, we can simply add the IPv6 details for a related IPv4 address.
Option2: Implement a separate, first-class IPv6 address or /64 prefix tracking/management and usage for all VMs and systemvms nic (this means account/domain level limits and new billing/records)
  *   Enable IPv6 on systemvms (specifically SSVM and CPVM) by default if zone has a IPv6 address block allocated/assigned for use for systemvms (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons of this?)
  *

Regards.

________________________________
From: Rohit Yadav <ro...@shapeblue.com>
Sent: Thursday, August 19, 2021 15:45
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks

Hi all,

I've taken feedback from this thread and wrote this design doc:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing

Kindly review and advise if I missed anything or anything that needs to be changed/updated. You may comment on the wiki directly too.

Kindly suggest your views on the following (also in the design doc above):

Outstanding Questions:

  *   Should admin or user be able to specify how VPC super CIDRs are created/needed; for example a user can ask for VPC with /60 super CIDR? Or should CloudStack automatically find/allocate a /64 for a new VPC tier from the root-admin configured /64-/48 block?
  *   Should we explore FRR and iBGP or other strategies to do dynamic routing which may not require advance/complex configuration in the VR or for the users/admin?
  *   With SLAAC and no dhcpv6, is there a way to support secondary IPv6 addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's nics?
  *   Any thoughts on UI/UX for firewall/routing management?
  *   Any other feature/support for isolated network or VPC feature that must be explored or supported such as PF, VPN, LB, vpc static routes, vpc gateway etc.
  *   For usage - should we have any consideration, or should we assume that IPv4 and IPv6 address will go together for every nic; so IPv6 usage for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new biling/usage needed?
  *   For smoketests, local dev-test should we explore ULA? Unique Local Address - in the range fc00::/7. Typically only within the ‘local’ half fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing. This prefix can be randomly generated at first install by CloudStack in a zone using zoneid etc?
  *   Should we expand support for VR diagnostic feature to include support for ipv6, traceroute6?
  *   Discuss - expand support/ability to allocate a /60, or /56 etc prefix to an individual VM in shared network (Wido's suggestion)


Regards.

________________________________
From: Wei ZHOU <us...@gmail.com>
Sent: Tuesday, August 17, 2021 21:16
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks

Thanks Kristaps, Wido, Rohit and Alex for your replies.

I am fine with not having stateful dhcpv6 and privacy extension/temporary
address in phase 1. If community decides not to do eventually , it is also
ok to me.

We could explore how to better use secondary ipv6 addresses as Wido
advised. It would be great if anyone share some user experience.

-Wei


On Tuesday, 17 August 2021, Kristaps Cudars <kr...@gmail.com>
wrote:

> Hi Wei,
>
> Published this month’s RFC 9099 and explains in different
> words/perspective what has been written by Alex, Rohit and Wido.
> https://www.rfc-editor.org/rfc/rfc9099.html
>
>
> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
> > Hi Wido,
> >
> > (cc to Rohit and Alex)
> >
> > It is a good suggestion to use FRR for ipv6. The configuration is quite
> > simple and the VMs can get SLAAC, routes, etc.
> >
> > Privacy extension looks not the same as what you mentioned. see
> > https://datatracker.ietf.org/doc/html/rfc4941
> >
> > You are right. To use static routing, the admins need to configure the
> > routes in the upstream router, and add some ipv6 ranges (eg /56 for VPCs
> > and /64 for isolated networks) and their next-hop  (which will be
> > configured in VRs) in CloudStack. CloudStack will pick up an IPv6 range
> and
> > assign it to an isolated network or vpc. @Rohit, correct me if I'm wrong.
> >
> > I have a question, it looks stateless dhcpv6 (SLAAC from router/VR,
> > router/dns etc via RA messages) will be the only option for now (related
> to
> > your pr https://github.com/apache/cloudstack/pull/3077) . Would it be
> good
> > to provide stateful dhcpv6 (which can be implemented by dnsmasq) as an
> > option in cloudstack ? The advantages are
> > (1) support other ipv6 cidr sizes than /64.
> > (2) we can assign a specified Ipv6 address to a vm. vm Ipv6 addresses can
> > be changed
> > (4) an Ipv6 addresses can be re-used by multiple vms.
> > The problem is, stateful dhcpv6 does not support routers,nameservers,
> etc.
> > we need to figure it out (probably use radvd/frr and dnsmasq both).
> >
> > -Wei
> >
> >
>



 

> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
> >
> > > Hi,
> > >
> > > See my inline responses:
> > >
> > > Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> > > > Hi all,
> > > >
> > > > Thanks for your feedback and ideas, I've gone ahead with discussing
> them
> > > with Alex and came up with a PoC/design which can be implemented in the
> > > following phases:
> > > >
> > > >    *   Phase1: implement ipv6 support in isolated networks and VPC
> with
> > > static routing
> > > >    *   Phase2: discuss and implement support for dynamic routing
> (TBD)
> > > >
> > > > For Phase1 here's the high-level proposal:
> > > >
> > > >    *   IPv6 address management:
> > > >       *   At the zone level root-admin specifies a /64 public range
> that
> > > will be used for VRs, then they can add a /48, or /56 IPv6 range for
> guest
> > > networks (to be used by isolated networks and VPC tiers)
> > > >       *   On creation of any IPv6 enabled isolated network or VPC
> tier,
> > > from the /48 or /56 block a /64 network is allocated/used
> > > >       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the
> zone
> > > (discuss: is privacy a concern, can privacy extensions rfc4941 of
> slaac be
> > > explored?)
> > >
> > > Privacy Extensions are only a concern for client devices which roam
> > > between different IPv6 networks.
> > >
> > > If you IPv6 address of a client keeps the same suffix (MAC based) and
> > > switches network then only the prefix (/64) will change.
> > >
> > > This way a network like Google, Facebook, etc could track your device
> > > moving from network to network if they only look at the last 64-bits of
> > > the IPv6 address.
> > >
> > > For servers this is not a problem as you already know in which network
> > > they are.
> > >
> > > >    *   Network offerings: root-admin can create new network offerings
> > > (with VPC too) that specifies a network stack option:
> > > >       *   ipv4 only (default, for backward compatibility all
> > > networks/offerings post-upgrade migrate to this option)
> > > >       *   ipv4-and-ipv6
> > > >       *   ipv6-only (this can be phase 1.b)
> > > >       *   A new routing option: static (phase1), dynamic (phase2,
> with
> > > multiple sub-options such as ospf/bgp etc...)
> > >
> > > This means that the network admin will need to statically route the
> IPv6
> > > subnet to the VR's outside IPv6 address, for example, on a JunOS
> router:
> > >
> > > set routing-options rib inet6.0 static route 2001:db8:500::/48 next-hop
> > > 2001:db8:100::50
> > >
> > > I'm assuming that 2001:db8:100::50 is the address of the VR on the
> > > outside (/64) network. In reality this will probably be a longer
> > > address, but this is for just the example.
> > >
> > > >    *   VR changes:
> > > >       *   VR gets its guest and public nics set to inet6 auto
> > > >       *   For each /64 allocated to guest network and VPC tiers,
> radvd
> > > is configured to do RA
> > >
> > > radvd is fine, but looking at phase 2 with dynamic routing you might
> > > already want to look into FRRouting. FRR can also advertise RAs while
> > > not doing any routing.
> > >
> > > interface ens4
> > >    no ipv6 nd suppress-ra
> > >    ipv6 nd prefix 2001:db8:500::/64
> > >    ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> > >
> > > See: http://docs.frrouting.org/en/latest/ipv6.html
> > >
> > > >       *   Firewall: a new ipv6 zone/chain is created for ipv6 where
> ipv6
> > > firewall rules (ACLs, ingress, egress) are implemented; ACLs between
> VPC
> > > tiers are managed/implemented by ipv6 firewall on VR
> > >
> > > Please take a look at the existing security_group.py script which
> > > implements RFC4890
> > >
> > > https://datatracker.ietf.org/doc/html/rfc4890
> > >
> > > ICMPv6 is a vital part of IPv6 and certain packets should always be
> > > allowed.
> > >
> > > >       *   It is assumed that static routes are created on the
> core/main
> > > router by the admin or automated using some scripts/tools; for this
> > > CloudStack will announce events with details of /64 networks and VR's
> > > public IPv6 address that can be consumed by a rabbitmq/message bus
> client
> > > (for example), or a custom cron job or script as part of orchestration.
> > > (this wouldn't be necessary for dynamic routing bgp with phase2)\\
> > >
> > > You would only need to announce the /48 or /56 allocated to the VR,
> > > that's all. You don't need to inform the upstream router about the /64
> > > subnets created within that larger subnet.
> > >
> > > >    *   Guest Networking: With SLAAC, it's easy for CloudStack to
> > > calculate allocate and use a /64 and determine the IPv6 address of VR
> nics
> > > and guest VM nics
> > > >       *   A user create an isolated network/VPC with an offering
> that is
> > > ipv6 enabled
> > > >       *   A user can manage firewall for the IPv6 address/guest nics;
> > > there'll be no port forward and LB feature though for IPv6
> > > >       *   A users can run workloads in the guest VMs that listen on
> > > publically routable ipv6 addresses
> > > >       *   Usage/billing etc continue to work, no change needed
> > > >
> > > > Network layout:
> > > >
> > > > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] ->
> > > [guest VMs/nics]
> > > > *core/ISP router needs static routes to be added (manually or
> > > automated), assumes a /48 or /56 configured for the zone
> > > >
> > > > Thoughts, feedback?
> > >
> > > Looks doable!
> > >
> > > Side-note: It would be very cool if you could use parts of this
> > > implementation to also route /48, /56, or /60 subnets to individual VMs
> > > in Shared networks.
> > >
> > > Why? This allows for running Docker containers with native IPv6 inside
> > > the VM or running a (Open)VPN server inside a VM which then assigns
> > > public IPv6 addresses to clients connected.
> > >
> > > Instead of routing the subnet to a VR we route the subnet to a single
> > > instance in a shared network.
> > >
> > > If we could then also move these subnets between Instances easily one
> > > can quickly migrate to a different instance while keeping the same IPv6
> > > subnet.
> > >
> > > Wido
> > >
> > > >
> > > > Proof-of-concept commentary: here's what I did to test the idea:
> > > >
> > > >    *   Created an isolated network and deployed a VM in my home lab
> > > > The VR running on KVM has following nics
> > > > eth0 - guest network
> > > > eth1 - link local
> > > > eth2 - public network
> > > >
> > > >    *   I setup a custom openwrt router on a RPi4 to serve as a
> toy-core
> > > router where I create a wan6 IPv6 tunnel using tunnel broker and I got
> a
> > > /48 allocated. My configuration looks like:
> > > > /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> > > > /64 - 2001:470:36:3e2::/64 (default allocated by)
> > > >
> > > > I create a LAN ipv6 (public network for CloudStack VR): at
> subnet/prefix
> > > 0:
> > > > LAN IPv6 address: 2001:470:ed36:0::1/64
> > > > Address mode: SLAAC+stateless DHCP (no dhcpv6)
> > > >    *
> > > >    *
> > > > In the isolated VR, I enabled ipv6 as:
> > > > net.ipv6.conf.all.disable_ipv6 = 0
> > > > net.ipv6.conf.all.forwarding = 1
> > > > net.ipv6.conf.all.accept_ra = 1
> > > > net.ipv6.conf.all.accept_redirects = 1
> > > > net.ipv6.conf.all.autoconf = 1
> > > >
> > > > Set up a IPv6 nameserver/dns in /etc/resolve.conf
> > > > And configured the nics:
> > > > echo iface eth0 inet6 auto >> /etc/network/interfaces
> > > > echo iface eth2 inet6 auto >> /etc/network/interfaces
> > > > /etc/init.d/networking restart
> > > > Next, restart ACS isolated network without cleanup to have it
> > > reconfigure IPv4 nics, firewall, NAT etc
> > > >
> > > >    *
> > > > Next, I created a /64 network for the isolated guest network on eth0
> of
> > > VR using radvd:
> > > >
> > > > # cat /etc/radvd.conf
> > > > interface eth0
> > > > {
> > > >      AdvSendAdvert on;
> > > >      MinRtrAdvInterval 5;
> > > >      MaxRtrAdvInterval 15;
> > > >      prefix 2001:470:ed36:1::/64
> > > >      {
> > > >          AdvOnLink on;
> > > >          AdvAutonomous on;
> > > >      };
> > > > };
> > > > systemctl restart radvd
> > > > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this
> > > ...:1::/64 network
> > > >    *   Finally I added a static route in toy core-router for the new
> /64
> > > IPv6 range in the isolated network
> > > > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev
> > > <local lan nic>
> > > >    *
> > > > ... and I enabled firewall rules to allow any traffic to pass for the
> > > new /64 network
> > > >
> > > > And voila all done! I create a domain AAAA record that points to my
> > > guest VM IPv6 address a test webserver on
> > > > http://ipv6-isolated-ntwk-demo.yadav.cloud/
> > > >
> > > > (Note: I'll get rid of the tunnel and request a new /48 block after a
> > > few days, sharing this solely for testing purposes)
> > > >
> > > >
> > > > Regards.
> > > >
> > > >
> > >
> >
>

Re: IPV6 in Isolated/VPC networks

Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi all,

I've taken feedback from this thread and wrote this design doc:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing

Kindly review and advise if I missed anything or anything that needs to be changed/updated. You may comment on the wiki directly too.

Kindly suggest your views on the following (also in the design doc above):

Outstanding Questions:

  *   Should admin or user be able to specify how VPC super CIDRs are created/needed; for example a user can ask for VPC with /60 super CIDR? Or should CloudStack automatically find/allocate a /64 for a new VPC tier from the root-admin configured /64-/48 block?
  *   Should we explore FRR and iBGP or other strategies to do dynamic routing which may not require advance/complex configuration in the VR or for the users/admin?
  *   With SLAAC and no dhcpv6, is there a way to support secondary IPv6 addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's nics?
  *   Any thoughts on UI/UX for firewall/routing management?
  *   Any other feature/support for isolated network or VPC feature that must be explored or supported such as PF, VPN, LB, vpc static routes, vpc gateway etc.
  *   For usage - should we have any consideration, or should we assume that IPv4 and IPv6 address will go together for every nic; so IPv6 usage for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new biling/usage needed?
  *   For smoketests, local dev-test should we explore ULA? Unique Local Address - in the range fc00::/7. Typically only within the ‘local’ half fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing. This prefix can be randomly generated at first install by CloudStack in a zone using zoneid etc?
  *   Should we expand support for VR diagnostic feature to include support for ipv6, traceroute6?
  *   Discuss - expand support/ability to allocate a /60, or /56 etc prefix to an individual VM in shared network (Wido's suggestion)


Regards.

________________________________
From: Wei ZHOU <us...@gmail.com>
Sent: Tuesday, August 17, 2021 21:16
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks

Thanks Kristaps, Wido, Rohit and Alex for your replies.

I am fine with not having stateful dhcpv6 and privacy extension/temporary
address in phase 1. If community decides not to do eventually , it is also
ok to me.

We could explore how to better use secondary ipv6 addresses as Wido
advised. It would be great if anyone share some user experience.

-Wei


On Tuesday, 17 August 2021, Kristaps Cudars <kr...@gmail.com>
wrote:

> Hi Wei,
>
> Published this month’s RFC 9099 and explains in different
> words/perspective what has been written by Alex, Rohit and Wido.
> https://www.rfc-editor.org/rfc/rfc9099.html
>
>
> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
> > Hi Wido,
> >
> > (cc to Rohit and Alex)
> >
> > It is a good suggestion to use FRR for ipv6. The configuration is quite
> > simple and the VMs can get SLAAC, routes, etc.
> >
> > Privacy extension looks not the same as what you mentioned. see
> > https://datatracker.ietf.org/doc/html/rfc4941
> >
> > You are right. To use static routing, the admins need to configure the
> > routes in the upstream router, and add some ipv6 ranges (eg /56 for VPCs
> > and /64 for isolated networks) and their next-hop  (which will be
> > configured in VRs) in CloudStack. CloudStack will pick up an IPv6 range
> and
> > assign it to an isolated network or vpc. @Rohit, correct me if I'm wrong.
> >
> > I have a question, it looks stateless dhcpv6 (SLAAC from router/VR,
> > router/dns etc via RA messages) will be the only option for now (related
> to
> > your pr https://github.com/apache/cloudstack/pull/3077) . Would it be
> good
> > to provide stateful dhcpv6 (which can be implemented by dnsmasq) as an
> > option in cloudstack ? The advantages are
> > (1) support other ipv6 cidr sizes than /64.
> > (2) we can assign a specified Ipv6 address to a vm. vm Ipv6 addresses can
> > be changed
> > (4) an Ipv6 addresses can be re-used by multiple vms.
> > The problem is, stateful dhcpv6 does not support routers,nameservers,
> etc.
> > we need to figure it out (probably use radvd/frr and dnsmasq both).
> >
> > -Wei
> >
> >
> 
 

> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
> >
> > > Hi,
> > >
> > > See my inline responses:
> > >
> > > Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> > > > Hi all,
> > > >
> > > > Thanks for your feedback and ideas, I've gone ahead with discussing
> them
> > > with Alex and came up with a PoC/design which can be implemented in the
> > > following phases:
> > > >
> > > >    *   Phase1: implement ipv6 support in isolated networks and VPC
> with
> > > static routing
> > > >    *   Phase2: discuss and implement support for dynamic routing
> (TBD)
> > > >
> > > > For Phase1 here's the high-level proposal:
> > > >
> > > >    *   IPv6 address management:
> > > >       *   At the zone level root-admin specifies a /64 public range
> that
> > > will be used for VRs, then they can add a /48, or /56 IPv6 range for
> guest
> > > networks (to be used by isolated networks and VPC tiers)
> > > >       *   On creation of any IPv6 enabled isolated network or VPC
> tier,
> > > from the /48 or /56 block a /64 network is allocated/used
> > > >       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the
> zone
> > > (discuss: is privacy a concern, can privacy extensions rfc4941 of
> slaac be
> > > explored?)
> > >
> > > Privacy Extensions are only a concern for client devices which roam
> > > between different IPv6 networks.
> > >
> > > If you IPv6 address of a client keeps the same suffix (MAC based) and
> > > switches network then only the prefix (/64) will change.
> > >
> > > This way a network like Google, Facebook, etc could track your device
> > > moving from network to network if they only look at the last 64-bits of
> > > the IPv6 address.
> > >
> > > For servers this is not a problem as you already know in which network
> > > they are.
> > >
> > > >    *   Network offerings: root-admin can create new network offerings
> > > (with VPC too) that specifies a network stack option:
> > > >       *   ipv4 only (default, for backward compatibility all
> > > networks/offerings post-upgrade migrate to this option)
> > > >       *   ipv4-and-ipv6
> > > >       *   ipv6-only (this can be phase 1.b)
> > > >       *   A new routing option: static (phase1), dynamic (phase2,
> with
> > > multiple sub-options such as ospf/bgp etc...)
> > >
> > > This means that the network admin will need to statically route the
> IPv6
> > > subnet to the VR's outside IPv6 address, for example, on a JunOS
> router:
> > >
> > > set routing-options rib inet6.0 static route 2001:db8:500::/48 next-hop
> > > 2001:db8:100::50
> > >
> > > I'm assuming that 2001:db8:100::50 is the address of the VR on the
> > > outside (/64) network. In reality this will probably be a longer
> > > address, but this is for just the example.
> > >
> > > >    *   VR changes:
> > > >       *   VR gets its guest and public nics set to inet6 auto
> > > >       *   For each /64 allocated to guest network and VPC tiers,
> radvd
> > > is configured to do RA
> > >
> > > radvd is fine, but looking at phase 2 with dynamic routing you might
> > > already want to look into FRRouting. FRR can also advertise RAs while
> > > not doing any routing.
> > >
> > > interface ens4
> > >    no ipv6 nd suppress-ra
> > >    ipv6 nd prefix 2001:db8:500::/64
> > >    ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> > >
> > > See: http://docs.frrouting.org/en/latest/ipv6.html
> > >
> > > >       *   Firewall: a new ipv6 zone/chain is created for ipv6 where
> ipv6
> > > firewall rules (ACLs, ingress, egress) are implemented; ACLs between
> VPC
> > > tiers are managed/implemented by ipv6 firewall on VR
> > >
> > > Please take a look at the existing security_group.py script which
> > > implements RFC4890
> > >
> > > https://datatracker.ietf.org/doc/html/rfc4890
> > >
> > > ICMPv6 is a vital part of IPv6 and certain packets should always be
> > > allowed.
> > >
> > > >       *   It is assumed that static routes are created on the
> core/main
> > > router by the admin or automated using some scripts/tools; for this
> > > CloudStack will announce events with details of /64 networks and VR's
> > > public IPv6 address that can be consumed by a rabbitmq/message bus
> client
> > > (for example), or a custom cron job or script as part of orchestration.
> > > (this wouldn't be necessary for dynamic routing bgp with phase2)\\
> > >
> > > You would only need to announce the /48 or /56 allocated to the VR,
> > > that's all. You don't need to inform the upstream router about the /64
> > > subnets created within that larger subnet.
> > >
> > > >    *   Guest Networking: With SLAAC, it's easy for CloudStack to
> > > calculate allocate and use a /64 and determine the IPv6 address of VR
> nics
> > > and guest VM nics
> > > >       *   A user create an isolated network/VPC with an offering
> that is
> > > ipv6 enabled
> > > >       *   A user can manage firewall for the IPv6 address/guest nics;
> > > there'll be no port forward and LB feature though for IPv6
> > > >       *   A users can run workloads in the guest VMs that listen on
> > > publically routable ipv6 addresses
> > > >       *   Usage/billing etc continue to work, no change needed
> > > >
> > > > Network layout:
> > > >
> > > > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] ->
> > > [guest VMs/nics]
> > > > *core/ISP router needs static routes to be added (manually or
> > > automated), assumes a /48 or /56 configured for the zone
> > > >
> > > > Thoughts, feedback?
> > >
> > > Looks doable!
> > >
> > > Side-note: It would be very cool if you could use parts of this
> > > implementation to also route /48, /56, or /60 subnets to individual VMs
> > > in Shared networks.
> > >
> > > Why? This allows for running Docker containers with native IPv6 inside
> > > the VM or running a (Open)VPN server inside a VM which then assigns
> > > public IPv6 addresses to clients connected.
> > >
> > > Instead of routing the subnet to a VR we route the subnet to a single
> > > instance in a shared network.
> > >
> > > If we could then also move these subnets between Instances easily one
> > > can quickly migrate to a different instance while keeping the same IPv6
> > > subnet.
> > >
> > > Wido
> > >
> > > >
> > > > Proof-of-concept commentary: here's what I did to test the idea:
> > > >
> > > >    *   Created an isolated network and deployed a VM in my home lab
> > > > The VR running on KVM has following nics
> > > > eth0 - guest network
> > > > eth1 - link local
> > > > eth2 - public network
> > > >
> > > >    *   I setup a custom openwrt router on a RPi4 to serve as a
> toy-core
> > > router where I create a wan6 IPv6 tunnel using tunnel broker and I got
> a
> > > /48 allocated. My configuration looks like:
> > > > /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> > > > /64 - 2001:470:36:3e2::/64 (default allocated by)
> > > >
> > > > I create a LAN ipv6 (public network for CloudStack VR): at
> subnet/prefix
> > > 0:
> > > > LAN IPv6 address: 2001:470:ed36:0::1/64
> > > > Address mode: SLAAC+stateless DHCP (no dhcpv6)
> > > >    *
> > > >    *
> > > > In the isolated VR, I enabled ipv6 as:
> > > > net.ipv6.conf.all.disable_ipv6 = 0
> > > > net.ipv6.conf.all.forwarding = 1
> > > > net.ipv6.conf.all.accept_ra = 1
> > > > net.ipv6.conf.all.accept_redirects = 1
> > > > net.ipv6.conf.all.autoconf = 1
> > > >
> > > > Set up a IPv6 nameserver/dns in /etc/resolve.conf
> > > > And configured the nics:
> > > > echo iface eth0 inet6 auto >> /etc/network/interfaces
> > > > echo iface eth2 inet6 auto >> /etc/network/interfaces
> > > > /etc/init.d/networking restart
> > > > Next, restart ACS isolated network without cleanup to have it
> > > reconfigure IPv4 nics, firewall, NAT etc
> > > >
> > > >    *
> > > > Next, I created a /64 network for the isolated guest network on eth0
> of
> > > VR using radvd:
> > > >
> > > > # cat /etc/radvd.conf
> > > > interface eth0
> > > > {
> > > >      AdvSendAdvert on;
> > > >      MinRtrAdvInterval 5;
> > > >      MaxRtrAdvInterval 15;
> > > >      prefix 2001:470:ed36:1::/64
> > > >      {
> > > >          AdvOnLink on;
> > > >          AdvAutonomous on;
> > > >      };
> > > > };
> > > > systemctl restart radvd
> > > > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this
> > > ...:1::/64 network
> > > >    *   Finally I added a static route in toy core-router for the new
> /64
> > > IPv6 range in the isolated network
> > > > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev
> > > <local lan nic>
> > > >    *
> > > > ... and I enabled firewall rules to allow any traffic to pass for the
> > > new /64 network
> > > >
> > > > And voila all done! I create a domain AAAA record that points to my
> > > guest VM IPv6 address a test webserver on
> > > > http://ipv6-isolated-ntwk-demo.yadav.cloud/
> > > >
> > > > (Note: I'll get rid of the tunnel and request a new /48 block after a
> > > few days, sharing this solely for testing purposes)
> > > >
> > > >
> > > > Regards.
> > > >
> > > >
> > >
> >
>

Re: IPV6 in Isolated/VPC networks

Posted by Wei ZHOU <us...@gmail.com>.
Thanks Kristaps, Wido, Rohit and Alex for your replies.

I am fine with not having stateful dhcpv6 and privacy extension/temporary
address in phase 1. If community decides not to do eventually , it is also
ok to me.

We could explore how to better use secondary ipv6 addresses as Wido
advised. It would be great if anyone share some user experience.

-Wei


On Tuesday, 17 August 2021, Kristaps Cudars <kr...@gmail.com>
wrote:

> Hi Wei,
>
> Published this month’s RFC 9099 and explains in different
> words/perspective what has been written by Alex, Rohit and Wido.
> https://www.rfc-editor.org/rfc/rfc9099.html
>
>
> On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote:
> > Hi Wido,
> >
> > (cc to Rohit and Alex)
> >
> > It is a good suggestion to use FRR for ipv6. The configuration is quite
> > simple and the VMs can get SLAAC, routes, etc.
> >
> > Privacy extension looks not the same as what you mentioned. see
> > https://datatracker.ietf.org/doc/html/rfc4941
> >
> > You are right. To use static routing, the admins need to configure the
> > routes in the upstream router, and add some ipv6 ranges (eg /56 for VPCs
> > and /64 for isolated networks) and their next-hop  (which will be
> > configured in VRs) in CloudStack. CloudStack will pick up an IPv6 range
> and
> > assign it to an isolated network or vpc. @Rohit, correct me if I'm wrong.
> >
> > I have a question, it looks stateless dhcpv6 (SLAAC from router/VR,
> > router/dns etc via RA messages) will be the only option for now (related
> to
> > your pr https://github.com/apache/cloudstack/pull/3077) . Would it be
> good
> > to provide stateful dhcpv6 (which can be implemented by dnsmasq) as an
> > option in cloudstack ? The advantages are
> > (1) support other ipv6 cidr sizes than /64.
> > (2) we can assign a specified Ipv6 address to a vm. vm Ipv6 addresses can
> > be changed
> > (4) an Ipv6 addresses can be re-used by multiple vms.
> > The problem is, stateful dhcpv6 does not support routers,nameservers,
> etc.
> > we need to figure it out (probably use radvd/frr and dnsmasq both).
> >
> > -Wei
> >
> >
> > On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
> >
> > > Hi,
> > >
> > > See my inline responses:
> > >
> > > Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> > > > Hi all,
> > > >
> > > > Thanks for your feedback and ideas, I've gone ahead with discussing
> them
> > > with Alex and came up with a PoC/design which can be implemented in the
> > > following phases:
> > > >
> > > >    *   Phase1: implement ipv6 support in isolated networks and VPC
> with
> > > static routing
> > > >    *   Phase2: discuss and implement support for dynamic routing
> (TBD)
> > > >
> > > > For Phase1 here's the high-level proposal:
> > > >
> > > >    *   IPv6 address management:
> > > >       *   At the zone level root-admin specifies a /64 public range
> that
> > > will be used for VRs, then they can add a /48, or /56 IPv6 range for
> guest
> > > networks (to be used by isolated networks and VPC tiers)
> > > >       *   On creation of any IPv6 enabled isolated network or VPC
> tier,
> > > from the /48 or /56 block a /64 network is allocated/used
> > > >       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the
> zone
> > > (discuss: is privacy a concern, can privacy extensions rfc4941 of
> slaac be
> > > explored?)
> > >
> > > Privacy Extensions are only a concern for client devices which roam
> > > between different IPv6 networks.
> > >
> > > If you IPv6 address of a client keeps the same suffix (MAC based) and
> > > switches network then only the prefix (/64) will change.
> > >
> > > This way a network like Google, Facebook, etc could track your device
> > > moving from network to network if they only look at the last 64-bits of
> > > the IPv6 address.
> > >
> > > For servers this is not a problem as you already know in which network
> > > they are.
> > >
> > > >    *   Network offerings: root-admin can create new network offerings
> > > (with VPC too) that specifies a network stack option:
> > > >       *   ipv4 only (default, for backward compatibility all
> > > networks/offerings post-upgrade migrate to this option)
> > > >       *   ipv4-and-ipv6
> > > >       *   ipv6-only (this can be phase 1.b)
> > > >       *   A new routing option: static (phase1), dynamic (phase2,
> with
> > > multiple sub-options such as ospf/bgp etc...)
> > >
> > > This means that the network admin will need to statically route the
> IPv6
> > > subnet to the VR's outside IPv6 address, for example, on a JunOS
> router:
> > >
> > > set routing-options rib inet6.0 static route 2001:db8:500::/48 next-hop
> > > 2001:db8:100::50
> > >
> > > I'm assuming that 2001:db8:100::50 is the address of the VR on the
> > > outside (/64) network. In reality this will probably be a longer
> > > address, but this is for just the example.
> > >
> > > >    *   VR changes:
> > > >       *   VR gets its guest and public nics set to inet6 auto
> > > >       *   For each /64 allocated to guest network and VPC tiers,
> radvd
> > > is configured to do RA
> > >
> > > radvd is fine, but looking at phase 2 with dynamic routing you might
> > > already want to look into FRRouting. FRR can also advertise RAs while
> > > not doing any routing.
> > >
> > > interface ens4
> > >    no ipv6 nd suppress-ra
> > >    ipv6 nd prefix 2001:db8:500::/64
> > >    ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> > >
> > > See: http://docs.frrouting.org/en/latest/ipv6.html
> > >
> > > >       *   Firewall: a new ipv6 zone/chain is created for ipv6 where
> ipv6
> > > firewall rules (ACLs, ingress, egress) are implemented; ACLs between
> VPC
> > > tiers are managed/implemented by ipv6 firewall on VR
> > >
> > > Please take a look at the existing security_group.py script which
> > > implements RFC4890
> > >
> > > https://datatracker.ietf.org/doc/html/rfc4890
> > >
> > > ICMPv6 is a vital part of IPv6 and certain packets should always be
> > > allowed.
> > >
> > > >       *   It is assumed that static routes are created on the
> core/main
> > > router by the admin or automated using some scripts/tools; for this
> > > CloudStack will announce events with details of /64 networks and VR's
> > > public IPv6 address that can be consumed by a rabbitmq/message bus
> client
> > > (for example), or a custom cron job or script as part of orchestration.
> > > (this wouldn't be necessary for dynamic routing bgp with phase2)\\
> > >
> > > You would only need to announce the /48 or /56 allocated to the VR,
> > > that's all. You don't need to inform the upstream router about the /64
> > > subnets created within that larger subnet.
> > >
> > > >    *   Guest Networking: With SLAAC, it's easy for CloudStack to
> > > calculate allocate and use a /64 and determine the IPv6 address of VR
> nics
> > > and guest VM nics
> > > >       *   A user create an isolated network/VPC with an offering
> that is
> > > ipv6 enabled
> > > >       *   A user can manage firewall for the IPv6 address/guest nics;
> > > there'll be no port forward and LB feature though for IPv6
> > > >       *   A users can run workloads in the guest VMs that listen on
> > > publically routable ipv6 addresses
> > > >       *   Usage/billing etc continue to work, no change needed
> > > >
> > > > Network layout:
> > > >
> > > > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] ->
> > > [guest VMs/nics]
> > > > *core/ISP router needs static routes to be added (manually or
> > > automated), assumes a /48 or /56 configured for the zone
> > > >
> > > > Thoughts, feedback?
> > >
> > > Looks doable!
> > >
> > > Side-note: It would be very cool if you could use parts of this
> > > implementation to also route /48, /56, or /60 subnets to individual VMs
> > > in Shared networks.
> > >
> > > Why? This allows for running Docker containers with native IPv6 inside
> > > the VM or running a (Open)VPN server inside a VM which then assigns
> > > public IPv6 addresses to clients connected.
> > >
> > > Instead of routing the subnet to a VR we route the subnet to a single
> > > instance in a shared network.
> > >
> > > If we could then also move these subnets between Instances easily one
> > > can quickly migrate to a different instance while keeping the same IPv6
> > > subnet.
> > >
> > > Wido
> > >
> > > >
> > > > Proof-of-concept commentary: here's what I did to test the idea:
> > > >
> > > >    *   Created an isolated network and deployed a VM in my home lab
> > > > The VR running on KVM has following nics
> > > > eth0 - guest network
> > > > eth1 - link local
> > > > eth2 - public network
> > > >
> > > >    *   I setup a custom openwrt router on a RPi4 to serve as a
> toy-core
> > > router where I create a wan6 IPv6 tunnel using tunnel broker and I got
> a
> > > /48 allocated. My configuration looks like:
> > > > /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> > > > /64 - 2001:470:36:3e2::/64 (default allocated by)
> > > >
> > > > I create a LAN ipv6 (public network for CloudStack VR): at
> subnet/prefix
> > > 0:
> > > > LAN IPv6 address: 2001:470:ed36:0::1/64
> > > > Address mode: SLAAC+stateless DHCP (no dhcpv6)
> > > >    *
> > > >    *
> > > > In the isolated VR, I enabled ipv6 as:
> > > > net.ipv6.conf.all.disable_ipv6 = 0
> > > > net.ipv6.conf.all.forwarding = 1
> > > > net.ipv6.conf.all.accept_ra = 1
> > > > net.ipv6.conf.all.accept_redirects = 1
> > > > net.ipv6.conf.all.autoconf = 1
> > > >
> > > > Set up a IPv6 nameserver/dns in /etc/resolve.conf
> > > > And configured the nics:
> > > > echo iface eth0 inet6 auto >> /etc/network/interfaces
> > > > echo iface eth2 inet6 auto >> /etc/network/interfaces
> > > > /etc/init.d/networking restart
> > > > Next, restart ACS isolated network without cleanup to have it
> > > reconfigure IPv4 nics, firewall, NAT etc
> > > >
> > > >    *
> > > > Next, I created a /64 network for the isolated guest network on eth0
> of
> > > VR using radvd:
> > > >
> > > > # cat /etc/radvd.conf
> > > > interface eth0
> > > > {
> > > >      AdvSendAdvert on;
> > > >      MinRtrAdvInterval 5;
> > > >      MaxRtrAdvInterval 15;
> > > >      prefix 2001:470:ed36:1::/64
> > > >      {
> > > >          AdvOnLink on;
> > > >          AdvAutonomous on;
> > > >      };
> > > > };
> > > > systemctl restart radvd
> > > > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this
> > > ...:1::/64 network
> > > >    *   Finally I added a static route in toy core-router for the new
> /64
> > > IPv6 range in the isolated network
> > > > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev
> > > <local lan nic>
> > > >    *
> > > > ... and I enabled firewall rules to allow any traffic to pass for the
> > > new /64 network
> > > >
> > > > And voila all done! I create a domain AAAA record that points to my
> > > guest VM IPv6 address a test webserver on
> > > > http://ipv6-isolated-ntwk-demo.yadav.cloud/
> > > >
> > > > (Note: I'll get rid of the tunnel and request a new /48 block after a
> > > few days, sharing this solely for testing purposes)
> > > >
> > > >
> > > > Regards.
> > > >
> > > >
> > >
> >
>

Re: IPV6 in Isolated/VPC networks

Posted by Kristaps Cudars <kr...@gmail.com>.
Hi Wei,

Published this month’s RFC 9099 and explains in different words/perspective what has been written by Alex, Rohit and Wido.  https://www.rfc-editor.org/rfc/rfc9099.html


On 2021/08/17 09:20:21, Wei ZHOU <us...@gmail.com> wrote: 
> Hi Wido,
> 
> (cc to Rohit and Alex)
> 
> It is a good suggestion to use FRR for ipv6. The configuration is quite
> simple and the VMs can get SLAAC, routes, etc.
> 
> Privacy extension looks not the same as what you mentioned. see
> https://datatracker.ietf.org/doc/html/rfc4941
> 
> You are right. To use static routing, the admins need to configure the
> routes in the upstream router, and add some ipv6 ranges (eg /56 for VPCs
> and /64 for isolated networks) and their next-hop  (which will be
> configured in VRs) in CloudStack. CloudStack will pick up an IPv6 range and
> assign it to an isolated network or vpc. @Rohit, correct me if I'm wrong.
> 
> I have a question, it looks stateless dhcpv6 (SLAAC from router/VR,
> router/dns etc via RA messages) will be the only option for now (related to
> your pr https://github.com/apache/cloudstack/pull/3077) . Would it be good
> to provide stateful dhcpv6 (which can be implemented by dnsmasq) as an
> option in cloudstack ? The advantages are
> (1) support other ipv6 cidr sizes than /64.
> (2) we can assign a specified Ipv6 address to a vm. vm Ipv6 addresses can
> be changed
> (4) an Ipv6 addresses can be re-used by multiple vms.
> The problem is, stateful dhcpv6 does not support routers,nameservers, etc.
> we need to figure it out (probably use radvd/frr and dnsmasq both).
> 
> -Wei
> 
> 
> On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:
> 
> > Hi,
> >
> > See my inline responses:
> >
> > Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> > > Hi all,
> > >
> > > Thanks for your feedback and ideas, I've gone ahead with discussing them
> > with Alex and came up with a PoC/design which can be implemented in the
> > following phases:
> > >
> > >    *   Phase1: implement ipv6 support in isolated networks and VPC with
> > static routing
> > >    *   Phase2: discuss and implement support for dynamic routing (TBD)
> > >
> > > For Phase1 here's the high-level proposal:
> > >
> > >    *   IPv6 address management:
> > >       *   At the zone level root-admin specifies a /64 public range that
> > will be used for VRs, then they can add a /48, or /56 IPv6 range for guest
> > networks (to be used by isolated networks and VPC tiers)
> > >       *   On creation of any IPv6 enabled isolated network or VPC tier,
> > from the /48 or /56 block a /64 network is allocated/used
> > >       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the zone
> > (discuss: is privacy a concern, can privacy extensions rfc4941 of slaac be
> > explored?)
> >
> > Privacy Extensions are only a concern for client devices which roam
> > between different IPv6 networks.
> >
> > If you IPv6 address of a client keeps the same suffix (MAC based) and
> > switches network then only the prefix (/64) will change.
> >
> > This way a network like Google, Facebook, etc could track your device
> > moving from network to network if they only look at the last 64-bits of
> > the IPv6 address.
> >
> > For servers this is not a problem as you already know in which network
> > they are.
> >
> > >    *   Network offerings: root-admin can create new network offerings
> > (with VPC too) that specifies a network stack option:
> > >       *   ipv4 only (default, for backward compatibility all
> > networks/offerings post-upgrade migrate to this option)
> > >       *   ipv4-and-ipv6
> > >       *   ipv6-only (this can be phase 1.b)
> > >       *   A new routing option: static (phase1), dynamic (phase2, with
> > multiple sub-options such as ospf/bgp etc...)
> >
> > This means that the network admin will need to statically route the IPv6
> > subnet to the VR's outside IPv6 address, for example, on a JunOS router:
> >
> > set routing-options rib inet6.0 static route 2001:db8:500::/48 next-hop
> > 2001:db8:100::50
> >
> > I'm assuming that 2001:db8:100::50 is the address of the VR on the
> > outside (/64) network. In reality this will probably be a longer
> > address, but this is for just the example.
> >
> > >    *   VR changes:
> > >       *   VR gets its guest and public nics set to inet6 auto
> > >       *   For each /64 allocated to guest network and VPC tiers, radvd
> > is configured to do RA
> >
> > radvd is fine, but looking at phase 2 with dynamic routing you might
> > already want to look into FRRouting. FRR can also advertise RAs while
> > not doing any routing.
> >
> > interface ens4
> >    no ipv6 nd suppress-ra
> >    ipv6 nd prefix 2001:db8:500::/64
> >    ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
> >
> > See: http://docs.frrouting.org/en/latest/ipv6.html
> >
> > >       *   Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6
> > firewall rules (ACLs, ingress, egress) are implemented; ACLs between VPC
> > tiers are managed/implemented by ipv6 firewall on VR
> >
> > Please take a look at the existing security_group.py script which
> > implements RFC4890
> >
> > https://datatracker.ietf.org/doc/html/rfc4890
> >
> > ICMPv6 is a vital part of IPv6 and certain packets should always be
> > allowed.
> >
> > >       *   It is assumed that static routes are created on the core/main
> > router by the admin or automated using some scripts/tools; for this
> > CloudStack will announce events with details of /64 networks and VR's
> > public IPv6 address that can be consumed by a rabbitmq/message bus client
> > (for example), or a custom cron job or script as part of orchestration.
> > (this wouldn't be necessary for dynamic routing bgp with phase2)\\
> >
> > You would only need to announce the /48 or /56 allocated to the VR,
> > that's all. You don't need to inform the upstream router about the /64
> > subnets created within that larger subnet.
> >
> > >    *   Guest Networking: With SLAAC, it's easy for CloudStack to
> > calculate allocate and use a /64 and determine the IPv6 address of VR nics
> > and guest VM nics
> > >       *   A user create an isolated network/VPC with an offering that is
> > ipv6 enabled
> > >       *   A user can manage firewall for the IPv6 address/guest nics;
> > there'll be no port forward and LB feature though for IPv6
> > >       *   A users can run workloads in the guest VMs that listen on
> > publically routable ipv6 addresses
> > >       *   Usage/billing etc continue to work, no change needed
> > >
> > > Network layout:
> > >
> > > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] ->
> > [guest VMs/nics]
> > > *core/ISP router needs static routes to be added (manually or
> > automated), assumes a /48 or /56 configured for the zone
> > >
> > > Thoughts, feedback?
> >
> > Looks doable!
> >
> > Side-note: It would be very cool if you could use parts of this
> > implementation to also route /48, /56, or /60 subnets to individual VMs
> > in Shared networks.
> >
> > Why? This allows for running Docker containers with native IPv6 inside
> > the VM or running a (Open)VPN server inside a VM which then assigns
> > public IPv6 addresses to clients connected.
> >
> > Instead of routing the subnet to a VR we route the subnet to a single
> > instance in a shared network.
> >
> > If we could then also move these subnets between Instances easily one
> > can quickly migrate to a different instance while keeping the same IPv6
> > subnet.
> >
> > Wido
> >
> > >
> > > Proof-of-concept commentary: here's what I did to test the idea:
> > >
> > >    *   Created an isolated network and deployed a VM in my home lab
> > > The VR running on KVM has following nics
> > > eth0 - guest network
> > > eth1 - link local
> > > eth2 - public network
> > >
> > >    *   I setup a custom openwrt router on a RPi4 to serve as a toy-core
> > router where I create a wan6 IPv6 tunnel using tunnel broker and I got a
> > /48 allocated. My configuration looks like:
> > > /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> > > /64 - 2001:470:36:3e2::/64 (default allocated by)
> > >
> > > I create a LAN ipv6 (public network for CloudStack VR): at subnet/prefix
> > 0:
> > > LAN IPv6 address: 2001:470:ed36:0::1/64
> > > Address mode: SLAAC+stateless DHCP (no dhcpv6)
> > >    *
> > >    *
> > > In the isolated VR, I enabled ipv6 as:
> > > net.ipv6.conf.all.disable_ipv6 = 0
> > > net.ipv6.conf.all.forwarding = 1
> > > net.ipv6.conf.all.accept_ra = 1
> > > net.ipv6.conf.all.accept_redirects = 1
> > > net.ipv6.conf.all.autoconf = 1
> > >
> > > Set up a IPv6 nameserver/dns in /etc/resolve.conf
> > > And configured the nics:
> > > echo iface eth0 inet6 auto >> /etc/network/interfaces
> > > echo iface eth2 inet6 auto >> /etc/network/interfaces
> > > /etc/init.d/networking restart
> > > Next, restart ACS isolated network without cleanup to have it
> > reconfigure IPv4 nics, firewall, NAT etc
> > >
> > >    *
> > > Next, I created a /64 network for the isolated guest network on eth0 of
> > VR using radvd:
> > >
> > > # cat /etc/radvd.conf
> > > interface eth0
> > > {
> > >      AdvSendAdvert on;
> > >      MinRtrAdvInterval 5;
> > >      MaxRtrAdvInterval 15;
> > >      prefix 2001:470:ed36:1::/64
> > >      {
> > >          AdvOnLink on;
> > >          AdvAutonomous on;
> > >      };
> > > };
> > > systemctl restart radvd
> > > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this
> > ...:1::/64 network
> > >    *   Finally I added a static route in toy core-router for the new /64
> > IPv6 range in the isolated network
> > > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev
> > <local lan nic>
> > >    *
> > > ... and I enabled firewall rules to allow any traffic to pass for the
> > new /64 network
> > >
> > > And voila all done! I create a domain AAAA record that points to my
> > guest VM IPv6 address a test webserver on
> > > http://ipv6-isolated-ntwk-demo.yadav.cloud/
> > >
> > > (Note: I'll get rid of the tunnel and request a new /48 block after a
> > few days, sharing this solely for testing purposes)
> > >
> > >
> > > Regards.
> > >
> > >
> >
> 

RE: IPV6 in Isolated/VPC networks

Posted by Alex Mattioli <Al...@shapeblue.com>.
+1 to keeping the scope tight on phase 1 and then expanding functionality later on.

 


-----Original Message-----
From: Rohit Yadav <ro...@shapeblue.com> 
Sent: 17 August 2021 11:26
To: dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks

Hi Wei,

Yes, root admins should add all the /64 and bigger /56, /48 blocks which CloudStack will use to calculate and allocate /64 block from to a IPv6 enabled isolated network or VPC tier; for every /64 allocation/assignment to such a network a static route for the /64 target with the ipv6 address of the VR as the gateway should be added to the core-router/ISP gateway.

Based on what I read, the RFCs and general IPv6 addressing practice are towards deprecating stateful dhcp(v6) and promote use of /64 network for a direct-attached-network (think like a L2 network or LAN) so I think we should only support SLAAC for isolated networks and VPC tiers, and IPv6 for servers won't need privacy extensions (as Wido advised in earlier reply).

If someone has a non-standard need, they may always create and use a shared network with custom-defined IPv6 ranges. Or, at least in the phase1 of the implementation I would prefer the scope to be tight and limited that is acceptable for the community.


Regards.

________________________________
From: Alex Mattioli <Al...@shapeblue.com>
Sent: Tuesday, August 17, 2021 14:59
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: RE: IPV6 in Isolated/VPC networks

Hi Wei,

That's correct. The network operator would need to create static routes for the /64s, both for Isolated networks and VPCs.  The next-hop for those being the outside interface of the Virtual Router, which will have an IP from the "public" /64.

Cheers
Alex




 


-----Original Message-----
From: Wei ZHOU <us...@gmail.com>
Sent: 17 August 2021 10:20
To: dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks

Hi Wido,

(cc to Rohit and Alex)

It is a good suggestion to use FRR for ipv6. The configuration is quite simple and the VMs can get SLAAC, routes, etc.

Privacy extension looks not the same as what you mentioned. see
https://datatracker.ietf.org/doc/html/rfc4941

You are right. To use static routing, the admins need to configure the routes in the upstream router, and add some ipv6 ranges (eg /56 for VPCs and /64 for isolated networks) and their next-hop  (which will be configured in VRs) in CloudStack. CloudStack will pick up an IPv6 range and assign it to an isolated network or vpc. @Rohit, correct me if I'm wrong.

I have a question, it looks stateless dhcpv6 (SLAAC from router/VR, router/dns etc via RA messages) will be the only option for now (related to your pr https://github.com/apache/cloudstack/pull/3077) . Would it be good to provide stateful dhcpv6 (which can be implemented by dnsmasq) as an option in cloudstack ? The advantages are
(1) support other ipv6 cidr sizes than /64.
(2) we can assign a specified Ipv6 address to a vm. vm Ipv6 addresses can be changed
(4) an Ipv6 addresses can be re-used by multiple vms.
The problem is, stateful dhcpv6 does not support routers,nameservers, etc.
we need to figure it out (probably use radvd/frr and dnsmasq both).

-Wei


On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:

> Hi,
>
> See my inline responses:
>
> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> > Hi all,
> >
> > Thanks for your feedback and ideas, I've gone ahead with discussing 
> > them
> with Alex and came up with a PoC/design which can be implemented in 
> the following phases:
> >
> >    *   Phase1: implement ipv6 support in isolated networks and VPC with
> static routing
> >    *   Phase2: discuss and implement support for dynamic routing (TBD)
> >
> > For Phase1 here's the high-level proposal:
> >
> >    *   IPv6 address management:
> >       *   At the zone level root-admin specifies a /64 public range that
> will be used for VRs, then they can add a /48, or /56 IPv6 range for 
> guest networks (to be used by isolated networks and VPC tiers)
> >       *   On creation of any IPv6 enabled isolated network or VPC tier,
> from the /48 or /56 block a /64 network is allocated/used
> >       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the zone
> (discuss: is privacy a concern, can privacy extensions rfc4941 of 
> slaac be
> explored?)
>
> Privacy Extensions are only a concern for client devices which roam 
> between different IPv6 networks.
>
> If you IPv6 address of a client keeps the same suffix (MAC based) and 
> switches network then only the prefix (/64) will change.
>
> This way a network like Google, Facebook, etc could track your device 
> moving from network to network if they only look at the last 64-bits 
> of the IPv6 address.
>
> For servers this is not a problem as you already know in which network 
> they are.
>
> >    *   Network offerings: root-admin can create new network offerings
> (with VPC too) that specifies a network stack option:
> >       *   ipv4 only (default, for backward compatibility all
> networks/offerings post-upgrade migrate to this option)
> >       *   ipv4-and-ipv6
> >       *   ipv6-only (this can be phase 1.b)
> >       *   A new routing option: static (phase1), dynamic (phase2, with
> multiple sub-options such as ospf/bgp etc...)
>
> This means that the network admin will need to statically route the
> IPv6 subnet to the VR's outside IPv6 address, for example, on a JunOS router:
>
> set routing-options rib inet6.0 static route 2001:db8:500::/48 
> next-hop
> 2001:db8:100::50
>
> I'm assuming that 2001:db8:100::50 is the address of the VR on the 
> outside (/64) network. In reality this will probably be a longer 
> address, but this is for just the example.
>
> >    *   VR changes:
> >       *   VR gets its guest and public nics set to inet6 auto
> >       *   For each /64 allocated to guest network and VPC tiers, radvd
> is configured to do RA
>
> radvd is fine, but looking at phase 2 with dynamic routing you might 
> already want to look into FRRouting. FRR can also advertise RAs while 
> not doing any routing.
>
> interface ens4
>    no ipv6 nd suppress-ra
>    ipv6 nd prefix 2001:db8:500::/64
>    ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>
> See: http://docs.frrouting.org/en/latest/ipv6.html
>
> >       *   Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6
> firewall rules (ACLs, ingress, egress) are implemented; ACLs between 
> VPC tiers are managed/implemented by ipv6 firewall on VR
>
> Please take a look at the existing security_group.py script which 
> implements RFC4890
>
> https://datatracker.ietf.org/doc/html/rfc4890
>
> ICMPv6 is a vital part of IPv6 and certain packets should always be 
> allowed.
>
> >       *   It is assumed that static routes are created on the core/main
> router by the admin or automated using some scripts/tools; for this 
> CloudStack will announce events with details of /64 networks and VR's 
> public IPv6 address that can be consumed by a rabbitmq/message bus 
> client (for example), or a custom cron job or script as part of orchestration.
> (this wouldn't be necessary for dynamic routing bgp with phase2)\\
>
> You would only need to announce the /48 or /56 allocated to the VR, 
> that's all. You don't need to inform the upstream router about the /64 
> subnets created within that larger subnet.
>
> >    *   Guest Networking: With SLAAC, it's easy for CloudStack to
> calculate allocate and use a /64 and determine the IPv6 address of VR 
> nics and guest VM nics
> >       *   A user create an isolated network/VPC with an offering that is
> ipv6 enabled
> >       *   A user can manage firewall for the IPv6 address/guest nics;
> there'll be no port forward and LB feature though for IPv6
> >       *   A users can run workloads in the guest VMs that listen on
> publically routable ipv6 addresses
> >       *   Usage/billing etc continue to work, no change needed
> >
> > Network layout:
> >
> > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN]
> > ->
> [guest VMs/nics]
> > *core/ISP router needs static routes to be added (manually or
> automated), assumes a /48 or /56 configured for the zone
> >
> > Thoughts, feedback?
>
> Looks doable!
>
> Side-note: It would be very cool if you could use parts of this 
> implementation to also route /48, /56, or /60 subnets to individual 
> VMs in Shared networks.
>
> Why? This allows for running Docker containers with native IPv6 inside 
> the VM or running a (Open)VPN server inside a VM which then assigns 
> public IPv6 addresses to clients connected.
>
> Instead of routing the subnet to a VR we route the subnet to a single 
> instance in a shared network.
>
> If we could then also move these subnets between Instances easily one 
> can quickly migrate to a different instance while keeping the same
> IPv6 subnet.
>
> Wido
>
> >
> > Proof-of-concept commentary: here's what I did to test the idea:
> >
> >    *   Created an isolated network and deployed a VM in my home lab
> > The VR running on KVM has following nics
> > eth0 - guest network
> > eth1 - link local
> > eth2 - public network
> >
> >    *   I setup a custom openwrt router on a RPi4 to serve as a toy-core
> router where I create a wan6 IPv6 tunnel using tunnel broker and I got 
> a
> /48 allocated. My configuration looks like:
> > /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> > /64 - 2001:470:36:3e2::/64 (default allocated by)
> >
> > I create a LAN ipv6 (public network for CloudStack VR): at 
> > subnet/prefix
> 0:
> > LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
> > SLAAC+stateless DHCP (no dhcpv6)
> >    *
> >    *
> > In the isolated VR, I enabled ipv6 as:
> > net.ipv6.conf.all.disable_ipv6 = 0
> > net.ipv6.conf.all.forwarding = 1
> > net.ipv6.conf.all.accept_ra = 1
> > net.ipv6.conf.all.accept_redirects = 1 net.ipv6.conf.all.autoconf =
> > 1
> >
> > Set up a IPv6 nameserver/dns in /etc/resolve.conf And configured the
> > nics:
> > echo iface eth0 inet6 auto >> /etc/network/interfaces echo iface
> > eth2 inet6 auto >> /etc/network/interfaces /etc/init.d/networking 
> > restart Next, restart ACS isolated network without cleanup to have 
> > it
> reconfigure IPv4 nics, firewall, NAT etc
> >
> >    *
> > Next, I created a /64 network for the isolated guest network on eth0 
> > of
> VR using radvd:
> >
> > # cat /etc/radvd.conf
> > interface eth0
> > {
> >      AdvSendAdvert on;
> >      MinRtrAdvInterval 5;
> >      MaxRtrAdvInterval 15;
> >      prefix 2001:470:ed36:1::/64
> >      {
> >          AdvOnLink on;
> >          AdvAutonomous on;
> >      };
> > };
> > systemctl restart radvd
> > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this
> ...:1::/64 network
> >    *   Finally I added a static route in toy core-router for the new /64
> IPv6 range in the isolated network
> > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev
> <local lan nic>
> >    *
> > ... and I enabled firewall rules to allow any traffic to pass for 
> > the
> new /64 network
> >
> > And voila all done! I create a domain AAAA record that points to my
> guest VM IPv6 address a test webserver on
> > http://ipv6-isolated-ntwk-demo.yadav.cloud/
> >
> > (Note: I'll get rid of the tunnel and request a new /48 block after 
> > a
> few days, sharing this solely for testing purposes)
> >
> >
> > Regards.
> >
> >
>


Re: IPV6 in Isolated/VPC networks

Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi Wei,

Yes, root admins should add all the /64 and bigger /56, /48 blocks which CloudStack will use to calculate and allocate /64 block from to a IPv6 enabled isolated network or VPC tier; for every /64 allocation/assignment to such a network a static route for the /64 target with the ipv6 address of the VR as the gateway should be added to the core-router/ISP gateway.

Based on what I read, the RFCs and general IPv6 addressing practice are towards deprecating stateful dhcp(v6) and promote use of /64 network for a direct-attached-network (think like a L2 network or LAN) so I think we should only support SLAAC for isolated networks and VPC tiers, and IPv6 for servers won't need privacy extensions (as Wido advised in earlier reply).

If someone has a non-standard need, they may always create and use a shared network with custom-defined IPv6 ranges. Or, at least in the phase1 of the implementation I would prefer the scope to be tight and limited that is acceptable for the community.


Regards.

________________________________
From: Alex Mattioli <Al...@shapeblue.com>
Sent: Tuesday, August 17, 2021 14:59
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: RE: IPV6 in Isolated/VPC networks

Hi Wei,

That's correct. The network operator would need to create static routes for the /64s, both for Isolated networks and VPCs.  The next-hop for those being the outside interface of the Virtual Router, which will have an IP from the "public" /64.

Cheers
Alex




 


-----Original Message-----
From: Wei ZHOU <us...@gmail.com>
Sent: 17 August 2021 10:20
To: dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks

Hi Wido,

(cc to Rohit and Alex)

It is a good suggestion to use FRR for ipv6. The configuration is quite simple and the VMs can get SLAAC, routes, etc.

Privacy extension looks not the same as what you mentioned. see
https://datatracker.ietf.org/doc/html/rfc4941

You are right. To use static routing, the admins need to configure the routes in the upstream router, and add some ipv6 ranges (eg /56 for VPCs and /64 for isolated networks) and their next-hop  (which will be configured in VRs) in CloudStack. CloudStack will pick up an IPv6 range and assign it to an isolated network or vpc. @Rohit, correct me if I'm wrong.

I have a question, it looks stateless dhcpv6 (SLAAC from router/VR, router/dns etc via RA messages) will be the only option for now (related to your pr https://github.com/apache/cloudstack/pull/3077) . Would it be good to provide stateful dhcpv6 (which can be implemented by dnsmasq) as an option in cloudstack ? The advantages are
(1) support other ipv6 cidr sizes than /64.
(2) we can assign a specified Ipv6 address to a vm. vm Ipv6 addresses can be changed
(4) an Ipv6 addresses can be re-used by multiple vms.
The problem is, stateful dhcpv6 does not support routers,nameservers, etc.
we need to figure it out (probably use radvd/frr and dnsmasq both).

-Wei


On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:

> Hi,
>
> See my inline responses:
>
> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> > Hi all,
> >
> > Thanks for your feedback and ideas, I've gone ahead with discussing
> > them
> with Alex and came up with a PoC/design which can be implemented in
> the following phases:
> >
> >    *   Phase1: implement ipv6 support in isolated networks and VPC with
> static routing
> >    *   Phase2: discuss and implement support for dynamic routing (TBD)
> >
> > For Phase1 here's the high-level proposal:
> >
> >    *   IPv6 address management:
> >       *   At the zone level root-admin specifies a /64 public range that
> will be used for VRs, then they can add a /48, or /56 IPv6 range for
> guest networks (to be used by isolated networks and VPC tiers)
> >       *   On creation of any IPv6 enabled isolated network or VPC tier,
> from the /48 or /56 block a /64 network is allocated/used
> >       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the zone
> (discuss: is privacy a concern, can privacy extensions rfc4941 of
> slaac be
> explored?)
>
> Privacy Extensions are only a concern for client devices which roam
> between different IPv6 networks.
>
> If you IPv6 address of a client keeps the same suffix (MAC based) and
> switches network then only the prefix (/64) will change.
>
> This way a network like Google, Facebook, etc could track your device
> moving from network to network if they only look at the last 64-bits
> of the IPv6 address.
>
> For servers this is not a problem as you already know in which network
> they are.
>
> >    *   Network offerings: root-admin can create new network offerings
> (with VPC too) that specifies a network stack option:
> >       *   ipv4 only (default, for backward compatibility all
> networks/offerings post-upgrade migrate to this option)
> >       *   ipv4-and-ipv6
> >       *   ipv6-only (this can be phase 1.b)
> >       *   A new routing option: static (phase1), dynamic (phase2, with
> multiple sub-options such as ospf/bgp etc...)
>
> This means that the network admin will need to statically route the
> IPv6 subnet to the VR's outside IPv6 address, for example, on a JunOS router:
>
> set routing-options rib inet6.0 static route 2001:db8:500::/48
> next-hop
> 2001:db8:100::50
>
> I'm assuming that 2001:db8:100::50 is the address of the VR on the
> outside (/64) network. In reality this will probably be a longer
> address, but this is for just the example.
>
> >    *   VR changes:
> >       *   VR gets its guest and public nics set to inet6 auto
> >       *   For each /64 allocated to guest network and VPC tiers, radvd
> is configured to do RA
>
> radvd is fine, but looking at phase 2 with dynamic routing you might
> already want to look into FRRouting. FRR can also advertise RAs while
> not doing any routing.
>
> interface ens4
>    no ipv6 nd suppress-ra
>    ipv6 nd prefix 2001:db8:500::/64
>    ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>
> See: http://docs.frrouting.org/en/latest/ipv6.html
>
> >       *   Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6
> firewall rules (ACLs, ingress, egress) are implemented; ACLs between
> VPC tiers are managed/implemented by ipv6 firewall on VR
>
> Please take a look at the existing security_group.py script which
> implements RFC4890
>
> https://datatracker.ietf.org/doc/html/rfc4890
>
> ICMPv6 is a vital part of IPv6 and certain packets should always be
> allowed.
>
> >       *   It is assumed that static routes are created on the core/main
> router by the admin or automated using some scripts/tools; for this
> CloudStack will announce events with details of /64 networks and VR's
> public IPv6 address that can be consumed by a rabbitmq/message bus
> client (for example), or a custom cron job or script as part of orchestration.
> (this wouldn't be necessary for dynamic routing bgp with phase2)\\
>
> You would only need to announce the /48 or /56 allocated to the VR,
> that's all. You don't need to inform the upstream router about the /64
> subnets created within that larger subnet.
>
> >    *   Guest Networking: With SLAAC, it's easy for CloudStack to
> calculate allocate and use a /64 and determine the IPv6 address of VR
> nics and guest VM nics
> >       *   A user create an isolated network/VPC with an offering that is
> ipv6 enabled
> >       *   A user can manage firewall for the IPv6 address/guest nics;
> there'll be no port forward and LB feature though for IPv6
> >       *   A users can run workloads in the guest VMs that listen on
> publically routable ipv6 addresses
> >       *   Usage/billing etc continue to work, no change needed
> >
> > Network layout:
> >
> > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN]
> > ->
> [guest VMs/nics]
> > *core/ISP router needs static routes to be added (manually or
> automated), assumes a /48 or /56 configured for the zone
> >
> > Thoughts, feedback?
>
> Looks doable!
>
> Side-note: It would be very cool if you could use parts of this
> implementation to also route /48, /56, or /60 subnets to individual
> VMs in Shared networks.
>
> Why? This allows for running Docker containers with native IPv6 inside
> the VM or running a (Open)VPN server inside a VM which then assigns
> public IPv6 addresses to clients connected.
>
> Instead of routing the subnet to a VR we route the subnet to a single
> instance in a shared network.
>
> If we could then also move these subnets between Instances easily one
> can quickly migrate to a different instance while keeping the same
> IPv6 subnet.
>
> Wido
>
> >
> > Proof-of-concept commentary: here's what I did to test the idea:
> >
> >    *   Created an isolated network and deployed a VM in my home lab
> > The VR running on KVM has following nics
> > eth0 - guest network
> > eth1 - link local
> > eth2 - public network
> >
> >    *   I setup a custom openwrt router on a RPi4 to serve as a toy-core
> router where I create a wan6 IPv6 tunnel using tunnel broker and I got
> a
> /48 allocated. My configuration looks like:
> > /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> > /64 - 2001:470:36:3e2::/64 (default allocated by)
> >
> > I create a LAN ipv6 (public network for CloudStack VR): at
> > subnet/prefix
> 0:
> > LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode:
> > SLAAC+stateless DHCP (no dhcpv6)
> >    *
> >    *
> > In the isolated VR, I enabled ipv6 as:
> > net.ipv6.conf.all.disable_ipv6 = 0
> > net.ipv6.conf.all.forwarding = 1
> > net.ipv6.conf.all.accept_ra = 1
> > net.ipv6.conf.all.accept_redirects = 1 net.ipv6.conf.all.autoconf =
> > 1
> >
> > Set up a IPv6 nameserver/dns in /etc/resolve.conf And configured the
> > nics:
> > echo iface eth0 inet6 auto >> /etc/network/interfaces echo iface
> > eth2 inet6 auto >> /etc/network/interfaces /etc/init.d/networking
> > restart Next, restart ACS isolated network without cleanup to have
> > it
> reconfigure IPv4 nics, firewall, NAT etc
> >
> >    *
> > Next, I created a /64 network for the isolated guest network on eth0
> > of
> VR using radvd:
> >
> > # cat /etc/radvd.conf
> > interface eth0
> > {
> >      AdvSendAdvert on;
> >      MinRtrAdvInterval 5;
> >      MaxRtrAdvInterval 15;
> >      prefix 2001:470:ed36:1::/64
> >      {
> >          AdvOnLink on;
> >          AdvAutonomous on;
> >      };
> > };
> > systemctl restart radvd
> > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this
> ...:1::/64 network
> >    *   Finally I added a static route in toy core-router for the new /64
> IPv6 range in the isolated network
> > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev
> <local lan nic>
> >    *
> > ... and I enabled firewall rules to allow any traffic to pass for
> > the
> new /64 network
> >
> > And voila all done! I create a domain AAAA record that points to my
> guest VM IPv6 address a test webserver on
> > http://ipv6-isolated-ntwk-demo.yadav.cloud/
> >
> > (Note: I'll get rid of the tunnel and request a new /48 block after
> > a
> few days, sharing this solely for testing purposes)
> >
> >
> > Regards.
> >
> >
>

RE: IPV6 in Isolated/VPC networks

Posted by Alex Mattioli <Al...@shapeblue.com>.
Hi Wei,

That's correct. The network operator would need to create static routes for the /64s, both for Isolated networks and VPCs.  The next-hop for those being the outside interface of the Virtual Router, which will have an IP from the "public" /64.

Cheers
Alex

 


-----Original Message-----
From: Wei ZHOU <us...@gmail.com> 
Sent: 17 August 2021 10:20
To: dev@cloudstack.apache.org
Subject: Re: IPV6 in Isolated/VPC networks

Hi Wido,

(cc to Rohit and Alex)

It is a good suggestion to use FRR for ipv6. The configuration is quite simple and the VMs can get SLAAC, routes, etc.

Privacy extension looks not the same as what you mentioned. see
https://datatracker.ietf.org/doc/html/rfc4941

You are right. To use static routing, the admins need to configure the routes in the upstream router, and add some ipv6 ranges (eg /56 for VPCs and /64 for isolated networks) and their next-hop  (which will be configured in VRs) in CloudStack. CloudStack will pick up an IPv6 range and assign it to an isolated network or vpc. @Rohit, correct me if I'm wrong.

I have a question, it looks stateless dhcpv6 (SLAAC from router/VR, router/dns etc via RA messages) will be the only option for now (related to your pr https://github.com/apache/cloudstack/pull/3077) . Would it be good to provide stateful dhcpv6 (which can be implemented by dnsmasq) as an option in cloudstack ? The advantages are
(1) support other ipv6 cidr sizes than /64.
(2) we can assign a specified Ipv6 address to a vm. vm Ipv6 addresses can be changed
(4) an Ipv6 addresses can be re-used by multiple vms.
The problem is, stateful dhcpv6 does not support routers,nameservers, etc.
we need to figure it out (probably use radvd/frr and dnsmasq both).

-Wei


On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:

> Hi,
>
> See my inline responses:
>
> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> > Hi all,
> >
> > Thanks for your feedback and ideas, I've gone ahead with discussing 
> > them
> with Alex and came up with a PoC/design which can be implemented in 
> the following phases:
> >
> >    *   Phase1: implement ipv6 support in isolated networks and VPC with
> static routing
> >    *   Phase2: discuss and implement support for dynamic routing (TBD)
> >
> > For Phase1 here's the high-level proposal:
> >
> >    *   IPv6 address management:
> >       *   At the zone level root-admin specifies a /64 public range that
> will be used for VRs, then they can add a /48, or /56 IPv6 range for 
> guest networks (to be used by isolated networks and VPC tiers)
> >       *   On creation of any IPv6 enabled isolated network or VPC tier,
> from the /48 or /56 block a /64 network is allocated/used
> >       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the zone
> (discuss: is privacy a concern, can privacy extensions rfc4941 of 
> slaac be
> explored?)
>
> Privacy Extensions are only a concern for client devices which roam 
> between different IPv6 networks.
>
> If you IPv6 address of a client keeps the same suffix (MAC based) and 
> switches network then only the prefix (/64) will change.
>
> This way a network like Google, Facebook, etc could track your device 
> moving from network to network if they only look at the last 64-bits 
> of the IPv6 address.
>
> For servers this is not a problem as you already know in which network 
> they are.
>
> >    *   Network offerings: root-admin can create new network offerings
> (with VPC too) that specifies a network stack option:
> >       *   ipv4 only (default, for backward compatibility all
> networks/offerings post-upgrade migrate to this option)
> >       *   ipv4-and-ipv6
> >       *   ipv6-only (this can be phase 1.b)
> >       *   A new routing option: static (phase1), dynamic (phase2, with
> multiple sub-options such as ospf/bgp etc...)
>
> This means that the network admin will need to statically route the 
> IPv6 subnet to the VR's outside IPv6 address, for example, on a JunOS router:
>
> set routing-options rib inet6.0 static route 2001:db8:500::/48 
> next-hop
> 2001:db8:100::50
>
> I'm assuming that 2001:db8:100::50 is the address of the VR on the 
> outside (/64) network. In reality this will probably be a longer 
> address, but this is for just the example.
>
> >    *   VR changes:
> >       *   VR gets its guest and public nics set to inet6 auto
> >       *   For each /64 allocated to guest network and VPC tiers, radvd
> is configured to do RA
>
> radvd is fine, but looking at phase 2 with dynamic routing you might 
> already want to look into FRRouting. FRR can also advertise RAs while 
> not doing any routing.
>
> interface ens4
>    no ipv6 nd suppress-ra
>    ipv6 nd prefix 2001:db8:500::/64
>    ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>
> See: http://docs.frrouting.org/en/latest/ipv6.html
>
> >       *   Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6
> firewall rules (ACLs, ingress, egress) are implemented; ACLs between 
> VPC tiers are managed/implemented by ipv6 firewall on VR
>
> Please take a look at the existing security_group.py script which 
> implements RFC4890
>
> https://datatracker.ietf.org/doc/html/rfc4890
>
> ICMPv6 is a vital part of IPv6 and certain packets should always be 
> allowed.
>
> >       *   It is assumed that static routes are created on the core/main
> router by the admin or automated using some scripts/tools; for this 
> CloudStack will announce events with details of /64 networks and VR's 
> public IPv6 address that can be consumed by a rabbitmq/message bus 
> client (for example), or a custom cron job or script as part of orchestration.
> (this wouldn't be necessary for dynamic routing bgp with phase2)\\
>
> You would only need to announce the /48 or /56 allocated to the VR, 
> that's all. You don't need to inform the upstream router about the /64 
> subnets created within that larger subnet.
>
> >    *   Guest Networking: With SLAAC, it's easy for CloudStack to
> calculate allocate and use a /64 and determine the IPv6 address of VR 
> nics and guest VM nics
> >       *   A user create an isolated network/VPC with an offering that is
> ipv6 enabled
> >       *   A user can manage firewall for the IPv6 address/guest nics;
> there'll be no port forward and LB feature though for IPv6
> >       *   A users can run workloads in the guest VMs that listen on
> publically routable ipv6 addresses
> >       *   Usage/billing etc continue to work, no change needed
> >
> > Network layout:
> >
> > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] 
> > ->
> [guest VMs/nics]
> > *core/ISP router needs static routes to be added (manually or
> automated), assumes a /48 or /56 configured for the zone
> >
> > Thoughts, feedback?
>
> Looks doable!
>
> Side-note: It would be very cool if you could use parts of this 
> implementation to also route /48, /56, or /60 subnets to individual 
> VMs in Shared networks.
>
> Why? This allows for running Docker containers with native IPv6 inside 
> the VM or running a (Open)VPN server inside a VM which then assigns 
> public IPv6 addresses to clients connected.
>
> Instead of routing the subnet to a VR we route the subnet to a single 
> instance in a shared network.
>
> If we could then also move these subnets between Instances easily one 
> can quickly migrate to a different instance while keeping the same 
> IPv6 subnet.
>
> Wido
>
> >
> > Proof-of-concept commentary: here's what I did to test the idea:
> >
> >    *   Created an isolated network and deployed a VM in my home lab
> > The VR running on KVM has following nics
> > eth0 - guest network
> > eth1 - link local
> > eth2 - public network
> >
> >    *   I setup a custom openwrt router on a RPi4 to serve as a toy-core
> router where I create a wan6 IPv6 tunnel using tunnel broker and I got 
> a
> /48 allocated. My configuration looks like:
> > /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> > /64 - 2001:470:36:3e2::/64 (default allocated by)
> >
> > I create a LAN ipv6 (public network for CloudStack VR): at 
> > subnet/prefix
> 0:
> > LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode: 
> > SLAAC+stateless DHCP (no dhcpv6)
> >    *
> >    *
> > In the isolated VR, I enabled ipv6 as:
> > net.ipv6.conf.all.disable_ipv6 = 0
> > net.ipv6.conf.all.forwarding = 1
> > net.ipv6.conf.all.accept_ra = 1
> > net.ipv6.conf.all.accept_redirects = 1 net.ipv6.conf.all.autoconf = 
> > 1
> >
> > Set up a IPv6 nameserver/dns in /etc/resolve.conf And configured the 
> > nics:
> > echo iface eth0 inet6 auto >> /etc/network/interfaces echo iface 
> > eth2 inet6 auto >> /etc/network/interfaces /etc/init.d/networking 
> > restart Next, restart ACS isolated network without cleanup to have 
> > it
> reconfigure IPv4 nics, firewall, NAT etc
> >
> >    *
> > Next, I created a /64 network for the isolated guest network on eth0 
> > of
> VR using radvd:
> >
> > # cat /etc/radvd.conf
> > interface eth0
> > {
> >      AdvSendAdvert on;
> >      MinRtrAdvInterval 5;
> >      MaxRtrAdvInterval 15;
> >      prefix 2001:470:ed36:1::/64
> >      {
> >          AdvOnLink on;
> >          AdvAutonomous on;
> >      };
> > };
> > systemctl restart radvd
> > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this
> ...:1::/64 network
> >    *   Finally I added a static route in toy core-router for the new /64
> IPv6 range in the isolated network
> > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev
> <local lan nic>
> >    *
> > ... and I enabled firewall rules to allow any traffic to pass for 
> > the
> new /64 network
> >
> > And voila all done! I create a domain AAAA record that points to my
> guest VM IPv6 address a test webserver on
> > http://ipv6-isolated-ntwk-demo.yadav.cloud/
> >
> > (Note: I'll get rid of the tunnel and request a new /48 block after 
> > a
> few days, sharing this solely for testing purposes)
> >
> >
> > Regards.
> >
> >
>

Re: IPV6 in Isolated/VPC networks

Posted by Wei ZHOU <us...@gmail.com>.
Hi Wido,

(cc to Rohit and Alex)

It is a good suggestion to use FRR for ipv6. The configuration is quite
simple and the VMs can get SLAAC, routes, etc.

Privacy extension looks not the same as what you mentioned. see
https://datatracker.ietf.org/doc/html/rfc4941

You are right. To use static routing, the admins need to configure the
routes in the upstream router, and add some ipv6 ranges (eg /56 for VPCs
and /64 for isolated networks) and their next-hop  (which will be
configured in VRs) in CloudStack. CloudStack will pick up an IPv6 range and
assign it to an isolated network or vpc. @Rohit, correct me if I'm wrong.

I have a question, it looks stateless dhcpv6 (SLAAC from router/VR,
router/dns etc via RA messages) will be the only option for now (related to
your pr https://github.com/apache/cloudstack/pull/3077) . Would it be good
to provide stateful dhcpv6 (which can be implemented by dnsmasq) as an
option in cloudstack ? The advantages are
(1) support other ipv6 cidr sizes than /64.
(2) we can assign a specified Ipv6 address to a vm. vm Ipv6 addresses can
be changed
(4) an Ipv6 addresses can be re-used by multiple vms.
The problem is, stateful dhcpv6 does not support routers,nameservers, etc.
we need to figure it out (probably use radvd/frr and dnsmasq both).

-Wei


On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <wi...@widodh.nl> wrote:

> Hi,
>
> See my inline responses:
>
> Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> > Hi all,
> >
> > Thanks for your feedback and ideas, I've gone ahead with discussing them
> with Alex and came up with a PoC/design which can be implemented in the
> following phases:
> >
> >    *   Phase1: implement ipv6 support in isolated networks and VPC with
> static routing
> >    *   Phase2: discuss and implement support for dynamic routing (TBD)
> >
> > For Phase1 here's the high-level proposal:
> >
> >    *   IPv6 address management:
> >       *   At the zone level root-admin specifies a /64 public range that
> will be used for VRs, then they can add a /48, or /56 IPv6 range for guest
> networks (to be used by isolated networks and VPC tiers)
> >       *   On creation of any IPv6 enabled isolated network or VPC tier,
> from the /48 or /56 block a /64 network is allocated/used
> >       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the zone
> (discuss: is privacy a concern, can privacy extensions rfc4941 of slaac be
> explored?)
>
> Privacy Extensions are only a concern for client devices which roam
> between different IPv6 networks.
>
> If you IPv6 address of a client keeps the same suffix (MAC based) and
> switches network then only the prefix (/64) will change.
>
> This way a network like Google, Facebook, etc could track your device
> moving from network to network if they only look at the last 64-bits of
> the IPv6 address.
>
> For servers this is not a problem as you already know in which network
> they are.
>
> >    *   Network offerings: root-admin can create new network offerings
> (with VPC too) that specifies a network stack option:
> >       *   ipv4 only (default, for backward compatibility all
> networks/offerings post-upgrade migrate to this option)
> >       *   ipv4-and-ipv6
> >       *   ipv6-only (this can be phase 1.b)
> >       *   A new routing option: static (phase1), dynamic (phase2, with
> multiple sub-options such as ospf/bgp etc...)
>
> This means that the network admin will need to statically route the IPv6
> subnet to the VR's outside IPv6 address, for example, on a JunOS router:
>
> set routing-options rib inet6.0 static route 2001:db8:500::/48 next-hop
> 2001:db8:100::50
>
> I'm assuming that 2001:db8:100::50 is the address of the VR on the
> outside (/64) network. In reality this will probably be a longer
> address, but this is for just the example.
>
> >    *   VR changes:
> >       *   VR gets its guest and public nics set to inet6 auto
> >       *   For each /64 allocated to guest network and VPC tiers, radvd
> is configured to do RA
>
> radvd is fine, but looking at phase 2 with dynamic routing you might
> already want to look into FRRouting. FRR can also advertise RAs while
> not doing any routing.
>
> interface ens4
>    no ipv6 nd suppress-ra
>    ipv6 nd prefix 2001:db8:500::/64
>    ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53
>
> See: http://docs.frrouting.org/en/latest/ipv6.html
>
> >       *   Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6
> firewall rules (ACLs, ingress, egress) are implemented; ACLs between VPC
> tiers are managed/implemented by ipv6 firewall on VR
>
> Please take a look at the existing security_group.py script which
> implements RFC4890
>
> https://datatracker.ietf.org/doc/html/rfc4890
>
> ICMPv6 is a vital part of IPv6 and certain packets should always be
> allowed.
>
> >       *   It is assumed that static routes are created on the core/main
> router by the admin or automated using some scripts/tools; for this
> CloudStack will announce events with details of /64 networks and VR's
> public IPv6 address that can be consumed by a rabbitmq/message bus client
> (for example), or a custom cron job or script as part of orchestration.
> (this wouldn't be necessary for dynamic routing bgp with phase2)\\
>
> You would only need to announce the /48 or /56 allocated to the VR,
> that's all. You don't need to inform the upstream router about the /64
> subnets created within that larger subnet.
>
> >    *   Guest Networking: With SLAAC, it's easy for CloudStack to
> calculate allocate and use a /64 and determine the IPv6 address of VR nics
> and guest VM nics
> >       *   A user create an isolated network/VPC with an offering that is
> ipv6 enabled
> >       *   A user can manage firewall for the IPv6 address/guest nics;
> there'll be no port forward and LB feature though for IPv6
> >       *   A users can run workloads in the guest VMs that listen on
> publically routable ipv6 addresses
> >       *   Usage/billing etc continue to work, no change needed
> >
> > Network layout:
> >
> > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] ->
> [guest VMs/nics]
> > *core/ISP router needs static routes to be added (manually or
> automated), assumes a /48 or /56 configured for the zone
> >
> > Thoughts, feedback?
>
> Looks doable!
>
> Side-note: It would be very cool if you could use parts of this
> implementation to also route /48, /56, or /60 subnets to individual VMs
> in Shared networks.
>
> Why? This allows for running Docker containers with native IPv6 inside
> the VM or running a (Open)VPN server inside a VM which then assigns
> public IPv6 addresses to clients connected.
>
> Instead of routing the subnet to a VR we route the subnet to a single
> instance in a shared network.
>
> If we could then also move these subnets between Instances easily one
> can quickly migrate to a different instance while keeping the same IPv6
> subnet.
>
> Wido
>
> >
> > Proof-of-concept commentary: here's what I did to test the idea:
> >
> >    *   Created an isolated network and deployed a VM in my home lab
> > The VR running on KVM has following nics
> > eth0 - guest network
> > eth1 - link local
> > eth2 - public network
> >
> >    *   I setup a custom openwrt router on a RPi4 to serve as a toy-core
> router where I create a wan6 IPv6 tunnel using tunnel broker and I got a
> /48 allocated. My configuration looks like:
> > /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> > /64 - 2001:470:36:3e2::/64 (default allocated by)
> >
> > I create a LAN ipv6 (public network for CloudStack VR): at subnet/prefix
> 0:
> > LAN IPv6 address: 2001:470:ed36:0::1/64
> > Address mode: SLAAC+stateless DHCP (no dhcpv6)
> >    *
> >    *
> > In the isolated VR, I enabled ipv6 as:
> > net.ipv6.conf.all.disable_ipv6 = 0
> > net.ipv6.conf.all.forwarding = 1
> > net.ipv6.conf.all.accept_ra = 1
> > net.ipv6.conf.all.accept_redirects = 1
> > net.ipv6.conf.all.autoconf = 1
> >
> > Set up a IPv6 nameserver/dns in /etc/resolve.conf
> > And configured the nics:
> > echo iface eth0 inet6 auto >> /etc/network/interfaces
> > echo iface eth2 inet6 auto >> /etc/network/interfaces
> > /etc/init.d/networking restart
> > Next, restart ACS isolated network without cleanup to have it
> reconfigure IPv4 nics, firewall, NAT etc
> >
> >    *
> > Next, I created a /64 network for the isolated guest network on eth0 of
> VR using radvd:
> >
> > # cat /etc/radvd.conf
> > interface eth0
> > {
> >      AdvSendAdvert on;
> >      MinRtrAdvInterval 5;
> >      MaxRtrAdvInterval 15;
> >      prefix 2001:470:ed36:1::/64
> >      {
> >          AdvOnLink on;
> >          AdvAutonomous on;
> >      };
> > };
> > systemctl restart radvd
> > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this
> ...:1::/64 network
> >    *   Finally I added a static route in toy core-router for the new /64
> IPv6 range in the isolated network
> > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev
> <local lan nic>
> >    *
> > ... and I enabled firewall rules to allow any traffic to pass for the
> new /64 network
> >
> > And voila all done! I create a domain AAAA record that points to my
> guest VM IPv6 address a test webserver on
> > http://ipv6-isolated-ntwk-demo.yadav.cloud/
> >
> > (Note: I'll get rid of the tunnel and request a new /48 block after a
> few days, sharing this solely for testing purposes)
> >
> >
> > Regards.
> >
> >
>

Re: IPV6 in Isolated/VPC networks

Posted by Rohit Yadav <ro...@shapeblue.com>.
Thanks Hean, Kristaps, Wido for your feedback.

I think we've some quorum and consensus on how we should proceed with IPv6 support with static routing (phase1). Based on my proof-of-concept and discussions, I believe we may target this feature as early as 4.17 and I welcome offer by Kristaps and others who may want to be involved in testing the feature as and when we'll develop it.

As the next step I'll write a short design doc on cwiki including some of the new ideas/suggestions and share it on this thread for another iteration.


Regards.

________________________________
From: Wido den Hollander <wi...@widodh.nl>
Sent: Friday, August 13, 2021 15:48
To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
Subject: Re: IPV6 in Isolated/VPC networks

Hi,

See my inline responses:

Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> Hi all,
>
> Thanks for your feedback and ideas, I've gone ahead with discussing them with Alex and came up with a PoC/design which can be implemented in the following phases:
>
>    *   Phase1: implement ipv6 support in isolated networks and VPC with static routing
>    *   Phase2: discuss and implement support for dynamic routing (TBD)
>
> For Phase1 here's the high-level proposal:
>
>    *   IPv6 address management:
>       *   At the zone level root-admin specifies a /64 public range that will be used for VRs, then they can add a /48, or /56 IPv6 range for guest networks (to be used by isolated networks and VPC tiers)
>       *   On creation of any IPv6 enabled isolated network or VPC tier, from the /48 or /56 block a /64 network is allocated/used
>       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the zone (discuss: is privacy a concern, can privacy extensions rfc4941 of slaac be explored?)

Privacy Extensions are only a concern for client devices which roam
between different IPv6 networks.

If you IPv6 address of a client keeps the same suffix (MAC based) and
switches network then only the prefix (/64) will change.

This way a network like Google, Facebook, etc could track your device
moving from network to network if they only look at the last 64-bits of
the IPv6 address.

For servers this is not a problem as you already know in which network
they are.

>    *   Network offerings: root-admin can create new network offerings (with VPC too) that specifies a network stack option:
>       *   ipv4 only (default, for backward compatibility all networks/offerings post-upgrade migrate to this option)
>       *   ipv4-and-ipv6
>       *   ipv6-only (this can be phase 1.b)
>       *   A new routing option: static (phase1), dynamic (phase2, with multiple sub-options such as ospf/bgp etc...)

This means that the network admin will need to statically route the IPv6
subnet to the VR's outside IPv6 address, for example, on a JunOS router:

set routing-options rib inet6.0 static route 2001:db8:500::/48 next-hop
2001:db8:100::50

I'm assuming that 2001:db8:100::50 is the address of the VR on the
outside (/64) network. In reality this will probably be a longer
address, but this is for just the example.

>    *   VR changes:
>       *   VR gets its guest and public nics set to inet6 auto
>       *   For each /64 allocated to guest network and VPC tiers, radvd is configured to do RA

radvd is fine, but looking at phase 2 with dynamic routing you might
already want to look into FRRouting. FRR can also advertise RAs while
not doing any routing.

interface ens4
   no ipv6 nd suppress-ra
   ipv6 nd prefix 2001:db8:500::/64
   ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53

See: http://docs.frrouting.org/en/latest/ipv6.html

>       *   Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6 firewall rules (ACLs, ingress, egress) are implemented; ACLs between VPC tiers are managed/implemented by ipv6 firewall on VR

Please take a look at the existing security_group.py script which
implements RFC4890

https://datatracker.ietf.org/doc/html/rfc4890

ICMPv6 is a vital part of IPv6 and certain packets should always be allowed.

>       *   It is assumed that static routes are created on the core/main router by the admin or automated using some scripts/tools; for this CloudStack will announce events with details of /64 networks and VR's public IPv6 address that can be consumed by a rabbitmq/message bus client (for example), or a custom cron job or script as part of orchestration. (this wouldn't be necessary for dynamic routing bgp with phase2)\\

You would only need to announce the /48 or /56 allocated to the VR,
that's all. You don't need to inform the upstream router about the /64
subnets created within that larger subnet.

>    *   Guest Networking: With SLAAC, it's easy for CloudStack to calculate allocate and use a /64 and determine the IPv6 address of VR nics and guest VM nics
>       *   A user create an isolated network/VPC with an offering that is ipv6 enabled
>       *   A user can manage firewall for the IPv6 address/guest nics; there'll be no port forward and LB feature though for IPv6
>       *   A users can run workloads in the guest VMs that listen on publically routable ipv6 addresses
>       *   Usage/billing etc continue to work, no change needed
>
> Network layout:
>
> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] -> [guest VMs/nics]
> *core/ISP router needs static routes to be added (manually or automated), assumes a /48 or /56 configured for the zone
>
> Thoughts, feedback?

Looks doable!

Side-note: It would be very cool if you could use parts of this
implementation to also route /48, /56, or /60 subnets to individual VMs
in Shared networks.

Why? This allows for running Docker containers with native IPv6 inside
the VM or running a (Open)VPN server inside a VM which then assigns
public IPv6 addresses to clients connected.

Instead of routing the subnet to a VR we route the subnet to a single
instance in a shared network.

If we could then also move these subnets between Instances easily one
can quickly migrate to a different instance while keeping the same IPv6
subnet.

Wido

>
> Proof-of-concept commentary: here's what I did to test the idea:
>
>    *   Created an isolated network and deployed a VM in my home lab
> The VR running on KVM has following nics
> eth0 - guest network
> eth1 - link local
> eth2 - public network
>
>    *   I setup a custom openwrt router on a RPi4 to serve as a toy-core router where I create a wan6 IPv6 tunnel using tunnel broker and I got a /48 allocated. My configuration looks like:
> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> /64 - 2001:470:36:3e2::/64 (default allocated by)
>
> I create a LAN ipv6 (public network for CloudStack VR): at subnet/prefix 0:
> LAN IPv6 address: 2001:470:ed36:0::1/64
> Address mode: SLAAC+stateless DHCP (no dhcpv6)
>    *
>    *
> In the isolated VR, I enabled ipv6 as:
> net.ipv6.conf.all.disable_ipv6 = 0
> net.ipv6.conf.all.forwarding = 1
> net.ipv6.conf.all.accept_ra = 1
> net.ipv6.conf.all.accept_redirects = 1
> net.ipv6.conf.all.autoconf = 1
>
> Set up a IPv6 nameserver/dns in /etc/resolve.conf
> And configured the nics:
> echo iface eth0 inet6 auto >> /etc/network/interfaces
> echo iface eth2 inet6 auto >> /etc/network/interfaces
> /etc/init.d/networking restart
> Next, restart ACS isolated network without cleanup to have it reconfigure IPv4 nics, firewall, NAT etc
>
>    *
> Next, I created a /64 network for the isolated guest network on eth0 of VR using radvd:
>
> # cat /etc/radvd.conf
> interface eth0
> {
>      AdvSendAdvert on;
>      MinRtrAdvInterval 5;
>      MaxRtrAdvInterval 15;
>      prefix 2001:470:ed36:1::/64
>      {
>          AdvOnLink on;
>          AdvAutonomous on;
>      };
> };
> systemctl restart radvd
> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this ...:1::/64 network
>    *   Finally I added a static route in toy core-router for the new /64 IPv6 range in the isolated network
> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev <local lan nic>
>    *
> ... and I enabled firewall rules to allow any traffic to pass for the new /64 network
>
> And voila all done! I create a domain AAAA record that points to my guest VM IPv6 address a test webserver on
> http://ipv6-isolated-ntwk-demo.yadav.cloud/
>
> (Note: I'll get rid of the tunnel and request a new /48 block after a few days, sharing this solely for testing purposes)
>
>
> Regards.
>
> ________________________________
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: Tuesday, July 20, 2021 12:46
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
>
>
>
> Op 19-07-2021 om 20:38 schreef Kristaps Cudars:
>> Hi Wido,
>>
>> I assume that flouting ip will not work grate with ingress/egress acl on VR.
>>
>>   From regular ACS user perspective:
>> I have Instance with dualstack its running web app on 443.
>> I want to swap instances for whatever reason.
>> In case of IPv4 change d-nat rule.
>> In case of IPv6 if flouting IP was not created upfront he will need to change dns entry that usually has 24h ttl. Inconvenience degradation in experience.
>>
>
> Yes, but, keep in mind that the IP you are using can also be terminated
> on the VR where HAProxy proxies request to the backend VM (could even be
> v4!)
>
> I'm not against DHCPv6, but I have seen many issues with implementing
> it. Therefor I always stick to SLAAC.
>
>>   From ACS admin perspective:
>> I don’t want to have these tickets in helpdesk.
>> You needed to create another flouting IP that it would be seamless- will not work as answer.
>>
>
> I understand that as well.
>
> Wido
>
>>
>> On 2021/07/19 09:05:54, Wido den Hollander <wi...@widodh.nl> wrote:
>>>
>>>
>>> Op 16-07-2021 om 21:46 schreef Kristaps Cudars:
>>>> Hi Wido,
>>>>
>>>> Your proposal is to sacrifice ability to reassign IPv6 to instance, have internal domain prefix, and list/db in ACS what IPv6 has been assigned to what instance and go with RA and SLAAC. For route signaling to switch use BGP/OSPFv3 or manual pre-creation.
>>>>
>>>
>>> You can still list the IPs which have been assigned. You'll know exactly
>>> what IPv6 address a VM has because of the prefix + MAC. Privacy
>>> Extensions need to be disabled in the VM.
>>>
>>> This already works in CloudStack in Shared Networks in this way.
>>>
>>> Using secondary IPs you can always have 'floating' IPv6 addressess.
>>>
>>> Wido
>>>
>>>> Option with RA and managed flag that DHCPv6 is in use to support preset information and ability to create route information from ACS is not an option as DHCPv6 its failing?
>>>>
>>>>
>>>> On 2021/07/16 15:17:42, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>
>>>>>
>>>>> Op 16-07-2021 om 16:42 schreef Hean Seng:
>>>>>> Hi Wido,
>>>>>>
>>>>>> In current setup,  each Cloudstack have own VR, so in this new  IPv6 subnet
>>>>>> allocation , each VR (which have Frr) will need to have peering with ISP
>>>>>> router (and either BGP or Static Route) , and there is 1000 Acocunts,  it
>>>>>> will 1000 BGP session with ISP router ,  Am I right for this ? or I
>>>>>> understand wrong .
>>>>>>
>>>>>
>>>>> Yes, that is correct. A /56 would also be sufficient or a /60 which is
>>>>> enough to allocate a few /64 subnets.
>>>>>
>>>>> 1000 BGP connections isn't really a problem for a proper router at the
>>>>> ISP. OSPF(v3) would be better, but as I said that's poorly supported.
>>>>>
>>>>> The ISP could also install 1000 static routes, but that means that the
>>>>> ISP's router needs to have those configured.
>>>>>
>>>>> http://docs.frrouting.org/en/latest/ospf6d.html
>>>>> (While looking up this URL I see that Frr recently put in a lot of work
>>>>> in OSPFv3, seems better now)
>>>>>
>>>>>> I understand IPv6 is different then IPv4, and in IPv6 it suppose each
>>>>>> devices have own IP. It just how to realize in easy way.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>
>
>
 

> On Fri, Jul 16, 2021 at 8:17 PM Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Op 16-07-2021 om 05:54 schreef Hean Seng:
>>>>>>>> Hi Wido,
>>>>>>>>
>>>>>>>> My initial thought is not like this,  it is the /48 at ISP router, and
>>>>>>> /64
>>>>>>>> subnet assign to AdvanceZoneVR,   AdvanceZoneVR responsible is
>>>>>>>> distribule IPv6 ip (from the assigned /64 sunet) to VM,  and not routing
>>>>>>>> the traffic,   in the VM that get the IPv6 IP will default route to ISP
>>>>>>>> router as gw.   It can may be a bridge over via Advancezone-VR.
>>>>>>>>
>>>>>>>
>>>>>>> How would you bridge this? That sounds like NAT?
>>>>>>>
>>>>>>> IPv6 is meant to be routed. Not to be translated or bridged in any way.
>>>>>>>
>>>>>>> The way a made the drawing is exactly how IPv6 should work in a VPC
>>>>>>> environment.
>>>>>>>
>>>>>>> Traffic flows through the VR where it can do firewalling of the traffic.
>>>>>>>
>>>>>>>> However, If do as the way described in the drawing, then i suppose will
>>>>>>> be
>>>>>>>> another kind of virtual router going to introduce , to get hold the /48
>>>>>>> in
>>>>>>>> this virtual router right ?
>>>>>>>>
>>>>>>>
>>>>>>> It can be the same VR. But keep in mind that IPv6 != IPv4.
>>>>>>>
>>>>>>> The VR will get Frr as a new daemon which can talk BGP with the upper
>>>>>>> network to route traffic.
>>>>>>>
>>>>>>>> After this,  The Advance Zone, NAT's  VR will peer with this new IPv6 VR
>>>>>>>> for getting the IPv6 /64 prefix ?
>>>>>>>>
>>>>>>>
>>>>>>> IPv4 will be behind NAT, but IPv6 will not be behind NAT.
>>>>>>>
>>>>>>>> If do in this way, then I guess  you just only need Static route, with
>>>>>>>> peering ip both end  as one /48 can have a lot of /64 on it.  And
>>>>>>> hardware
>>>>>>>> budgeting for new IPv6-VR will become very important, as all traffic will
>>>>>>>> need to pass over it .
>>>>>>>>
>>>>>>>
>>>>>>> Routing or NAT is the same for the VR. You don't need a very beefy VR
>>>>>>> for this.
>>>>>>>
>>>>>>>> It will be like
>>>>>>>>
>>>>>>>> ISP Router  ------ >  (new IPV6-VR ) ---- > AdvanceZone-VR ----> VM
>>>>>>>>
>>>>>>>> Relationship of (new IPv6 VR) and AdvanceZone-VR , may be considering on
>>>>>>>> OSPF instead of  BGP , otherwise few thousand of AdvanceZone-VR wil have
>>>>>>>> few thousand of BGP session. on new-IPv6-VR
>>>>>>>>
>>>>>>>> Also, I suppose we cannot do ISP router. -->. Advancezone VR direct,   ,
>>>>>>>> otherwise ISP router will be full of /64 prefix route either on BGP( Many
>>>>>>>> BGP Session) , or  Many Static route .   If few thousand account, ti will
>>>>>>>> be few thousand of BGP session with ISP router or few thousand static
>>>>>>> route
>>>>>>>> which  is not possible .
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jul 15, 2021 at 10:47 PM Wido den Hollander <wi...@widodh.nl>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> But you still need routing. See the attached PNG (and draw.io XML).
>>>>>>>>>
>>>>>>>>> You need to route the /48 subnet TO the VR which can then route it to
>>>>>>>>> the Virtual Networks behind the VR.
>>>>>>>>>
>>>>>>>>> There is no other way then routing with either BGP or a Static route.
>>>>>>>>>
>>>>>>>>> Wido
>>>>>>>>>
>>>>>>>>> Op 15-07-2021 om 12:39 schreef Hean Seng:
>>>>>>>>>> Or explain like this :
>>>>>>>>>>
>>>>>>>>>> 1) Cloudstack generate list of /64 subnet from /48 that Network admin
>>>>>>>>>> assigned to Cloudstack
>>>>>>>>>> 2) Cloudsack allocated the subnet (that generated from step1) to
>>>>>>> Virtual
>>>>>>>>>> Router, one Virtual Router have one subniet /64
>>>>>>>>>> 3) Virtual Router allocate single IPv6 (within the range of /64
>>>>>>>>>> allocated to VR)  to VM
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng <heanseng@gmail.com
>>>>>>>>>> <ma...@gmail.com>> wrote:
>>>>>>>>>>
>>>>>>>>>>          Hi Wido,
>>>>>>>>>>
>>>>>>>>>>          I think the /48 is at physical router as gateway , and subnet of
>>>>>>> /64
>>>>>>>>>>          at VR of Cloudstack.   Cloudstack only keep which /48 prefix and
>>>>>>>>>>          vlan information of this /48 to be later split the  /64. to VR.
>>>>>>>>>>
>>>>>>>>>>          And the instances is getting singe IPv6 of /64  IP.   The VR is
>>>>>>>>>>          getting /64.  The default gateway shall goes to /48 of physical
>>>>>>>>>>          router ip .   In this case ,does not need any BGP router .
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>          Similar concept as IPv4 :
>>>>>>>>>>
>>>>>>>>>>          /48 subnet of IPv6 is equivalent to current /24 subnet of IPv4
>>>>>>> that
>>>>>>>>>>          created in Network.
>>>>>>>>>>          and /64  of IPv6 is equivalent to single IP of IPv4 assign to VM.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>          On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander <
>>>>>>> wido@widodh.nl
>>>>>>>>>>          <ma...@widodh.nl>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>              Op 14-07-2021 om 16:44 schreef Hean Seng:
>>>>>>>>>>               > Hi
>>>>>>>>>>               >
>>>>>>>>>>               > I replied in another thread, i think do not need implement
>>>>>>>>>>              BGP or OSPF,
>>>>>>>>>>               > that would be complicated .
>>>>>>>>>>               >
>>>>>>>>>>               > We only need assign  IPv6 's /64 prefix to Virtual Router
>>>>>>>>>>              (VR) in NAT
>>>>>>>>>>               > zone, and the VR responsible to deliver single IPv6 to VM
>>>>>>> via
>>>>>>>>>>              DHCP6.
>>>>>>>>>>               >
>>>>>>>>>>               > In VR, you need to have Default IPv6 route to  Physical
>>>>>>>>>>              Router's /48. IP
>>>>>>>>>>               > as IPv6 Gateway.  Thens should be done .
>>>>>>>>>>               >
>>>>>>>>>>               > Example :
>>>>>>>>>>               > Physical Router Interface
>>>>>>>>>>               >   IPv6 IP : 2000:aaaa::1/48
>>>>>>>>>>               >
>>>>>>>>>>               > Cloudstack  virtual router : 2000:aaaa:200:201::1/64 with
>>>>>>>>>>              default ipv6
>>>>>>>>>>               > route to router ip 2000:aaaa::1
>>>>>>>>>>               > and Clodustack Virtual router dhcp allocate IP to VM , and
>>>>>>>>>>              VM will have
>>>>>>>>>>               > default route to VR. IPv6 2000:aaaa:200:201::1
>>>>>>>>>>               >
>>>>>>>>>>               > So in cloudstack need to allow  user to enter ,  IPv6
>>>>>>>>>>              gwateway , and
>>>>>>>>>>               > the  /48 Ipv6 prefix , then it will self allocate the /64
>>>>>>> ip
>>>>>>>>>>              to the VR ,
>>>>>>>>>>               > and maintain make sure not ovelap allocation
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>
>>>>>>>>>>              But NAT is truly not the solution with IPv6. IPv6 is supposed
>>>>>>> to
>>>>>>>>> be
>>>>>>>>>>              routable. In addition you should avoid DHCPv6 as much as
>>>>>>>>>>              possible as
>>>>>>>>>>              that's not really the intended use-case for address allocation
>>>>>>>>>>              with IPv6.
>>>>>>>>>>
>>>>>>>>>>              In order to route an /48 IPv6 subnet to the VR you have a few
>>>>>>>>>>              possibilities:
>>>>>>>>>>
>>>>>>>>>>              - Static route from the upperlying routers which are outside
>>>>>>> of
>>>>>>>>>>              CloudStack
>>>>>>>>>>              - BGP
>>>>>>>>>>              - OSPFv3 (broken in most cases!)
>>>>>>>>>>              - DHCPv6 Prefix Delegation
>>>>>>>>>>
>>>>>>>>>>              BGP and/or Static routes are still the best bet here.
>>>>>>>>>>
>>>>>>>>>>              So what you do is that you tell CloudStack that you will route
>>>>>>>>>>              2001:db8::/48 to the VR, the VR can then use that to split it
>>>>>>> up
>>>>>>>>>>              into
>>>>>>>>>>              multiple /64 subnets going towards the instances:
>>>>>>>>>>
>>>>>>>>>>              - 2001:db8::/64
>>>>>>>>>>              - 2001:db8:1::/64
>>>>>>>>>>              - 2001:db8:2::/64
>>>>>>>>>>              ...
>>>>>>>>>>              - 2001:db8:f::/64
>>>>>>>>>>
>>>>>>>>>>              And go on.
>>>>>>>>>>
>>>>>>>>>>              In case of BGP you indeed have to tell the VR a few things:
>>>>>>>>>>
>>>>>>>>>>              - It's own AS number
>>>>>>>>>>              - The peer's address(es)
>>>>>>>>>>
>>>>>>>>>>              With FRR you can simply say:
>>>>>>>>>>
>>>>>>>>>>              neighbor 2001:db8:4fa::179 remote-as external
>>>>>>>>>>
>>>>>>>>>>              The /48 you need to have at the VR anyway in case of either a
>>>>>>>>>>              static
>>>>>>>>>>              route or BGP.
>>>>>>>>>>
>>>>>>>>>>              We just need to add a NullRoute on the VR for that /48 so that
>>>>>>>>>>              traffic
>>>>>>>>>>              will not be routed to the upper gateway in case of the VR
>>>>>>> can't
>>>>>>>>>>              find a
>>>>>>>>>>              route.
>>>>>>>>>>
>>>>>>>>>>              Wido
>>>>>>>>>>
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli
>>>>>>>>>>               > <Alex.Mattioli@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>
>>>>>>>>>>              <mailto:Alex.Mattioli@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>>> wrote:
>>>>>>>>>>               >
>>>>>>>>>>               >     Hi Wido,
>>>>>>>>>>               >     That's pretty much in line with our thoughts, thanks
>>>>>>> for
>>>>>>>>>>              the input.
>>>>>>>>>>               >     I believe we agree on the following points then:
>>>>>>>>>>               >
>>>>>>>>>>               >     - FRR with BGP (no OSPF)
>>>>>>>>>>               >     - Route /48 (or/56) down to the VR
>>>>>>>>>>               >     - /64 per network
>>>>>>>>>>               >     - SLACC for IP addressing
>>>>>>>>>>               >
>>>>>>>>>>               >     I believe the next big question is then "on which level
>>>>>>>>>>              of ACS do we
>>>>>>>>>>               >     manage AS numbers?".  I see two options:
>>>>>>>>>>               >     1) Private AS number on a per-zone basis
>>>>>>>>>>               >     2) Root Admin assigned AS number on a domain/account
>>>>>>> basis
>>>>>>>>>>               >     3) End-user driven AS number on a per network basis
>>>>>>> (for
>>>>>>>>>>              bring your
>>>>>>>>>>               >     own AS and IP scenario)
>>>>>>>>>>               >
>>>>>>>>>>               >     Thoughts?
>>>>>>>>>>               >
>>>>>>>>>>               >     Cheers
>>>>>>>>>>               >     Alex
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >     -----Original Message-----
>>>>>>>>>>               >     From: Wido den Hollander <wido@widodh.nl
>>>>>>>>>>              <ma...@widodh.nl> <mailto:wido@widodh.nl
>>>>>>>>>>              <ma...@widodh.nl>>>
>>>>>>>>>>               >     Sent: 13 July 2021 15:08
>>>>>>>>>>               >     To: dev@cloudstack.apache.org
>>>>>>>>>>              <ma...@cloudstack.apache.org>
>>>>>>>>>>              <mailto:dev@cloudstack.apache.org
>>>>>>>>>>              <ma...@cloudstack.apache.org>>;
>>>>>>>>>>               >     Alex Mattioli <Alex.Mattioli@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>
>>>>>>>>>>               >     <mailto:Alex.Mattioli@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>>>
>>>>>>>>>>               >     Cc: Wei Zhou <Wei.Zhou@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>
>>>>>>>>>>               >     <mailto:Wei.Zhou@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>>>; Rohit Yadav
>>>>>>>>>>               >     <rohit.yadav@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>
>>>>>>>>>>              <mailto:rohit.yadav@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>>>;
>>>>>>>>>>               >     Gabriel Beims Bräscher <gabriel@pcextreme.nl
>>>>>>>>>>              <ma...@pcextreme.nl>
>>>>>>>>>>               >     <mailto:gabriel@pcextreme.nl <mailto:
>>>>>>> gabriel@pcextreme.nl
>>>>>>>>>>>>
>>>>>>>>>>               >     Subject: Re: IPV6 in Isolated/VPC networks
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >     On 7/7/21 1:16 PM, Alex Mattioli wrote:
>>>>>>>>>>               >      > Hi all,
>>>>>>>>>>               >      > @Wei Zhou<mailto:Wei.Zhou@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>
>>>>>>>>>>               >     <mailto:Wei.Zhou@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>>> @Rohit
>>>>>>>>>>               >     Yadav<mailto:rohit.yadav@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>
>>>>>>>>>>               >     <mailto:rohit.yadav@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>>> and myself are
>>>>>>>>>>              investigating how
>>>>>>>>>>               >     to enable IPV6 support on Isolated and VPC networks and
>>>>>>>>>>              would like
>>>>>>>>>>               >     your input on it.
>>>>>>>>>>               >      > At the moment we are looking at implementing FRR
>>>>>>> with
>>>>>>>>>>              BGP (and
>>>>>>>>>>               >     possibly OSPF) on the ACS VR.
>>>>>>>>>>               >      >
>>>>>>>>>>               >      > We are looking for requirements, recommendations,
>>>>>>>>>>              ideas, rants,
>>>>>>>>>>               >     etc...etc...
>>>>>>>>>>               >      >
>>>>>>>>>>               >
>>>>>>>>>>               >     Ok! Here we go.
>>>>>>>>>>               >
>>>>>>>>>>               >     I think that you mean that the VR will actually route
>>>>>>> the
>>>>>>>>>>              IPv6
>>>>>>>>>>               >     traffic and for that you need to have a way of getting
>>>>>>> a
>>>>>>>>>>              subnet
>>>>>>>>>>               >     routed to the VR.
>>>>>>>>>>               >
>>>>>>>>>>               >     BGP is probably you best bet here. Although OSPFv3
>>>>>>>>>>              technically
>>>>>>>>>>               >     supports this it is very badly implemented in Frr for
>>>>>>>>>>              example.
>>>>>>>>>>               >
>>>>>>>>>>               >     Now FRR is a very good router and one of the fancy
>>>>>>>>>>              features it
>>>>>>>>>>               >     supports is BGP Unnumered. This allows for auto
>>>>>>>>>>              configuration of BGP
>>>>>>>>>>               >     over a L2 network when both sides are sending Router
>>>>>>>>>>              Advertisements.
>>>>>>>>>>               >     This is very easy for flexible BGP configurations where
>>>>>>>>>>              both sides
>>>>>>>>>>               >     have dynamic IPs.
>>>>>>>>>>               >
>>>>>>>>>>               >     What you want to do is that you get a /56, /48 or
>>>>>>>>>>              something which is
>>>>>>>>>>               >      >/64 bits routed to the VR.
>>>>>>>>>>               >
>>>>>>>>>>               >     Now you can sub-segment this into separate /64 subnets.
>>>>>>>>>>              You don't
>>>>>>>>>>               >     want to go smaller then a /64 is that prevents you from
>>>>>>>>>>              using SLAAC
>>>>>>>>>>               >     for IPv6 address configuration. This is how it works
>>>>>>> for
>>>>>>>>>>              Shared
>>>>>>>>>>               >     Networks now in Basic and Advanced Zones.
>>>>>>>>>>               >
>>>>>>>>>>               >     FRR can now also send out the Router Advertisements on
>>>>>>>>>>              the downlinks
>>>>>>>>>>               >     sending out:
>>>>>>>>>>               >
>>>>>>>>>>               >     - DNS servers
>>>>>>>>>>               >     - DNS domain
>>>>>>>>>>               >     - Prefix (/64) to be used
>>>>>>>>>>               >
>>>>>>>>>>               >     There is no need for DHCPv6. You can calculate the IPv6
>>>>>>>>>>              address the
>>>>>>>>>>               >     VM will obtain by using the MAC and the prefix.
>>>>>>>>>>               >
>>>>>>>>>>               >     So in short:
>>>>>>>>>>               >
>>>>>>>>>>               >     - Using BGP you routed a /48 to the VR
>>>>>>>>>>               >     - Now you split this into /64 subnets towards the
>>>>>>>>>>              isolated networks
>>>>>>>>>>               >
>>>>>>>>>>               >     Wido
>>>>>>>>>>               >
>>>>>>>>>>               >      > Alex Mattioli
>>>>>>>>>>               >      >
>>>>>>>>>>               >      >
>>>>>>>>>>               >      >
>>>>>>>>>>               >      >
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               > --
>>>>>>>>>>               > Regards,
>>>>>>>>>>               > Hean Seng
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>          --
>>>>>>>>>>          Regards,
>>>>>>>>>>          Hean Seng
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Hean Seng
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>

Re: IPV6 in Isolated/VPC networks

Posted by Wido den Hollander <wi...@widodh.nl>.
Hi,

See my inline responses:

Op 11-08-2021 om 14:26 schreef Rohit Yadav:
> Hi all,
> 
> Thanks for your feedback and ideas, I've gone ahead with discussing them with Alex and came up with a PoC/design which can be implemented in the following phases:
> 
>    *   Phase1: implement ipv6 support in isolated networks and VPC with static routing
>    *   Phase2: discuss and implement support for dynamic routing (TBD)
> 
> For Phase1 here's the high-level proposal:
> 
>    *   IPv6 address management:
>       *   At the zone level root-admin specifies a /64 public range that will be used for VRs, then they can add a /48, or /56 IPv6 range for guest networks (to be used by isolated networks and VPC tiers)
>       *   On creation of any IPv6 enabled isolated network or VPC tier, from the /48 or /56 block a /64 network is allocated/used
>       *   We assume SLAAC and autoconfiguration, no DHCPv6 in the zone (discuss: is privacy a concern, can privacy extensions rfc4941 of slaac be explored?)

Privacy Extensions are only a concern for client devices which roam 
between different IPv6 networks.

If you IPv6 address of a client keeps the same suffix (MAC based) and 
switches network then only the prefix (/64) will change.

This way a network like Google, Facebook, etc could track your device 
moving from network to network if they only look at the last 64-bits of 
the IPv6 address.

For servers this is not a problem as you already know in which network 
they are.

>    *   Network offerings: root-admin can create new network offerings (with VPC too) that specifies a network stack option:
>       *   ipv4 only (default, for backward compatibility all networks/offerings post-upgrade migrate to this option)
>       *   ipv4-and-ipv6
>       *   ipv6-only (this can be phase 1.b)
>       *   A new routing option: static (phase1), dynamic (phase2, with multiple sub-options such as ospf/bgp etc...)

This means that the network admin will need to statically route the IPv6 
subnet to the VR's outside IPv6 address, for example, on a JunOS router:

set routing-options rib inet6.0 static route 2001:db8:500::/48 next-hop 
2001:db8:100::50

I'm assuming that 2001:db8:100::50 is the address of the VR on the 
outside (/64) network. In reality this will probably be a longer 
address, but this is for just the example.

>    *   VR changes:
>       *   VR gets its guest and public nics set to inet6 auto
>       *   For each /64 allocated to guest network and VPC tiers, radvd is configured to do RA

radvd is fine, but looking at phase 2 with dynamic routing you might 
already want to look into FRRouting. FRR can also advertise RAs while 
not doing any routing.

interface ens4
   no ipv6 nd suppress-ra
   ipv6 nd prefix 2001:db8:500::/64
   ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53

See: http://docs.frrouting.org/en/latest/ipv6.html

>       *   Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6 firewall rules (ACLs, ingress, egress) are implemented; ACLs between VPC tiers are managed/implemented by ipv6 firewall on VR

Please take a look at the existing security_group.py script which 
implements RFC4890

https://datatracker.ietf.org/doc/html/rfc4890

ICMPv6 is a vital part of IPv6 and certain packets should always be allowed.

>       *   It is assumed that static routes are created on the core/main router by the admin or automated using some scripts/tools; for this CloudStack will announce events with details of /64 networks and VR's public IPv6 address that can be consumed by a rabbitmq/message bus client (for example), or a custom cron job or script as part of orchestration. (this wouldn't be necessary for dynamic routing bgp with phase2)\\

You would only need to announce the /48 or /56 allocated to the VR, 
that's all. You don't need to inform the upstream router about the /64 
subnets created within that larger subnet.

>    *   Guest Networking: With SLAAC, it's easy for CloudStack to calculate allocate and use a /64 and determine the IPv6 address of VR nics and guest VM nics
>       *   A user create an isolated network/VPC with an offering that is ipv6 enabled
>       *   A user can manage firewall for the IPv6 address/guest nics; there'll be no port forward and LB feature though for IPv6
>       *   A users can run workloads in the guest VMs that listen on publically routable ipv6 addresses
>       *   Usage/billing etc continue to work, no change needed
> 
> Network layout:
> 
> [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] -> [guest VMs/nics]
> *core/ISP router needs static routes to be added (manually or automated), assumes a /48 or /56 configured for the zone
> 
> Thoughts, feedback?

Looks doable!

Side-note: It would be very cool if you could use parts of this 
implementation to also route /48, /56, or /60 subnets to individual VMs 
in Shared networks.

Why? This allows for running Docker containers with native IPv6 inside 
the VM or running a (Open)VPN server inside a VM which then assigns 
public IPv6 addresses to clients connected.

Instead of routing the subnet to a VR we route the subnet to a single 
instance in a shared network.

If we could then also move these subnets between Instances easily one 
can quickly migrate to a different instance while keeping the same IPv6 
subnet.

Wido

> 
> Proof-of-concept commentary: here's what I did to test the idea:
> 
>    *   Created an isolated network and deployed a VM in my home lab
> The VR running on KVM has following nics
> eth0 - guest network
> eth1 - link local
> eth2 - public network
> 
>    *   I setup a custom openwrt router on a RPi4 to serve as a toy-core router where I create a wan6 IPv6 tunnel using tunnel broker and I got a /48 allocated. My configuration looks like:
> /48 - 2001:470:ed36::/48 (allocated by tunnel broker)
> /64 - 2001:470:36:3e2::/64 (default allocated by)
> 
> I create a LAN ipv6 (public network for CloudStack VR): at subnet/prefix 0:
> LAN IPv6 address: 2001:470:ed36:0::1/64
> Address mode: SLAAC+stateless DHCP (no dhcpv6)
>    *
>    *
> In the isolated VR, I enabled ipv6 as:
> net.ipv6.conf.all.disable_ipv6 = 0
> net.ipv6.conf.all.forwarding = 1
> net.ipv6.conf.all.accept_ra = 1
> net.ipv6.conf.all.accept_redirects = 1
> net.ipv6.conf.all.autoconf = 1
> 
> Set up a IPv6 nameserver/dns in /etc/resolve.conf
> And configured the nics:
> echo iface eth0 inet6 auto >> /etc/network/interfaces
> echo iface eth2 inet6 auto >> /etc/network/interfaces
> /etc/init.d/networking restart
> Next, restart ACS isolated network without cleanup to have it reconfigure IPv4 nics, firewall, NAT etc
> 
>    *
> Next, I created a /64 network for the isolated guest network on eth0 of VR using radvd:
> 
> # cat /etc/radvd.conf
> interface eth0
> {
>      AdvSendAdvert on;
>      MinRtrAdvInterval 5;
>      MaxRtrAdvInterval 15;
>      prefix 2001:470:ed36:1::/64
>      {
>          AdvOnLink on;
>          AdvAutonomous on;
>      };
> };
> systemctl restart radvd
> All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this ...:1::/64 network
>    *   Finally I added a static route in toy core-router for the new /64 IPv6 range in the isolated network
> 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev <local lan nic>
>    *
> ... and I enabled firewall rules to allow any traffic to pass for the new /64 network
> 
> And voila all done! I create a domain AAAA record that points to my guest VM IPv6 address a test webserver on
> http://ipv6-isolated-ntwk-demo.yadav.cloud/
> 
> (Note: I'll get rid of the tunnel and request a new /48 block after a few days, sharing this solely for testing purposes)
> 
> 
> Regards.
> 
> ________________________________
> From: Wido den Hollander <wi...@widodh.nl>
> Sent: Tuesday, July 20, 2021 12:46
> To: dev@cloudstack.apache.org <de...@cloudstack.apache.org>
> Subject: Re: IPV6 in Isolated/VPC networks
> 
> 
> 
> Op 19-07-2021 om 20:38 schreef Kristaps Cudars:
>> Hi Wido,
>>
>> I assume that flouting ip will not work grate with ingress/egress acl on VR.
>>
>>   From regular ACS user perspective:
>> I have Instance with dualstack its running web app on 443.
>> I want to swap instances for whatever reason.
>> In case of IPv4 change d-nat rule.
>> In case of IPv6 if flouting IP was not created upfront he will need to change dns entry that usually has 24h ttl. Inconvenience degradation in experience.
>>
> 
> Yes, but, keep in mind that the IP you are using can also be terminated
> on the VR where HAProxy proxies request to the backend VM (could even be
> v4!)
> 
> I'm not against DHCPv6, but I have seen many issues with implementing
> it. Therefor I always stick to SLAAC.
> 
>>   From ACS admin perspective:
>> I don’t want to have these tickets in helpdesk.
>> You needed to create another flouting IP that it would be seamless- will not work as answer.
>>
> 
> I understand that as well.
> 
> Wido
> 
>>
>> On 2021/07/19 09:05:54, Wido den Hollander <wi...@widodh.nl> wrote:
>>>
>>>
>>> Op 16-07-2021 om 21:46 schreef Kristaps Cudars:
>>>> Hi Wido,
>>>>
>>>> Your proposal is to sacrifice ability to reassign IPv6 to instance, have internal domain prefix, and list/db in ACS what IPv6 has been assigned to what instance and go with RA and SLAAC. For route signaling to switch use BGP/OSPFv3 or manual pre-creation.
>>>>
>>>
>>> You can still list the IPs which have been assigned. You'll know exactly
>>> what IPv6 address a VM has because of the prefix + MAC. Privacy
>>> Extensions need to be disabled in the VM.
>>>
>>> This already works in CloudStack in Shared Networks in this way.
>>>
>>> Using secondary IPs you can always have 'floating' IPv6 addressess.
>>>
>>> Wido
>>>
>>>> Option with RA and managed flag that DHCPv6 is in use to support preset information and ability to create route information from ACS is not an option as DHCPv6 its failing?
>>>>
>>>>
>>>> On 2021/07/16 15:17:42, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>
>>>>>
>>>>> Op 16-07-2021 om 16:42 schreef Hean Seng:
>>>>>> Hi Wido,
>>>>>>
>>>>>> In current setup,  each Cloudstack have own VR, so in this new  IPv6 subnet
>>>>>> allocation , each VR (which have Frr) will need to have peering with ISP
>>>>>> router (and either BGP or Static Route) , and there is 1000 Acocunts,  it
>>>>>> will 1000 BGP session with ISP router ,  Am I right for this ? or I
>>>>>> understand wrong .
>>>>>>
>>>>>
>>>>> Yes, that is correct. A /56 would also be sufficient or a /60 which is
>>>>> enough to allocate a few /64 subnets.
>>>>>
>>>>> 1000 BGP connections isn't really a problem for a proper router at the
>>>>> ISP. OSPF(v3) would be better, but as I said that's poorly supported.
>>>>>
>>>>> The ISP could also install 1000 static routes, but that means that the
>>>>> ISP's router needs to have those configured.
>>>>>
>>>>> http://docs.frrouting.org/en/latest/ospf6d.html
>>>>> (While looking up this URL I see that Frr recently put in a lot of work
>>>>> in OSPFv3, seems better now)
>>>>>
>>>>>> I understand IPv6 is different then IPv4, and in IPv6 it suppose each
>>>>>> devices have own IP. It just how to realize in easy way.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>   
> 
>> On Fri, Jul 16, 2021 at 8:17 PM Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Op 16-07-2021 om 05:54 schreef Hean Seng:
>>>>>>>> Hi Wido,
>>>>>>>>
>>>>>>>> My initial thought is not like this,  it is the /48 at ISP router, and
>>>>>>> /64
>>>>>>>> subnet assign to AdvanceZoneVR,   AdvanceZoneVR responsible is
>>>>>>>> distribule IPv6 ip (from the assigned /64 sunet) to VM,  and not routing
>>>>>>>> the traffic,   in the VM that get the IPv6 IP will default route to ISP
>>>>>>>> router as gw.   It can may be a bridge over via Advancezone-VR.
>>>>>>>>
>>>>>>>
>>>>>>> How would you bridge this? That sounds like NAT?
>>>>>>>
>>>>>>> IPv6 is meant to be routed. Not to be translated or bridged in any way.
>>>>>>>
>>>>>>> The way a made the drawing is exactly how IPv6 should work in a VPC
>>>>>>> environment.
>>>>>>>
>>>>>>> Traffic flows through the VR where it can do firewalling of the traffic.
>>>>>>>
>>>>>>>> However, If do as the way described in the drawing, then i suppose will
>>>>>>> be
>>>>>>>> another kind of virtual router going to introduce , to get hold the /48
>>>>>>> in
>>>>>>>> this virtual router right ?
>>>>>>>>
>>>>>>>
>>>>>>> It can be the same VR. But keep in mind that IPv6 != IPv4.
>>>>>>>
>>>>>>> The VR will get Frr as a new daemon which can talk BGP with the upper
>>>>>>> network to route traffic.
>>>>>>>
>>>>>>>> After this,  The Advance Zone, NAT's  VR will peer with this new IPv6 VR
>>>>>>>> for getting the IPv6 /64 prefix ?
>>>>>>>>
>>>>>>>
>>>>>>> IPv4 will be behind NAT, but IPv6 will not be behind NAT.
>>>>>>>
>>>>>>>> If do in this way, then I guess  you just only need Static route, with
>>>>>>>> peering ip both end  as one /48 can have a lot of /64 on it.  And
>>>>>>> hardware
>>>>>>>> budgeting for new IPv6-VR will become very important, as all traffic will
>>>>>>>> need to pass over it .
>>>>>>>>
>>>>>>>
>>>>>>> Routing or NAT is the same for the VR. You don't need a very beefy VR
>>>>>>> for this.
>>>>>>>
>>>>>>>> It will be like
>>>>>>>>
>>>>>>>> ISP Router  ------ >  (new IPV6-VR ) ---- > AdvanceZone-VR ----> VM
>>>>>>>>
>>>>>>>> Relationship of (new IPv6 VR) and AdvanceZone-VR , may be considering on
>>>>>>>> OSPF instead of  BGP , otherwise few thousand of AdvanceZone-VR wil have
>>>>>>>> few thousand of BGP session. on new-IPv6-VR
>>>>>>>>
>>>>>>>> Also, I suppose we cannot do ISP router. -->. Advancezone VR direct,   ,
>>>>>>>> otherwise ISP router will be full of /64 prefix route either on BGP( Many
>>>>>>>> BGP Session) , or  Many Static route .   If few thousand account, ti will
>>>>>>>> be few thousand of BGP session with ISP router or few thousand static
>>>>>>> route
>>>>>>>> which  is not possible .
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jul 15, 2021 at 10:47 PM Wido den Hollander <wi...@widodh.nl>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> But you still need routing. See the attached PNG (and draw.io XML).
>>>>>>>>>
>>>>>>>>> You need to route the /48 subnet TO the VR which can then route it to
>>>>>>>>> the Virtual Networks behind the VR.
>>>>>>>>>
>>>>>>>>> There is no other way then routing with either BGP or a Static route.
>>>>>>>>>
>>>>>>>>> Wido
>>>>>>>>>
>>>>>>>>> Op 15-07-2021 om 12:39 schreef Hean Seng:
>>>>>>>>>> Or explain like this :
>>>>>>>>>>
>>>>>>>>>> 1) Cloudstack generate list of /64 subnet from /48 that Network admin
>>>>>>>>>> assigned to Cloudstack
>>>>>>>>>> 2) Cloudsack allocated the subnet (that generated from step1) to
>>>>>>> Virtual
>>>>>>>>>> Router, one Virtual Router have one subniet /64
>>>>>>>>>> 3) Virtual Router allocate single IPv6 (within the range of /64
>>>>>>>>>> allocated to VR)  to VM
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng <heanseng@gmail.com
>>>>>>>>>> <ma...@gmail.com>> wrote:
>>>>>>>>>>
>>>>>>>>>>          Hi Wido,
>>>>>>>>>>
>>>>>>>>>>          I think the /48 is at physical router as gateway , and subnet of
>>>>>>> /64
>>>>>>>>>>          at VR of Cloudstack.   Cloudstack only keep which /48 prefix and
>>>>>>>>>>          vlan information of this /48 to be later split the  /64. to VR.
>>>>>>>>>>
>>>>>>>>>>          And the instances is getting singe IPv6 of /64  IP.   The VR is
>>>>>>>>>>          getting /64.  The default gateway shall goes to /48 of physical
>>>>>>>>>>          router ip .   In this case ,does not need any BGP router .
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>          Similar concept as IPv4 :
>>>>>>>>>>
>>>>>>>>>>          /48 subnet of IPv6 is equivalent to current /24 subnet of IPv4
>>>>>>> that
>>>>>>>>>>          created in Network.
>>>>>>>>>>          and /64  of IPv6 is equivalent to single IP of IPv4 assign to VM.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>          On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander <
>>>>>>> wido@widodh.nl
>>>>>>>>>>          <ma...@widodh.nl>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>              Op 14-07-2021 om 16:44 schreef Hean Seng:
>>>>>>>>>>               > Hi
>>>>>>>>>>               >
>>>>>>>>>>               > I replied in another thread, i think do not need implement
>>>>>>>>>>              BGP or OSPF,
>>>>>>>>>>               > that would be complicated .
>>>>>>>>>>               >
>>>>>>>>>>               > We only need assign  IPv6 's /64 prefix to Virtual Router
>>>>>>>>>>              (VR) in NAT
>>>>>>>>>>               > zone, and the VR responsible to deliver single IPv6 to VM
>>>>>>> via
>>>>>>>>>>              DHCP6.
>>>>>>>>>>               >
>>>>>>>>>>               > In VR, you need to have Default IPv6 route to  Physical
>>>>>>>>>>              Router's /48. IP
>>>>>>>>>>               > as IPv6 Gateway.  Thens should be done .
>>>>>>>>>>               >
>>>>>>>>>>               > Example :
>>>>>>>>>>               > Physical Router Interface
>>>>>>>>>>               >   IPv6 IP : 2000:aaaa::1/48
>>>>>>>>>>               >
>>>>>>>>>>               > Cloudstack  virtual router : 2000:aaaa:200:201::1/64 with
>>>>>>>>>>              default ipv6
>>>>>>>>>>               > route to router ip 2000:aaaa::1
>>>>>>>>>>               > and Clodustack Virtual router dhcp allocate IP to VM , and
>>>>>>>>>>              VM will have
>>>>>>>>>>               > default route to VR. IPv6 2000:aaaa:200:201::1
>>>>>>>>>>               >
>>>>>>>>>>               > So in cloudstack need to allow  user to enter ,  IPv6
>>>>>>>>>>              gwateway , and
>>>>>>>>>>               > the  /48 Ipv6 prefix , then it will self allocate the /64
>>>>>>> ip
>>>>>>>>>>              to the VR ,
>>>>>>>>>>               > and maintain make sure not ovelap allocation
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>
>>>>>>>>>>              But NAT is truly not the solution with IPv6. IPv6 is supposed
>>>>>>> to
>>>>>>>>> be
>>>>>>>>>>              routable. In addition you should avoid DHCPv6 as much as
>>>>>>>>>>              possible as
>>>>>>>>>>              that's not really the intended use-case for address allocation
>>>>>>>>>>              with IPv6.
>>>>>>>>>>
>>>>>>>>>>              In order to route an /48 IPv6 subnet to the VR you have a few
>>>>>>>>>>              possibilities:
>>>>>>>>>>
>>>>>>>>>>              - Static route from the upperlying routers which are outside
>>>>>>> of
>>>>>>>>>>              CloudStack
>>>>>>>>>>              - BGP
>>>>>>>>>>              - OSPFv3 (broken in most cases!)
>>>>>>>>>>              - DHCPv6 Prefix Delegation
>>>>>>>>>>
>>>>>>>>>>              BGP and/or Static routes are still the best bet here.
>>>>>>>>>>
>>>>>>>>>>              So what you do is that you tell CloudStack that you will route
>>>>>>>>>>              2001:db8::/48 to the VR, the VR can then use that to split it
>>>>>>> up
>>>>>>>>>>              into
>>>>>>>>>>              multiple /64 subnets going towards the instances:
>>>>>>>>>>
>>>>>>>>>>              - 2001:db8::/64
>>>>>>>>>>              - 2001:db8:1::/64
>>>>>>>>>>              - 2001:db8:2::/64
>>>>>>>>>>              ...
>>>>>>>>>>              - 2001:db8:f::/64
>>>>>>>>>>
>>>>>>>>>>              And go on.
>>>>>>>>>>
>>>>>>>>>>              In case of BGP you indeed have to tell the VR a few things:
>>>>>>>>>>
>>>>>>>>>>              - It's own AS number
>>>>>>>>>>              - The peer's address(es)
>>>>>>>>>>
>>>>>>>>>>              With FRR you can simply say:
>>>>>>>>>>
>>>>>>>>>>              neighbor 2001:db8:4fa::179 remote-as external
>>>>>>>>>>
>>>>>>>>>>              The /48 you need to have at the VR anyway in case of either a
>>>>>>>>>>              static
>>>>>>>>>>              route or BGP.
>>>>>>>>>>
>>>>>>>>>>              We just need to add a NullRoute on the VR for that /48 so that
>>>>>>>>>>              traffic
>>>>>>>>>>              will not be routed to the upper gateway in case of the VR
>>>>>>> can't
>>>>>>>>>>              find a
>>>>>>>>>>              route.
>>>>>>>>>>
>>>>>>>>>>              Wido
>>>>>>>>>>
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli
>>>>>>>>>>               > <Alex.Mattioli@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>
>>>>>>>>>>              <mailto:Alex.Mattioli@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>>> wrote:
>>>>>>>>>>               >
>>>>>>>>>>               >     Hi Wido,
>>>>>>>>>>               >     That's pretty much in line with our thoughts, thanks
>>>>>>> for
>>>>>>>>>>              the input.
>>>>>>>>>>               >     I believe we agree on the following points then:
>>>>>>>>>>               >
>>>>>>>>>>               >     - FRR with BGP (no OSPF)
>>>>>>>>>>               >     - Route /48 (or/56) down to the VR
>>>>>>>>>>               >     - /64 per network
>>>>>>>>>>               >     - SLACC for IP addressing
>>>>>>>>>>               >
>>>>>>>>>>               >     I believe the next big question is then "on which level
>>>>>>>>>>              of ACS do we
>>>>>>>>>>               >     manage AS numbers?".  I see two options:
>>>>>>>>>>               >     1) Private AS number on a per-zone basis
>>>>>>>>>>               >     2) Root Admin assigned AS number on a domain/account
>>>>>>> basis
>>>>>>>>>>               >     3) End-user driven AS number on a per network basis
>>>>>>> (for
>>>>>>>>>>              bring your
>>>>>>>>>>               >     own AS and IP scenario)
>>>>>>>>>>               >
>>>>>>>>>>               >     Thoughts?
>>>>>>>>>>               >
>>>>>>>>>>               >     Cheers
>>>>>>>>>>               >     Alex
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >     -----Original Message-----
>>>>>>>>>>               >     From: Wido den Hollander <wido@widodh.nl
>>>>>>>>>>              <ma...@widodh.nl> <mailto:wido@widodh.nl
>>>>>>>>>>              <ma...@widodh.nl>>>
>>>>>>>>>>               >     Sent: 13 July 2021 15:08
>>>>>>>>>>               >     To: dev@cloudstack.apache.org
>>>>>>>>>>              <ma...@cloudstack.apache.org>
>>>>>>>>>>              <mailto:dev@cloudstack.apache.org
>>>>>>>>>>              <ma...@cloudstack.apache.org>>;
>>>>>>>>>>               >     Alex Mattioli <Alex.Mattioli@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>
>>>>>>>>>>               >     <mailto:Alex.Mattioli@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>>>
>>>>>>>>>>               >     Cc: Wei Zhou <Wei.Zhou@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>
>>>>>>>>>>               >     <mailto:Wei.Zhou@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>>>; Rohit Yadav
>>>>>>>>>>               >     <rohit.yadav@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>
>>>>>>>>>>              <mailto:rohit.yadav@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>>>;
>>>>>>>>>>               >     Gabriel Beims Bräscher <gabriel@pcextreme.nl
>>>>>>>>>>              <ma...@pcextreme.nl>
>>>>>>>>>>               >     <mailto:gabriel@pcextreme.nl <mailto:
>>>>>>> gabriel@pcextreme.nl
>>>>>>>>>>>>
>>>>>>>>>>               >     Subject: Re: IPV6 in Isolated/VPC networks
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >     On 7/7/21 1:16 PM, Alex Mattioli wrote:
>>>>>>>>>>               >      > Hi all,
>>>>>>>>>>               >      > @Wei Zhou<mailto:Wei.Zhou@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>
>>>>>>>>>>               >     <mailto:Wei.Zhou@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>>> @Rohit
>>>>>>>>>>               >     Yadav<mailto:rohit.yadav@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>
>>>>>>>>>>               >     <mailto:rohit.yadav@shapeblue.com
>>>>>>>>>>              <ma...@shapeblue.com>>> and myself are
>>>>>>>>>>              investigating how
>>>>>>>>>>               >     to enable IPV6 support on Isolated and VPC networks and
>>>>>>>>>>              would like
>>>>>>>>>>               >     your input on it.
>>>>>>>>>>               >      > At the moment we are looking at implementing FRR
>>>>>>> with
>>>>>>>>>>              BGP (and
>>>>>>>>>>               >     possibly OSPF) on the ACS VR.
>>>>>>>>>>               >      >
>>>>>>>>>>               >      > We are looking for requirements, recommendations,
>>>>>>>>>>              ideas, rants,
>>>>>>>>>>               >     etc...etc...
>>>>>>>>>>               >      >
>>>>>>>>>>               >
>>>>>>>>>>               >     Ok! Here we go.
>>>>>>>>>>               >
>>>>>>>>>>               >     I think that you mean that the VR will actually route
>>>>>>> the
>>>>>>>>>>              IPv6
>>>>>>>>>>               >     traffic and for that you need to have a way of getting
>>>>>>> a
>>>>>>>>>>              subnet
>>>>>>>>>>               >     routed to the VR.
>>>>>>>>>>               >
>>>>>>>>>>               >     BGP is probably you best bet here. Although OSPFv3
>>>>>>>>>>              technically
>>>>>>>>>>               >     supports this it is very badly implemented in Frr for
>>>>>>>>>>              example.
>>>>>>>>>>               >
>>>>>>>>>>               >     Now FRR is a very good router and one of the fancy
>>>>>>>>>>              features it
>>>>>>>>>>               >     supports is BGP Unnumered. This allows for auto
>>>>>>>>>>              configuration of BGP
>>>>>>>>>>               >     over a L2 network when both sides are sending Router
>>>>>>>>>>              Advertisements.
>>>>>>>>>>               >     This is very easy for flexible BGP configurations where
>>>>>>>>>>              both sides
>>>>>>>>>>               >     have dynamic IPs.
>>>>>>>>>>               >
>>>>>>>>>>               >     What you want to do is that you get a /56, /48 or
>>>>>>>>>>              something which is
>>>>>>>>>>               >      >/64 bits routed to the VR.
>>>>>>>>>>               >
>>>>>>>>>>               >     Now you can sub-segment this into separate /64 subnets.
>>>>>>>>>>              You don't
>>>>>>>>>>               >     want to go smaller then a /64 is that prevents you from
>>>>>>>>>>              using SLAAC
>>>>>>>>>>               >     for IPv6 address configuration. This is how it works
>>>>>>> for
>>>>>>>>>>              Shared
>>>>>>>>>>               >     Networks now in Basic and Advanced Zones.
>>>>>>>>>>               >
>>>>>>>>>>               >     FRR can now also send out the Router Advertisements on
>>>>>>>>>>              the downlinks
>>>>>>>>>>               >     sending out:
>>>>>>>>>>               >
>>>>>>>>>>               >     - DNS servers
>>>>>>>>>>               >     - DNS domain
>>>>>>>>>>               >     - Prefix (/64) to be used
>>>>>>>>>>               >
>>>>>>>>>>               >     There is no need for DHCPv6. You can calculate the IPv6
>>>>>>>>>>              address the
>>>>>>>>>>               >     VM will obtain by using the MAC and the prefix.
>>>>>>>>>>               >
>>>>>>>>>>               >     So in short:
>>>>>>>>>>               >
>>>>>>>>>>               >     - Using BGP you routed a /48 to the VR
>>>>>>>>>>               >     - Now you split this into /64 subnets towards the
>>>>>>>>>>              isolated networks
>>>>>>>>>>               >
>>>>>>>>>>               >     Wido
>>>>>>>>>>               >
>>>>>>>>>>               >      > Alex Mattioli
>>>>>>>>>>               >      >
>>>>>>>>>>               >      >
>>>>>>>>>>               >      >
>>>>>>>>>>               >      >
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               >
>>>>>>>>>>               > --
>>>>>>>>>>               > Regards,
>>>>>>>>>>               > Hean Seng
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>          --
>>>>>>>>>>          Regards,
>>>>>>>>>>          Hean Seng
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Hean Seng
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>