You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Rohit Yadav <ro...@shapeblue.com> on 2021/08/11 16:37:03 UTC
[DISCUSS] NextGen VR?
All,
We've over the years create a VR that largely is stable but we've discussed the pain of maintaining, extending, and upgrading VRs both in lists and in user-groups and CCC conferences.
On a high-level the pain points are:
* It's difficult to debug, investigate VR for operators and support team
* Maintaining the VR code, fixing bugs, implementing features is a pain for developers; further the xml&json databags based programming model is confusing
* Any fix or changes requires a new systemvmtemplate or VR codebase whose patching requires restarting the VR or destroying an old VR
* No uniform VR programming API (current approach is SSH+databags and execute a script), this makes testing VR difficult and QA in isolation is not possible
* Users forget to register the right systemvmtemplate for a new ACS version and upgrades fail
* Others (please share yours)?
Among these pain points my colleagues have proposed a PR targeting 4.16 [1] that aims to unify systemvmtemplate as a building block that is bundled as part of ACS rpm/deb/* packages which CloudStack will automatically seed/register/use with which upgrades will be as simple as a yum update or apt-get upgrade. Further, my colleagues and I are exploring a live patch API which in near future that can patch a running systemvm/VR using systemvm.iso (or deprecate systemvm.iso and use ssh/scp to patch?) without requiring to reboot/recreate it. Hopefully, this addresses some of those pain points. We request the community for your feedback and review/participation in the PR.
Open questions, topics to discuss and gather feedback:
* VR programming: Should we explore a new light-weight VR agent that provides an API (restful/grpc, or CLI?), some mechanism of live patching VR code, packages, and kernel?
* Refactoring isolated network and VPC codebases into a unified codebase and feature sets (assumption that isolated network are largely a VPC with single tier), does it benefit the community, users, and developers?
* Underlying OS:
* should we consider something other than Debian, any suggestions?
* or explore stable/widely used and maintained opensource router distributions such as OpenWRT [2] which ships with a UI and CLI/configuration system UCI [3]? The cons of the approach are a new dependency and some likely missing packages.
* In current VR codebase, most of the effort is spent in implementing/maintaining router packages/configure codebase which we can get rid of by depending on a stable Linux router distro which ships with some API/config system. Not choosing an existing router distribution means we continue to DIY router programming+config management codebase.
* Any other ideas?
Thoughts, feedback?
[1] https://github.com/apache/cloudstack/pull/4329
[2] https://github.com/openwrt/openwrt/graphs/contributors
[3] https://openwrt.org/docs/guide-user/base-system/uci
Regards.
Re: [DISCUSS] NextGen VR?
Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi PL,
I think Openwrt can do all of them, similar to Debian' apt-get Openwrt ships with a opkg package management tool that supports thousands of pkgs and using opkg you can install any of the packages such as strongswan/ipsec, haproxy, java/jre for systemvm agents etc, customise the rc.d startups scripts etc.
In the meanwhile I'm also exploring Debian and some IPC and rpc mechanisms for the VR agent with some success using named pipes. One issue why VR programming is slow is because for every operation, currently we first create and scp a json file and then call a python script to process unprocessed json databags, this two part process can be combined into a single process where we scp the json databag (or similar rpc payload) directly to a named pipe path which a VR agent listens to and reads and processes (much like a http server), this would make two scp/ssh sessions unnecessary. Still exploring and doing some PoC in this direction.
Regards.
________________________________
From: Pierre-Luc Dion <pd...@apache.org>
Sent: Monday, August 16, 2021 8:24:20 PM
To: dev <de...@cloudstack.apache.org>
Subject: Re: [DISCUSS] NextGen VR?
Hi Rohit,
So look like OpenWRT could be a good alternative then, but how would you
handle other services such as load-balancing, vpn,...? would that be from
another Service VMs, kind of doing service mesh ?
Cheers,
On Mon, Aug 16, 2021 at 5:37 AM Rohit Yadav <ro...@shapeblue.com>
wrote:
> Hi PL,
>
> I tested OpenWRT on KVM, and I don't see any performance loss in terms of
> bandwidth/IO and memory, in fact it seems to be faster than the Debian
> based systemvmtemplate due to:
>
> * Smaller kernel size (the Debian kernel includes all sorts of device
> drivers for sound and what not which is not necessary for VR/systemvm)
> * Faster ssh (the dropbear ssh kernel is on-par with openssh-server
> but has smaller footprint and missing features such as sftp that we don't
> need in VR/systemvm; the ssh and programming seemed really fast compared to
> Debian's openssh-server)
> * Really low memory consumption (Debian10+ eats up some RAM due to
> non-optimised services, esp systemd compared to OpenWRT)
> * Lua based really-fast APIs and bus (openwrt has
> https://openwrt.org/docs/techref/ubus, I'm looking into how we can build
> something similar for our Debian based systemvmtemplate if migrating to
> OpenWRT is going to be a pain or unacceptable by the community)
>
> Based on your comment I found that VyOS rolling release has API support (
> https://blog.vyos.io/vyos-rolling-release-has-got-an-http-api) however
> that is probably not supported for their lts/legacy release, we'll also
> face issue with installing/configuring custom packages (java etc) that we
> need for systemvm/VR.
>
>
> Regards.
>
> ________________________________
> From: Pierre-Luc Dion <pd...@apache.org>
> Sent: Friday, August 13, 2021 21:51
> To: dev <de...@cloudstack.apache.org>
> Subject: Re: [DISCUSS] NextGen VR?
>
> From a certain point of view, OpenWRT is most likely not suitable for our
> needs, this project is highly oriented for router appliance platforms and
> kernel is most likely highly optimised for a small memory footprint and
> might not have performance and virtualization in mind.
>
> It's a shame that there is not some sort of API driven open source VyOS :-S
>
> Anything useful that would come out of the CloudNative projects?
>
>
> On Fri, Aug 13, 2021 at 5:48 AM Rohit Yadav <ro...@shapeblue.com>
> wrote:
>
> > Hi Pierre-Luc,
> >
> > Thanks for replying.
> >
> > We've another thread on-going to discuss IPv6 support where we've indeed
> > discussed idea of introducing dynamic routing capabilities in the VR in a
> > future phase with FRR in the VR.
> >
> > For the VR agent idea, I'm indeed thinking of API driven way (the agent
> > exposes an API for CloudStack and a custom CLI perhaps) for it to manage
> > iptables, software packages/services such as dnsmasq etc. Right now it's
> > possible to configure some of the services in network offering (for ex.
> if
> > you don't want LB etc).
> >
> > I've indeed looked at some alternatives:
> >
> > * Alpine: I experimented a new systemvmtemplate based on Alpine (
> > https://github.com/shapeblue/alpine-systemvm) but we saw many cons with
> > Alpine and the gains weren't significant so we decided to drop the idea
> of
> > moving from Debian to Alpine.
> > * VyOS: I've concerns about project and community (I could be wrong)
> > to have our VR depend on it as the "default" provider; though I see an
> old
> > PR
> > * OpenWRT: Among available router distributions, I really liked it -
> > it has (a) a good health and active community and commit activity (
> > https://github.com/openwrt/openwrt/graphs/contributors), (b) multi-years
> > (10+yrs) of releases (https://openwrt.org/about/history), (c) a
> > configuration system (UCI) and UI (LUCI) which can be used to manage
> > services, interfaces, firewall, routes etc with a Lua based API, (d) opkg
> > packaging system, most software packages or equivalent are available that
> > our Debian based systemvmtemplate needs/uses (java jre workaround from
> > Alpine though), (e) good documentation on extending features/writing
> > plugins etc, (f) they tend to do regular releases and use Linux/LTS
> kernel.
> > * pfSense/opnSense: the major cons are FreeBSD-based kernel and
> skills
> > that CloudStack community may not have in terms of debugging,
> investigation
> > etc like they have for Linux
> >
> > Among the above, OpenWRT is probably worth exploring in my opinion; other
> > some firewall/iptables and iproute2 (interfaces and routes) API/library
> > that can be used by the VR agent on Debian to program the firewall (acls,
> > firewall, pf), IP/interface management and routing features in the VR
> > (currently it's all done in a manual sense which is untested and often
> > source of bugs and issues).
> >
> > Lastly, we already have this PR which aims to do automatic
> > systemvmtemplate registeration/installation/seeding and handle upgrades
> > (targetted for 4.16): https://github.com/apache/cloudstack/pull/4329 The
> > PR also unifies the systemvm template use to be extended for CKS use-case
> > (i.e. no separate template installation required for CKS).
> >
> >
> > Regards.
> >
> > ________________________________
> > From: Pierre-Luc Dion <pd...@apache.org>
> > Sent: Thursday, August 12, 2021 16:45
> > To: dev <de...@cloudstack.apache.org>
> > Subject: Re: [DISCUSS] NextGen VR?
> >
> > Hi Rohit,
> >
> > Like it! Our VR system is due for some rethinking!
> > I don't have much points to add to issues you highlight, it seems pretty
> > complete.
> >
> > Here are some more features or ideas that would be interesting to see in
> a
> > new VR system:
> > - Use or support for routing protocol, such as BGP or OSBP so we could
> > provide a more dynamic PrivateGateway concept. using FRR[1]?
> > - have an API driven way to configure IPtables and other network
> services.
> > - could we decouple network services such as LB, VPNserver, gateways
> from
> > the VR ?
> >
> > Debian has been a pain for building VR because the iso defined in our
> > config need constant update, but on the other hand it's been proved to
> be a
> > reliable OS, we saw some of our VR with uptime over 1000 days.
> >
> > As an alternative OS, I'd be curious to look at VyOS[2] or Alpine Linux.
> >
> > From a certain perspective, us providing the systemvm template make sure
> > that systemVM will deploy. work reliably and make it a single template to
> > test. Compared to a system that would just provide RPM/DEB packages and
> > mechanism to push configs, this could require to test all kinds of
> template
> > scenarios, since users could use any version of distro to deploy their
> > systemVM/VRs.
> >
> > "Users forget to register the right systemvmtemplate for a new ACS
> version
> > and upgrades fail": Maybe we could automatically register the new
> template
> > during the upgrade process? This feature used to exist in the Citrix
> > CloudPlatform.
> >
> >
> > [1] https://frrouting.org/
> > [2] https://github.com/vyos/vyos-1x
> >
> > On Wed, Aug 11, 2021 at 12:37 PM Rohit Yadav <ro...@shapeblue.com>
> > wrote:
> >
> > > All,
> > >
> > > We've over the years create a VR that largely is stable but we've
> > > discussed the pain of maintaining, extending, and upgrading VRs both in
> > > lists and in user-groups and CCC conferences.
> > >
> > > On a high-level the pain points are:
> > >
> > > * It's difficult to debug, investigate VR for operators and support
> > > team
> > > * Maintaining the VR code, fixing bugs, implementing features is a
> > > pain for developers; further the xml&json databags based programming
> > model
> > > is confusing
> > > * Any fix or changes requires a new systemvmtemplate or VR codebase
> > > whose patching requires restarting the VR or destroying an old VR
> > > * No uniform VR programming API (current approach is SSH+databags
> and
> > > execute a script), this makes testing VR difficult and QA in isolation
> is
> > > not possible
> > > * Users forget to register the right systemvmtemplate for a new ACS
> > > version and upgrades fail
> > > * Others (please share yours)?
> > >
> > > Among these pain points my colleagues have proposed a PR targeting 4.16
> > > [1] that aims to unify systemvmtemplate as a building block that is
> > bundled
> > > as part of ACS rpm/deb/* packages which CloudStack will automatically
> > > seed/register/use with which upgrades will be as simple as a yum update
> > or
> > > apt-get upgrade. Further, my colleagues and I are exploring a live
> patch
> > > API which in near future that can patch a running systemvm/VR using
> > > systemvm.iso (or deprecate systemvm.iso and use ssh/scp to patch?)
> > without
> > > requiring to reboot/recreate it. Hopefully, this addresses some of
> those
> > > pain points. We request the community for your feedback and
> > > review/participation in the PR.
> > >
> > > Open questions, topics to discuss and gather feedback:
> > >
> > > * VR programming: Should we explore a new light-weight VR agent
> that
> > > provides an API (restful/grpc, or CLI?), some mechanism of live
> patching
> > VR
> > > code, packages, and kernel?
> > > * Refactoring isolated network and VPC codebases into a unified
> > > codebase and feature sets (assumption that isolated network are
> largely a
> > > VPC with single tier), does it benefit the community, users, and
> > developers?
> > > * Underlying OS:
> > > * should we consider something other than Debian, any
> suggestions?
> > > * or explore stable/widely used and maintained opensource router
> > > distributions such as OpenWRT [2] which ships with a UI and
> > > CLI/configuration system UCI [3]? The cons of the approach are a new
> > > dependency and some likely missing packages.
> > > * In current VR codebase, most of the effort is spent in
> > > implementing/maintaining router packages/configure codebase which we
> can
> > > get rid of by depending on a stable Linux router distro which ships
> with
> > > some API/config system. Not choosing an existing router distribution
> > means
> > > we continue to DIY router programming+config management codebase.
> > > * Any other ideas?
> > >
> > > Thoughts, feedback?
> > >
> > > [1] https://github.com/apache/cloudstack/pull/4329
> > > [2] https://github.com/openwrt/openwrt/graphs/contributors
> > > [3] https://openwrt.org/docs/guide-user/base-system/uci
> > >
> > >
> > > Regards.
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
Re: [DISCUSS] NextGen VR?
Posted by Rafael del Valle <rv...@privaz.io.INVALID>.
Hi All,
Perhaps you also want to have a look at OPNSense. I think others also mentioned it before.
It is mature and stable. It is based on hardened free-bsd.
I have worked with Ansible Modules to automate its deployment and I can see a lot of benefits that would make integration easy. For example its configuration is robust and comes from a single file that can recreate the whole firewall configuration: Routes, VPNs, Certificate Authorities, NAT Rules, DHCP Servers, Reservations, Resolvers, Optional Packages, Intrusion Detection, etc.
Some functions, like Rules, are available via API. I know the project team and they are working on improving API so that all the firewall options are available also via API.
Perhaps there is a natural fit for ACS.
Rafael
On Mon, 2021-08-16 04:54 PM, Pierre-Luc Dion <pd...@apache.org> wrote:
> Hi Rohit,
>
> So look like OpenWRT could be a good alternative then, but how would you
> handle other services such as load-balancing, vpn,...? would that be from
> another Service VMs, kind of doing service mesh ?
>
> Cheers,
>
>
>
> On Mon, Aug 16, 2021 at 5:37 AM Rohit Yadav " target="_blank"><ro...@shapeblue.com>
> wrote:
>
> > Hi PL,
> >
> > I tested OpenWRT on KVM, and I don't see any performance loss in terms of
> > bandwidth/IO and memory, in fact it seems to be faster than the Debian
> > based systemvmtemplate due to:
> >
> > * Smaller kernel size (the Debian kernel includes all sorts of device
> > drivers for sound and what not which is not necessary for VR/systemvm)
> > * Faster ssh (the dropbear ssh kernel is on-par with openssh-server
> > but has smaller footprint and missing features such as sftp that we don't
> > need in VR/systemvm; the ssh and programming seemed really fast compared to
> > Debian's openssh-server)
> > * Really low memory consumption (Debian10+ eats up some RAM due to
> > non-optimised services, esp systemd compared to OpenWRT)
> > * Lua based really-fast APIs and bus (openwrt has
> > https://openwrt.org/docs/techref/ubus, I'm looking into how we can build
> > something similar for our Debian based systemvmtemplate if migrating to
> > OpenWRT is going to be a pain or unacceptable by the community)
> >
> > Based on your comment I found that VyOS rolling release has API support (
> > https://blog.vyos.io/vyos-rolling-release-has-got-an-http-api) however
> > that is probably not supported for their lts/legacy release, we'll also
> > face issue with installing/configuring custom packages (java etc) that we
> > need for systemvm/VR.
> >
> >
> > Regards.
> >
> > ________________________________
> > From: Pierre-Luc Dion " target="_blank"><pd...@apache.org>
> > Sent: Friday, August 13, 2021 21:51
> > To: dev " target="_blank"><de...@cloudstack.apache.org>
> > Subject: Re: [DISCUSS] NextGen VR?
> >
> > From a certain point of view, OpenWRT is most likely not suitable for our
> > needs, this project is highly oriented for router appliance platforms and
> > kernel is most likely highly optimised for a small memory footprint and
> > might not have performance and virtualization in mind.
> >
> > It's a shame that there is not some sort of API driven open source VyOS :-S
> >
> > Anything useful that would come out of the CloudNative projects?
> >
> >
> > On Fri, Aug 13, 2021 at 5:48 AM Rohit Yadav " target="_blank"><ro...@shapeblue.com>
> > wrote:
> >
> > > Hi Pierre-Luc,
> > >
> > > Thanks for replying.
> > >
> > > We've another thread on-going to discuss IPv6 support where we've indeed
> > > discussed idea of introducing dynamic routing capabilities in the VR in a
> > > future phase with FRR in the VR.
> > >
> > > For the VR agent idea, I'm indeed thinking of API driven way (the agent
> > > exposes an API for CloudStack and a custom CLI perhaps) for it to manage
> > > iptables, software packages/services such as dnsmasq etc. Right now it's
> > > possible to configure some of the services in network offering (for ex.
> > if
> > > you don't want LB etc).
> > >
> > > I've indeed looked at some alternatives:
> > >
> > > * Alpine: I experimented a new systemvmtemplate based on Alpine (
> > > https://github.com/shapeblue/alpine-systemvm) but we saw many cons with
> > > Alpine and the gains weren't significant so we decided to drop the idea
> > of
> > > moving from Debian to Alpine.
> > > * VyOS: I've concerns about project and community (I could be wrong)
> > > to have our VR depend on it as the "default" provider; though I see an
> > old
> > > PR
> > > * OpenWRT: Among available router distributions, I really liked it -
> > > it has (a) a good health and active community and commit activity (
> > > https://github.com/openwrt/openwrt/graphs/contributors), (b) multi-years
> > > (10+yrs) of releases (https://openwrt.org/about/history), (c) a
> > > configuration system (UCI) and UI (LUCI) which can be used to manage
> > > services, interfaces, firewall, routes etc with a Lua based API, (d) opkg
> > > packaging system, most software packages or equivalent are available that
> > > our Debian based systemvmtemplate needs/uses (java jre workaround from
> > > Alpine though), (e) good documentation on extending features/writing
> > > plugins etc, (f) they tend to do regular releases and use Linux/LTS
> > kernel.
> > > * pfSense/opnSense: the major cons are FreeBSD-based kernel and
> > skills
> > > that CloudStack community may not have in terms of debugging,
> > investigation
> > > etc like they have for Linux
> > >
> > > Among the above, OpenWRT is probably worth exploring in my opinion; other
> > > some firewall/iptables and iproute2 (interfaces and routes) API/library
> > > that can be used by the VR agent on Debian to program the firewall (acls,
> > > firewall, pf), IP/interface management and routing features in the VR
> > > (currently it's all done in a manual sense which is untested and often
> > > source of bugs and issues).
> > >
> > > Lastly, we already have this PR which aims to do automatic
> > > systemvmtemplate registeration/installation/seeding and handle upgrades
> > > (targetted for 4.16): https://github.com/apache/cloudstack/pull/4329 The
> > > PR also unifies the systemvm template use to be extended for CKS use-case
> > > (i.e. no separate template installation required for CKS).
> > >
> > >
> > > Regards.
> > >
> > > ________________________________
> > > From: Pierre-Luc Dion " target="_blank"><pd...@apache.org>
> > > Sent: Thursday, August 12, 2021 16:45
> > > To: dev " target="_blank"><de...@cloudstack.apache.org>
> > > Subject: Re: [DISCUSS] NextGen VR?
> > >
> > > Hi Rohit,
> > >
> > > Like it! Our VR system is due for some rethinking!
> > > I don't have much points to add to issues you highlight, it seems pretty
> > > complete.
> > >
> > > Here are some more features or ideas that would be interesting to see in
> > a
> > > new VR system:
> > > - Use or support for routing protocol, such as BGP or OSBP so we could
> > > provide a more dynamic PrivateGateway concept. using FRR[1]?
> > > - have an API driven way to configure IPtables and other network
> > services.
> > > - could we decouple network services such as LB, VPNserver, gateways
> > from
> > > the VR ?
> > >
> > > Debian has been a pain for building VR because the iso defined in our
> > > config need constant update, but on the other hand it's been proved to
> > be a
> > > reliable OS, we saw some of our VR with uptime over 1000 days.
> > >
> > > As an alternative OS, I'd be curious to look at VyOS[2] or Alpine Linux.
> > >
> > > From a certain perspective, us providing the systemvm template make sure
> > > that systemVM will deploy. work reliably and make it a single template to
> > > test. Compared to a system that would just provide RPM/DEB packages and
> > > mechanism to push configs, this could require to test all kinds of
> > template
> > > scenarios, since users could use any version of distro to deploy their
> > > systemVM/VRs.
> > >
> > > "Users forget to register the right systemvmtemplate for a new ACS
> > version
> > > and upgrades fail": Maybe we could automatically register the new
> > template
> > > during the upgrade process? This feature used to exist in the Citrix
> > > CloudPlatform.
> > >
> > >
> > > [1] https://frrouting.org/
> > > [2] https://github.com/vyos/vyos-1x
> > >
> > > On Wed, Aug 11, 2021 at 12:37 PM Rohit Yadav " target="_blank"><ro...@shapeblue.com>
> > > wrote:
> > >
> > > > All,
> > > >
> > > > We've over the years create a VR that largely is stable but we've
> > > > discussed the pain of maintaining, extending, and upgrading VRs both in
> > > > lists and in user-groups and CCC conferences.
> > > >
> > > > On a high-level the pain points are:
> > > >
> > > > * It's difficult to debug, investigate VR for operators and support
> > > > team
> > > > * Maintaining the VR code, fixing bugs, implementing features is a
> > > > pain for developers; further the xml&json databags based programming
> > > model
> > > > is confusing
> > > > * Any fix or changes requires a new systemvmtemplate or VR codebase
> > > > whose patching requires restarting the VR or destroying an old VR
> > > > * No uniform VR programming API (current approach is SSH+databags
> > and
> > > > execute a script), this makes testing VR difficult and QA in isolation
> > is
> > > > not possible
> > > > * Users forget to register the right systemvmtemplate for a new ACS
> > > > version and upgrades fail
> > > > * Others (please share yours)?
> > > >
> > > > Among these pain points my colleagues have proposed a PR targeting 4.16
> > > > [1] that aims to unify systemvmtemplate as a building block that is
> > > bundled
> > > > as part of ACS rpm/deb/* packages which CloudStack will automatically
> > > > seed/register/use with which upgrades will be as simple as a yum update
> > > or
> > > > apt-get upgrade. Further, my colleagues and I are exploring a live
> > patch
> > > > API which in near future that can patch a running systemvm/VR using
> > > > systemvm.iso (or deprecate systemvm.iso and use ssh/scp to patch?)
> > > without
> > > > requiring to reboot/recreate it. Hopefully, this addresses some of
> > those
> > > > pain points. We request the community for your feedback and
> > > > review/participation in the PR.
> > > >
> > > > Open questions, topics to discuss and gather feedback:
> > > >
> > > > * VR programming: Should we explore a new light-weight VR agent
> > that
> > > > provides an API (restful/grpc, or CLI?), some mechanism of live
> > patching
> > > VR
> > > > code, packages, and kernel?
> > > > * Refactoring isolated network and VPC codebases into a unified
> > > > codebase and feature sets (assumption that isolated network are
> > largely a
> > > > VPC with single tier), does it benefit the community, users, and
> > > developers?
> > > > * Underlying OS:
> > > > * should we consider something other than Debian, any
> > suggestions?
> > > > * or explore stable/widely used and maintained opensource router
> > > > distributions such as OpenWRT [2] which ships with a UI and
> > > > CLI/configuration system UCI [3]? The cons of the approach are a new
> > > > dependency and some likely missing packages.
> > > > * In current VR codebase, most of the effort is spent in
> > > > implementing/maintaining router packages/configure codebase which we
> > can
> > > > get rid of by depending on a stable Linux router distro which ships
> > with
> > > > some API/config system. Not choosing an existing router distribution
> > > means
> > > > we continue to DIY router programming+config management codebase.
> > > > * Any other ideas?
> > > >
> > > > Thoughts, feedback?
> > > >
> > > > [1] https://github.com/apache/cloudstack/pull/4329
> > > > [2] https://github.com/openwrt/openwrt/graphs/contributors
> > > > [3] https://openwrt.org/docs/guide-user/base-system/uci
> > > >
> > > >
> > > > Regards.
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
Re: [DISCUSS] NextGen VR?
Posted by Pierre-Luc Dion <pd...@apache.org>.
Hi Rohit,
So look like OpenWRT could be a good alternative then, but how would you
handle other services such as load-balancing, vpn,...? would that be from
another Service VMs, kind of doing service mesh ?
Cheers,
On Mon, Aug 16, 2021 at 5:37 AM Rohit Yadav <ro...@shapeblue.com>
wrote:
> Hi PL,
>
> I tested OpenWRT on KVM, and I don't see any performance loss in terms of
> bandwidth/IO and memory, in fact it seems to be faster than the Debian
> based systemvmtemplate due to:
>
> * Smaller kernel size (the Debian kernel includes all sorts of device
> drivers for sound and what not which is not necessary for VR/systemvm)
> * Faster ssh (the dropbear ssh kernel is on-par with openssh-server
> but has smaller footprint and missing features such as sftp that we don't
> need in VR/systemvm; the ssh and programming seemed really fast compared to
> Debian's openssh-server)
> * Really low memory consumption (Debian10+ eats up some RAM due to
> non-optimised services, esp systemd compared to OpenWRT)
> * Lua based really-fast APIs and bus (openwrt has
> https://openwrt.org/docs/techref/ubus, I'm looking into how we can build
> something similar for our Debian based systemvmtemplate if migrating to
> OpenWRT is going to be a pain or unacceptable by the community)
>
> Based on your comment I found that VyOS rolling release has API support (
> https://blog.vyos.io/vyos-rolling-release-has-got-an-http-api) however
> that is probably not supported for their lts/legacy release, we'll also
> face issue with installing/configuring custom packages (java etc) that we
> need for systemvm/VR.
>
>
> Regards.
>
> ________________________________
> From: Pierre-Luc Dion <pd...@apache.org>
> Sent: Friday, August 13, 2021 21:51
> To: dev <de...@cloudstack.apache.org>
> Subject: Re: [DISCUSS] NextGen VR?
>
> From a certain point of view, OpenWRT is most likely not suitable for our
> needs, this project is highly oriented for router appliance platforms and
> kernel is most likely highly optimised for a small memory footprint and
> might not have performance and virtualization in mind.
>
> It's a shame that there is not some sort of API driven open source VyOS :-S
>
> Anything useful that would come out of the CloudNative projects?
>
>
> On Fri, Aug 13, 2021 at 5:48 AM Rohit Yadav <ro...@shapeblue.com>
> wrote:
>
> > Hi Pierre-Luc,
> >
> > Thanks for replying.
> >
> > We've another thread on-going to discuss IPv6 support where we've indeed
> > discussed idea of introducing dynamic routing capabilities in the VR in a
> > future phase with FRR in the VR.
> >
> > For the VR agent idea, I'm indeed thinking of API driven way (the agent
> > exposes an API for CloudStack and a custom CLI perhaps) for it to manage
> > iptables, software packages/services such as dnsmasq etc. Right now it's
> > possible to configure some of the services in network offering (for ex.
> if
> > you don't want LB etc).
> >
> > I've indeed looked at some alternatives:
> >
> > * Alpine: I experimented a new systemvmtemplate based on Alpine (
> > https://github.com/shapeblue/alpine-systemvm) but we saw many cons with
> > Alpine and the gains weren't significant so we decided to drop the idea
> of
> > moving from Debian to Alpine.
> > * VyOS: I've concerns about project and community (I could be wrong)
> > to have our VR depend on it as the "default" provider; though I see an
> old
> > PR
> > * OpenWRT: Among available router distributions, I really liked it -
> > it has (a) a good health and active community and commit activity (
> > https://github.com/openwrt/openwrt/graphs/contributors), (b) multi-years
> > (10+yrs) of releases (https://openwrt.org/about/history), (c) a
> > configuration system (UCI) and UI (LUCI) which can be used to manage
> > services, interfaces, firewall, routes etc with a Lua based API, (d) opkg
> > packaging system, most software packages or equivalent are available that
> > our Debian based systemvmtemplate needs/uses (java jre workaround from
> > Alpine though), (e) good documentation on extending features/writing
> > plugins etc, (f) they tend to do regular releases and use Linux/LTS
> kernel.
> > * pfSense/opnSense: the major cons are FreeBSD-based kernel and
> skills
> > that CloudStack community may not have in terms of debugging,
> investigation
> > etc like they have for Linux
> >
> > Among the above, OpenWRT is probably worth exploring in my opinion; other
> > some firewall/iptables and iproute2 (interfaces and routes) API/library
> > that can be used by the VR agent on Debian to program the firewall (acls,
> > firewall, pf), IP/interface management and routing features in the VR
> > (currently it's all done in a manual sense which is untested and often
> > source of bugs and issues).
> >
> > Lastly, we already have this PR which aims to do automatic
> > systemvmtemplate registeration/installation/seeding and handle upgrades
> > (targetted for 4.16): https://github.com/apache/cloudstack/pull/4329 The
> > PR also unifies the systemvm template use to be extended for CKS use-case
> > (i.e. no separate template installation required for CKS).
> >
> >
> > Regards.
> >
> > ________________________________
> > From: Pierre-Luc Dion <pd...@apache.org>
> > Sent: Thursday, August 12, 2021 16:45
> > To: dev <de...@cloudstack.apache.org>
> > Subject: Re: [DISCUSS] NextGen VR?
> >
> > Hi Rohit,
> >
> > Like it! Our VR system is due for some rethinking!
> > I don't have much points to add to issues you highlight, it seems pretty
> > complete.
> >
> > Here are some more features or ideas that would be interesting to see in
> a
> > new VR system:
> > - Use or support for routing protocol, such as BGP or OSBP so we could
> > provide a more dynamic PrivateGateway concept. using FRR[1]?
> > - have an API driven way to configure IPtables and other network
> services.
> > - could we decouple network services such as LB, VPNserver, gateways
> from
> > the VR ?
> >
> > Debian has been a pain for building VR because the iso defined in our
> > config need constant update, but on the other hand it's been proved to
> be a
> > reliable OS, we saw some of our VR with uptime over 1000 days.
> >
> > As an alternative OS, I'd be curious to look at VyOS[2] or Alpine Linux.
> >
> > From a certain perspective, us providing the systemvm template make sure
> > that systemVM will deploy. work reliably and make it a single template to
> > test. Compared to a system that would just provide RPM/DEB packages and
> > mechanism to push configs, this could require to test all kinds of
> template
> > scenarios, since users could use any version of distro to deploy their
> > systemVM/VRs.
> >
> > "Users forget to register the right systemvmtemplate for a new ACS
> version
> > and upgrades fail": Maybe we could automatically register the new
> template
> > during the upgrade process? This feature used to exist in the Citrix
> > CloudPlatform.
> >
> >
> > [1] https://frrouting.org/
> > [2] https://github.com/vyos/vyos-1x
> >
> > On Wed, Aug 11, 2021 at 12:37 PM Rohit Yadav <ro...@shapeblue.com>
> > wrote:
> >
> > > All,
> > >
> > > We've over the years create a VR that largely is stable but we've
> > > discussed the pain of maintaining, extending, and upgrading VRs both in
> > > lists and in user-groups and CCC conferences.
> > >
> > > On a high-level the pain points are:
> > >
> > > * It's difficult to debug, investigate VR for operators and support
> > > team
> > > * Maintaining the VR code, fixing bugs, implementing features is a
> > > pain for developers; further the xml&json databags based programming
> > model
> > > is confusing
> > > * Any fix or changes requires a new systemvmtemplate or VR codebase
> > > whose patching requires restarting the VR or destroying an old VR
> > > * No uniform VR programming API (current approach is SSH+databags
> and
> > > execute a script), this makes testing VR difficult and QA in isolation
> is
> > > not possible
> > > * Users forget to register the right systemvmtemplate for a new ACS
> > > version and upgrades fail
> > > * Others (please share yours)?
> > >
> > > Among these pain points my colleagues have proposed a PR targeting 4.16
> > > [1] that aims to unify systemvmtemplate as a building block that is
> > bundled
> > > as part of ACS rpm/deb/* packages which CloudStack will automatically
> > > seed/register/use with which upgrades will be as simple as a yum update
> > or
> > > apt-get upgrade. Further, my colleagues and I are exploring a live
> patch
> > > API which in near future that can patch a running systemvm/VR using
> > > systemvm.iso (or deprecate systemvm.iso and use ssh/scp to patch?)
> > without
> > > requiring to reboot/recreate it. Hopefully, this addresses some of
> those
> > > pain points. We request the community for your feedback and
> > > review/participation in the PR.
> > >
> > > Open questions, topics to discuss and gather feedback:
> > >
> > > * VR programming: Should we explore a new light-weight VR agent
> that
> > > provides an API (restful/grpc, or CLI?), some mechanism of live
> patching
> > VR
> > > code, packages, and kernel?
> > > * Refactoring isolated network and VPC codebases into a unified
> > > codebase and feature sets (assumption that isolated network are
> largely a
> > > VPC with single tier), does it benefit the community, users, and
> > developers?
> > > * Underlying OS:
> > > * should we consider something other than Debian, any
> suggestions?
> > > * or explore stable/widely used and maintained opensource router
> > > distributions such as OpenWRT [2] which ships with a UI and
> > > CLI/configuration system UCI [3]? The cons of the approach are a new
> > > dependency and some likely missing packages.
> > > * In current VR codebase, most of the effort is spent in
> > > implementing/maintaining router packages/configure codebase which we
> can
> > > get rid of by depending on a stable Linux router distro which ships
> with
> > > some API/config system. Not choosing an existing router distribution
> > means
> > > we continue to DIY router programming+config management codebase.
> > > * Any other ideas?
> > >
> > > Thoughts, feedback?
> > >
> > > [1] https://github.com/apache/cloudstack/pull/4329
> > > [2] https://github.com/openwrt/openwrt/graphs/contributors
> > > [3] https://openwrt.org/docs/guide-user/base-system/uci
> > >
> > >
> > > Regards.
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
Re: [DISCUSS] NextGen VR?
Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi PL,
I tested OpenWRT on KVM, and I don't see any performance loss in terms of bandwidth/IO and memory, in fact it seems to be faster than the Debian based systemvmtemplate due to:
* Smaller kernel size (the Debian kernel includes all sorts of device drivers for sound and what not which is not necessary for VR/systemvm)
* Faster ssh (the dropbear ssh kernel is on-par with openssh-server but has smaller footprint and missing features such as sftp that we don't need in VR/systemvm; the ssh and programming seemed really fast compared to Debian's openssh-server)
* Really low memory consumption (Debian10+ eats up some RAM due to non-optimised services, esp systemd compared to OpenWRT)
* Lua based really-fast APIs and bus (openwrt has https://openwrt.org/docs/techref/ubus, I'm looking into how we can build something similar for our Debian based systemvmtemplate if migrating to OpenWRT is going to be a pain or unacceptable by the community)
Based on your comment I found that VyOS rolling release has API support (https://blog.vyos.io/vyos-rolling-release-has-got-an-http-api) however that is probably not supported for their lts/legacy release, we'll also face issue with installing/configuring custom packages (java etc) that we need for systemvm/VR.
Regards.
________________________________
From: Pierre-Luc Dion <pd...@apache.org>
Sent: Friday, August 13, 2021 21:51
To: dev <de...@cloudstack.apache.org>
Subject: Re: [DISCUSS] NextGen VR?
From a certain point of view, OpenWRT is most likely not suitable for our
needs, this project is highly oriented for router appliance platforms and
kernel is most likely highly optimised for a small memory footprint and
might not have performance and virtualization in mind.
It's a shame that there is not some sort of API driven open source VyOS :-S
Anything useful that would come out of the CloudNative projects?
On Fri, Aug 13, 2021 at 5:48 AM Rohit Yadav <ro...@shapeblue.com>
wrote:
> Hi Pierre-Luc,
>
> Thanks for replying.
>
> We've another thread on-going to discuss IPv6 support where we've indeed
> discussed idea of introducing dynamic routing capabilities in the VR in a
> future phase with FRR in the VR.
>
> For the VR agent idea, I'm indeed thinking of API driven way (the agent
> exposes an API for CloudStack and a custom CLI perhaps) for it to manage
> iptables, software packages/services such as dnsmasq etc. Right now it's
> possible to configure some of the services in network offering (for ex. if
> you don't want LB etc).
>
> I've indeed looked at some alternatives:
>
> * Alpine: I experimented a new systemvmtemplate based on Alpine (
> https://github.com/shapeblue/alpine-systemvm) but we saw many cons with
> Alpine and the gains weren't significant so we decided to drop the idea of
> moving from Debian to Alpine.
> * VyOS: I've concerns about project and community (I could be wrong)
> to have our VR depend on it as the "default" provider; though I see an old
> PR
> * OpenWRT: Among available router distributions, I really liked it -
> it has (a) a good health and active community and commit activity (
> https://github.com/openwrt/openwrt/graphs/contributors), (b) multi-years
> (10+yrs) of releases (https://openwrt.org/about/history), (c) a
> configuration system (UCI) and UI (LUCI) which can be used to manage
> services, interfaces, firewall, routes etc with a Lua based API, (d) opkg
> packaging system, most software packages or equivalent are available that
> our Debian based systemvmtemplate needs/uses (java jre workaround from
> Alpine though), (e) good documentation on extending features/writing
> plugins etc, (f) they tend to do regular releases and use Linux/LTS kernel.
> * pfSense/opnSense: the major cons are FreeBSD-based kernel and skills
> that CloudStack community may not have in terms of debugging, investigation
> etc like they have for Linux
>
> Among the above, OpenWRT is probably worth exploring in my opinion; other
> some firewall/iptables and iproute2 (interfaces and routes) API/library
> that can be used by the VR agent on Debian to program the firewall (acls,
> firewall, pf), IP/interface management and routing features in the VR
> (currently it's all done in a manual sense which is untested and often
> source of bugs and issues).
>
> Lastly, we already have this PR which aims to do automatic
> systemvmtemplate registeration/installation/seeding and handle upgrades
> (targetted for 4.16): https://github.com/apache/cloudstack/pull/4329 The
> PR also unifies the systemvm template use to be extended for CKS use-case
> (i.e. no separate template installation required for CKS).
>
>
> Regards.
>
> ________________________________
> From: Pierre-Luc Dion <pd...@apache.org>
> Sent: Thursday, August 12, 2021 16:45
> To: dev <de...@cloudstack.apache.org>
> Subject: Re: [DISCUSS] NextGen VR?
>
> Hi Rohit,
>
> Like it! Our VR system is due for some rethinking!
> I don't have much points to add to issues you highlight, it seems pretty
> complete.
>
> Here are some more features or ideas that would be interesting to see in a
> new VR system:
> - Use or support for routing protocol, such as BGP or OSBP so we could
> provide a more dynamic PrivateGateway concept. using FRR[1]?
> - have an API driven way to configure IPtables and other network services.
> - could we decouple network services such as LB, VPNserver, gateways from
> the VR ?
>
> Debian has been a pain for building VR because the iso defined in our
> config need constant update, but on the other hand it's been proved to be a
> reliable OS, we saw some of our VR with uptime over 1000 days.
>
> As an alternative OS, I'd be curious to look at VyOS[2] or Alpine Linux.
>
> From a certain perspective, us providing the systemvm template make sure
> that systemVM will deploy. work reliably and make it a single template to
> test. Compared to a system that would just provide RPM/DEB packages and
> mechanism to push configs, this could require to test all kinds of template
> scenarios, since users could use any version of distro to deploy their
> systemVM/VRs.
>
> "Users forget to register the right systemvmtemplate for a new ACS version
> and upgrades fail": Maybe we could automatically register the new template
> during the upgrade process? This feature used to exist in the Citrix
> CloudPlatform.
>
>
> [1] https://frrouting.org/
> [2] https://github.com/vyos/vyos-1x
>
> On Wed, Aug 11, 2021 at 12:37 PM Rohit Yadav <ro...@shapeblue.com>
> wrote:
>
> > All,
> >
> > We've over the years create a VR that largely is stable but we've
> > discussed the pain of maintaining, extending, and upgrading VRs both in
> > lists and in user-groups and CCC conferences.
> >
> > On a high-level the pain points are:
> >
> > * It's difficult to debug, investigate VR for operators and support
> > team
> > * Maintaining the VR code, fixing bugs, implementing features is a
> > pain for developers; further the xml&json databags based programming
> model
> > is confusing
> > * Any fix or changes requires a new systemvmtemplate or VR codebase
> > whose patching requires restarting the VR or destroying an old VR
> > * No uniform VR programming API (current approach is SSH+databags and
> > execute a script), this makes testing VR difficult and QA in isolation is
> > not possible
> > * Users forget to register the right systemvmtemplate for a new ACS
> > version and upgrades fail
> > * Others (please share yours)?
> >
> > Among these pain points my colleagues have proposed a PR targeting 4.16
> > [1] that aims to unify systemvmtemplate as a building block that is
> bundled
> > as part of ACS rpm/deb/* packages which CloudStack will automatically
> > seed/register/use with which upgrades will be as simple as a yum update
> or
> > apt-get upgrade. Further, my colleagues and I are exploring a live patch
> > API which in near future that can patch a running systemvm/VR using
> > systemvm.iso (or deprecate systemvm.iso and use ssh/scp to patch?)
> without
> > requiring to reboot/recreate it. Hopefully, this addresses some of those
> > pain points. We request the community for your feedback and
> > review/participation in the PR.
> >
> > Open questions, topics to discuss and gather feedback:
> >
> > * VR programming: Should we explore a new light-weight VR agent that
> > provides an API (restful/grpc, or CLI?), some mechanism of live patching
> VR
> > code, packages, and kernel?
> > * Refactoring isolated network and VPC codebases into a unified
> > codebase and feature sets (assumption that isolated network are largely a
> > VPC with single tier), does it benefit the community, users, and
> developers?
> > * Underlying OS:
> > * should we consider something other than Debian, any suggestions?
> > * or explore stable/widely used and maintained opensource router
> > distributions such as OpenWRT [2] which ships with a UI and
> > CLI/configuration system UCI [3]? The cons of the approach are a new
> > dependency and some likely missing packages.
> > * In current VR codebase, most of the effort is spent in
> > implementing/maintaining router packages/configure codebase which we can
> > get rid of by depending on a stable Linux router distro which ships with
> > some API/config system. Not choosing an existing router distribution
> means
> > we continue to DIY router programming+config management codebase.
> > * Any other ideas?
> >
> > Thoughts, feedback?
> >
> > [1] https://github.com/apache/cloudstack/pull/4329
> > [2] https://github.com/openwrt/openwrt/graphs/contributors
> > [3] https://openwrt.org/docs/guide-user/base-system/uci
> >
> >
> > Regards.
> >
> >
> >
> >
>
>
>
>
Re: [DISCUSS] NextGen VR?
Posted by Pierre-Luc Dion <pd...@apache.org>.
From a certain point of view, OpenWRT is most likely not suitable for our
needs, this project is highly oriented for router appliance platforms and
kernel is most likely highly optimised for a small memory footprint and
might not have performance and virtualization in mind.
It's a shame that there is not some sort of API driven open source VyOS :-S
Anything useful that would come out of the CloudNative projects?
On Fri, Aug 13, 2021 at 5:48 AM Rohit Yadav <ro...@shapeblue.com>
wrote:
> Hi Pierre-Luc,
>
> Thanks for replying.
>
> We've another thread on-going to discuss IPv6 support where we've indeed
> discussed idea of introducing dynamic routing capabilities in the VR in a
> future phase with FRR in the VR.
>
> For the VR agent idea, I'm indeed thinking of API driven way (the agent
> exposes an API for CloudStack and a custom CLI perhaps) for it to manage
> iptables, software packages/services such as dnsmasq etc. Right now it's
> possible to configure some of the services in network offering (for ex. if
> you don't want LB etc).
>
> I've indeed looked at some alternatives:
>
> * Alpine: I experimented a new systemvmtemplate based on Alpine (
> https://github.com/shapeblue/alpine-systemvm) but we saw many cons with
> Alpine and the gains weren't significant so we decided to drop the idea of
> moving from Debian to Alpine.
> * VyOS: I've concerns about project and community (I could be wrong)
> to have our VR depend on it as the "default" provider; though I see an old
> PR
> * OpenWRT: Among available router distributions, I really liked it -
> it has (a) a good health and active community and commit activity (
> https://github.com/openwrt/openwrt/graphs/contributors), (b) multi-years
> (10+yrs) of releases (https://openwrt.org/about/history), (c) a
> configuration system (UCI) and UI (LUCI) which can be used to manage
> services, interfaces, firewall, routes etc with a Lua based API, (d) opkg
> packaging system, most software packages or equivalent are available that
> our Debian based systemvmtemplate needs/uses (java jre workaround from
> Alpine though), (e) good documentation on extending features/writing
> plugins etc, (f) they tend to do regular releases and use Linux/LTS kernel.
> * pfSense/opnSense: the major cons are FreeBSD-based kernel and skills
> that CloudStack community may not have in terms of debugging, investigation
> etc like they have for Linux
>
> Among the above, OpenWRT is probably worth exploring in my opinion; other
> some firewall/iptables and iproute2 (interfaces and routes) API/library
> that can be used by the VR agent on Debian to program the firewall (acls,
> firewall, pf), IP/interface management and routing features in the VR
> (currently it's all done in a manual sense which is untested and often
> source of bugs and issues).
>
> Lastly, we already have this PR which aims to do automatic
> systemvmtemplate registeration/installation/seeding and handle upgrades
> (targetted for 4.16): https://github.com/apache/cloudstack/pull/4329 The
> PR also unifies the systemvm template use to be extended for CKS use-case
> (i.e. no separate template installation required for CKS).
>
>
> Regards.
>
> ________________________________
> From: Pierre-Luc Dion <pd...@apache.org>
> Sent: Thursday, August 12, 2021 16:45
> To: dev <de...@cloudstack.apache.org>
> Subject: Re: [DISCUSS] NextGen VR?
>
> Hi Rohit,
>
> Like it! Our VR system is due for some rethinking!
> I don't have much points to add to issues you highlight, it seems pretty
> complete.
>
> Here are some more features or ideas that would be interesting to see in a
> new VR system:
> - Use or support for routing protocol, such as BGP or OSBP so we could
> provide a more dynamic PrivateGateway concept. using FRR[1]?
> - have an API driven way to configure IPtables and other network services.
> - could we decouple network services such as LB, VPNserver, gateways from
> the VR ?
>
> Debian has been a pain for building VR because the iso defined in our
> config need constant update, but on the other hand it's been proved to be a
> reliable OS, we saw some of our VR with uptime over 1000 days.
>
> As an alternative OS, I'd be curious to look at VyOS[2] or Alpine Linux.
>
> From a certain perspective, us providing the systemvm template make sure
> that systemVM will deploy. work reliably and make it a single template to
> test. Compared to a system that would just provide RPM/DEB packages and
> mechanism to push configs, this could require to test all kinds of template
> scenarios, since users could use any version of distro to deploy their
> systemVM/VRs.
>
> "Users forget to register the right systemvmtemplate for a new ACS version
> and upgrades fail": Maybe we could automatically register the new template
> during the upgrade process? This feature used to exist in the Citrix
> CloudPlatform.
>
>
> [1] https://frrouting.org/
> [2] https://github.com/vyos/vyos-1x
>
> On Wed, Aug 11, 2021 at 12:37 PM Rohit Yadav <ro...@shapeblue.com>
> wrote:
>
> > All,
> >
> > We've over the years create a VR that largely is stable but we've
> > discussed the pain of maintaining, extending, and upgrading VRs both in
> > lists and in user-groups and CCC conferences.
> >
> > On a high-level the pain points are:
> >
> > * It's difficult to debug, investigate VR for operators and support
> > team
> > * Maintaining the VR code, fixing bugs, implementing features is a
> > pain for developers; further the xml&json databags based programming
> model
> > is confusing
> > * Any fix or changes requires a new systemvmtemplate or VR codebase
> > whose patching requires restarting the VR or destroying an old VR
> > * No uniform VR programming API (current approach is SSH+databags and
> > execute a script), this makes testing VR difficult and QA in isolation is
> > not possible
> > * Users forget to register the right systemvmtemplate for a new ACS
> > version and upgrades fail
> > * Others (please share yours)?
> >
> > Among these pain points my colleagues have proposed a PR targeting 4.16
> > [1] that aims to unify systemvmtemplate as a building block that is
> bundled
> > as part of ACS rpm/deb/* packages which CloudStack will automatically
> > seed/register/use with which upgrades will be as simple as a yum update
> or
> > apt-get upgrade. Further, my colleagues and I are exploring a live patch
> > API which in near future that can patch a running systemvm/VR using
> > systemvm.iso (or deprecate systemvm.iso and use ssh/scp to patch?)
> without
> > requiring to reboot/recreate it. Hopefully, this addresses some of those
> > pain points. We request the community for your feedback and
> > review/participation in the PR.
> >
> > Open questions, topics to discuss and gather feedback:
> >
> > * VR programming: Should we explore a new light-weight VR agent that
> > provides an API (restful/grpc, or CLI?), some mechanism of live patching
> VR
> > code, packages, and kernel?
> > * Refactoring isolated network and VPC codebases into a unified
> > codebase and feature sets (assumption that isolated network are largely a
> > VPC with single tier), does it benefit the community, users, and
> developers?
> > * Underlying OS:
> > * should we consider something other than Debian, any suggestions?
> > * or explore stable/widely used and maintained opensource router
> > distributions such as OpenWRT [2] which ships with a UI and
> > CLI/configuration system UCI [3]? The cons of the approach are a new
> > dependency and some likely missing packages.
> > * In current VR codebase, most of the effort is spent in
> > implementing/maintaining router packages/configure codebase which we can
> > get rid of by depending on a stable Linux router distro which ships with
> > some API/config system. Not choosing an existing router distribution
> means
> > we continue to DIY router programming+config management codebase.
> > * Any other ideas?
> >
> > Thoughts, feedback?
> >
> > [1] https://github.com/apache/cloudstack/pull/4329
> > [2] https://github.com/openwrt/openwrt/graphs/contributors
> > [3] https://openwrt.org/docs/guide-user/base-system/uci
> >
> >
> > Regards.
> >
> >
> >
> >
>
>
>
>
Re: [DISCUSS] NextGen VR?
Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi Pierre-Luc,
Thanks for replying.
We've another thread on-going to discuss IPv6 support where we've indeed discussed idea of introducing dynamic routing capabilities in the VR in a future phase with FRR in the VR.
For the VR agent idea, I'm indeed thinking of API driven way (the agent exposes an API for CloudStack and a custom CLI perhaps) for it to manage iptables, software packages/services such as dnsmasq etc. Right now it's possible to configure some of the services in network offering (for ex. if you don't want LB etc).
I've indeed looked at some alternatives:
* Alpine: I experimented a new systemvmtemplate based on Alpine (https://github.com/shapeblue/alpine-systemvm) but we saw many cons with Alpine and the gains weren't significant so we decided to drop the idea of moving from Debian to Alpine.
* VyOS: I've concerns about project and community (I could be wrong) to have our VR depend on it as the "default" provider; though I see an old PR
* OpenWRT: Among available router distributions, I really liked it - it has (a) a good health and active community and commit activity (https://github.com/openwrt/openwrt/graphs/contributors), (b) multi-years (10+yrs) of releases (https://openwrt.org/about/history), (c) a configuration system (UCI) and UI (LUCI) which can be used to manage services, interfaces, firewall, routes etc with a Lua based API, (d) opkg packaging system, most software packages or equivalent are available that our Debian based systemvmtemplate needs/uses (java jre workaround from Alpine though), (e) good documentation on extending features/writing plugins etc, (f) they tend to do regular releases and use Linux/LTS kernel.
* pfSense/opnSense: the major cons are FreeBSD-based kernel and skills that CloudStack community may not have in terms of debugging, investigation etc like they have for Linux
Among the above, OpenWRT is probably worth exploring in my opinion; other some firewall/iptables and iproute2 (interfaces and routes) API/library that can be used by the VR agent on Debian to program the firewall (acls, firewall, pf), IP/interface management and routing features in the VR (currently it's all done in a manual sense which is untested and often source of bugs and issues).
Lastly, we already have this PR which aims to do automatic systemvmtemplate registeration/installation/seeding and handle upgrades (targetted for 4.16): https://github.com/apache/cloudstack/pull/4329 The PR also unifies the systemvm template use to be extended for CKS use-case (i.e. no separate template installation required for CKS).
Regards.
________________________________
From: Pierre-Luc Dion <pd...@apache.org>
Sent: Thursday, August 12, 2021 16:45
To: dev <de...@cloudstack.apache.org>
Subject: Re: [DISCUSS] NextGen VR?
Hi Rohit,
Like it! Our VR system is due for some rethinking!
I don't have much points to add to issues you highlight, it seems pretty
complete.
Here are some more features or ideas that would be interesting to see in a
new VR system:
- Use or support for routing protocol, such as BGP or OSBP so we could
provide a more dynamic PrivateGateway concept. using FRR[1]?
- have an API driven way to configure IPtables and other network services.
- could we decouple network services such as LB, VPNserver, gateways from
the VR ?
Debian has been a pain for building VR because the iso defined in our
config need constant update, but on the other hand it's been proved to be a
reliable OS, we saw some of our VR with uptime over 1000 days.
As an alternative OS, I'd be curious to look at VyOS[2] or Alpine Linux.
From a certain perspective, us providing the systemvm template make sure
that systemVM will deploy. work reliably and make it a single template to
test. Compared to a system that would just provide RPM/DEB packages and
mechanism to push configs, this could require to test all kinds of template
scenarios, since users could use any version of distro to deploy their
systemVM/VRs.
"Users forget to register the right systemvmtemplate for a new ACS version
and upgrades fail": Maybe we could automatically register the new template
during the upgrade process? This feature used to exist in the Citrix
CloudPlatform.
[1] https://frrouting.org/
[2] https://github.com/vyos/vyos-1x
On Wed, Aug 11, 2021 at 12:37 PM Rohit Yadav <ro...@shapeblue.com>
wrote:
> All,
>
> We've over the years create a VR that largely is stable but we've
> discussed the pain of maintaining, extending, and upgrading VRs both in
> lists and in user-groups and CCC conferences.
>
> On a high-level the pain points are:
>
> * It's difficult to debug, investigate VR for operators and support
> team
> * Maintaining the VR code, fixing bugs, implementing features is a
> pain for developers; further the xml&json databags based programming model
> is confusing
> * Any fix or changes requires a new systemvmtemplate or VR codebase
> whose patching requires restarting the VR or destroying an old VR
> * No uniform VR programming API (current approach is SSH+databags and
> execute a script), this makes testing VR difficult and QA in isolation is
> not possible
> * Users forget to register the right systemvmtemplate for a new ACS
> version and upgrades fail
> * Others (please share yours)?
>
> Among these pain points my colleagues have proposed a PR targeting 4.16
> [1] that aims to unify systemvmtemplate as a building block that is bundled
> as part of ACS rpm/deb/* packages which CloudStack will automatically
> seed/register/use with which upgrades will be as simple as a yum update or
> apt-get upgrade. Further, my colleagues and I are exploring a live patch
> API which in near future that can patch a running systemvm/VR using
> systemvm.iso (or deprecate systemvm.iso and use ssh/scp to patch?) without
> requiring to reboot/recreate it. Hopefully, this addresses some of those
> pain points. We request the community for your feedback and
> review/participation in the PR.
>
> Open questions, topics to discuss and gather feedback:
>
> * VR programming: Should we explore a new light-weight VR agent that
> provides an API (restful/grpc, or CLI?), some mechanism of live patching VR
> code, packages, and kernel?
> * Refactoring isolated network and VPC codebases into a unified
> codebase and feature sets (assumption that isolated network are largely a
> VPC with single tier), does it benefit the community, users, and developers?
> * Underlying OS:
> * should we consider something other than Debian, any suggestions?
> * or explore stable/widely used and maintained opensource router
> distributions such as OpenWRT [2] which ships with a UI and
> CLI/configuration system UCI [3]? The cons of the approach are a new
> dependency and some likely missing packages.
> * In current VR codebase, most of the effort is spent in
> implementing/maintaining router packages/configure codebase which we can
> get rid of by depending on a stable Linux router distro which ships with
> some API/config system. Not choosing an existing router distribution means
> we continue to DIY router programming+config management codebase.
> * Any other ideas?
>
> Thoughts, feedback?
>
> [1] https://github.com/apache/cloudstack/pull/4329
> [2] https://github.com/openwrt/openwrt/graphs/contributors
> [3] https://openwrt.org/docs/guide-user/base-system/uci
>
>
> Regards.
>
>
>
>
Re: [DISCUSS] NextGen VR?
Posted by Pierre-Luc Dion <pd...@apache.org>.
Hi Rohit,
Like it! Our VR system is due for some rethinking!
I don't have much points to add to issues you highlight, it seems pretty
complete.
Here are some more features or ideas that would be interesting to see in a
new VR system:
- Use or support for routing protocol, such as BGP or OSBP so we could
provide a more dynamic PrivateGateway concept. using FRR[1]?
- have an API driven way to configure IPtables and other network services.
- could we decouple network services such as LB, VPNserver, gateways from
the VR ?
Debian has been a pain for building VR because the iso defined in our
config need constant update, but on the other hand it's been proved to be a
reliable OS, we saw some of our VR with uptime over 1000 days.
As an alternative OS, I'd be curious to look at VyOS[2] or Alpine Linux.
From a certain perspective, us providing the systemvm template make sure
that systemVM will deploy. work reliably and make it a single template to
test. Compared to a system that would just provide RPM/DEB packages and
mechanism to push configs, this could require to test all kinds of template
scenarios, since users could use any version of distro to deploy their
systemVM/VRs.
"Users forget to register the right systemvmtemplate for a new ACS version
and upgrades fail": Maybe we could automatically register the new template
during the upgrade process? This feature used to exist in the Citrix
CloudPlatform.
[1] https://frrouting.org/
[2] https://github.com/vyos/vyos-1x
On Wed, Aug 11, 2021 at 12:37 PM Rohit Yadav <ro...@shapeblue.com>
wrote:
> All,
>
> We've over the years create a VR that largely is stable but we've
> discussed the pain of maintaining, extending, and upgrading VRs both in
> lists and in user-groups and CCC conferences.
>
> On a high-level the pain points are:
>
> * It's difficult to debug, investigate VR for operators and support
> team
> * Maintaining the VR code, fixing bugs, implementing features is a
> pain for developers; further the xml&json databags based programming model
> is confusing
> * Any fix or changes requires a new systemvmtemplate or VR codebase
> whose patching requires restarting the VR or destroying an old VR
> * No uniform VR programming API (current approach is SSH+databags and
> execute a script), this makes testing VR difficult and QA in isolation is
> not possible
> * Users forget to register the right systemvmtemplate for a new ACS
> version and upgrades fail
> * Others (please share yours)?
>
> Among these pain points my colleagues have proposed a PR targeting 4.16
> [1] that aims to unify systemvmtemplate as a building block that is bundled
> as part of ACS rpm/deb/* packages which CloudStack will automatically
> seed/register/use with which upgrades will be as simple as a yum update or
> apt-get upgrade. Further, my colleagues and I are exploring a live patch
> API which in near future that can patch a running systemvm/VR using
> systemvm.iso (or deprecate systemvm.iso and use ssh/scp to patch?) without
> requiring to reboot/recreate it. Hopefully, this addresses some of those
> pain points. We request the community for your feedback and
> review/participation in the PR.
>
> Open questions, topics to discuss and gather feedback:
>
> * VR programming: Should we explore a new light-weight VR agent that
> provides an API (restful/grpc, or CLI?), some mechanism of live patching VR
> code, packages, and kernel?
> * Refactoring isolated network and VPC codebases into a unified
> codebase and feature sets (assumption that isolated network are largely a
> VPC with single tier), does it benefit the community, users, and developers?
> * Underlying OS:
> * should we consider something other than Debian, any suggestions?
> * or explore stable/widely used and maintained opensource router
> distributions such as OpenWRT [2] which ships with a UI and
> CLI/configuration system UCI [3]? The cons of the approach are a new
> dependency and some likely missing packages.
> * In current VR codebase, most of the effort is spent in
> implementing/maintaining router packages/configure codebase which we can
> get rid of by depending on a stable Linux router distro which ships with
> some API/config system. Not choosing an existing router distribution means
> we continue to DIY router programming+config management codebase.
> * Any other ideas?
>
> Thoughts, feedback?
>
> [1] https://github.com/apache/cloudstack/pull/4329
> [2] https://github.com/openwrt/openwrt/graphs/contributors
> [3] https://openwrt.org/docs/guide-user/base-system/uci
>
>
> Regards.
>
>
>
>