You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Nux! <nu...@li.nux.ro> on 2017/10/31 18:54:25 UTC

HTTPS LB and x-forwarded-for

Hello,

Of the people running an LB (VR) with https backends, how do you deal with the lack of x-forwarded-for since for port 443 there's just simple TCP balancing?

Has anyone thought of terminating SSL in the VR instead? Ideas?

Cheers

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

RE: HTTPS LB and x-forwarded-for

Posted by Paul Angus <pa...@shapeblue.com>.
Playing devils advocate,

Is this really CloudStack's job?  If an end-user wants a really high performance load-balancer, shouldn't they use a virtualised NetScaler/F5/KEMP.. whatever.
I definitely would like to see us enable the passthrough of more host processor features to guests though. 



Kind regards,

Paul Angus

paul.angus@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-----Original Message-----
From: Simon Weller [mailto:sweller@ena.com.INVALID] 
Sent: 08 November 2017 14:13
To: kmoossavi@cloudops.com; dev@cloudstack.apache.org; wstevens@cloudops.com
Cc: users@cloudstack.apache.org
Subject: RE: HTTPS LB and x-forwarded-for

I'm assuming we would have the standard openssl version with Intel TLS offload though, right? RHEL ships their FIPS compliant version that strips all the acceleration out. The cpu instruction sets should be passed through from the host, so hopefully that will make a massive difference to decryption speeds and cpu load.

Simon Weller/615-312-6068

-----Original Message-----
From: Pierre-Luc Dion [pdion@cloudops.com]
Received: Wednesday, 08 Nov 2017, 9:00AM
To: dev@cloudstack.apache.org [dev@cloudstack.apache.org]; Khosrow Moossavi [kmoossavi@cloudops.com]; Will Stevens [wstevens@cloudops.com]
CC: users [users@cloudstack.apache.org]
Subject: Re: HTTPS LB and x-forwarded-for

Same challenge here too!

Let's look at improving Load-balancing offering from cloudstack, I guest we should do a feature spec draft soon..,  from my perspective, doing SSL offload on the VR could be problematic if the VR spec if too small, and the default spec of the VR being 1vcpu@256MB, considering it can be the router of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What would be your thought about this ?

I'd be curious to have a LB offering in ACS where it would deploy a redundant traefik[1] beside the VR for doing http and https Load-balancing.
I think it would also be useful if the API of that traefik instance would be available from within the VPC or LBnetwork so is API would be accessible to other apps orchestration tools such as  kubernetes or rancher.

traefik or not, here is what I think is needed by cloudstack in the LB
improvement:

- support http, https (X-Forwarded-For)
- basic persistence tuning (API already exist)
- better backend monitoring, currently only a tcp connect validate if the webserver is up.
- ssl offload
- metric collection, more stats, maybe just export the tool status page to the private network.
- Container world support, right now if you have Rancher or kubernetes cluster, you need to deploy your own LB solution behing mostlikely a static nat., If cloudstack would deploy a traefik instance, Kub or Rancher could reuse this instance and managed it to properly do LB between containers.


What would be your prefered LB tool:
haproxy, traefik or nginx?

CloudStack already have to code to handle SSL certs per projects and accounts if not mistaking because that code was added to support NetScaler as Load-balancer in the past. so one less thing to think about :-)


[1] https://traefik.io/


PL,



On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:

> Thanks Andrija,
>
> LB outside of the VR sounds like a good idea. An appliance based on, 
> say cloud-init + ansible and so on could do the trick; alas it'd need 
> to be outside ACS.
> I guess as users we could maybe come up with a spec for an 
> improvement, at least we'd have something the devs could look at whenever it is possible.
>
> Regards,
> Lucian
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Andrija Panic" <an...@gmail.com>
> > To: "dev" <de...@cloudstack.apache.org>
> > Cc: "users" <us...@cloudstack.apache.org>
> > Sent: Thursday, 2 November, 2017 23:21:37
> > Subject: Re: HTTPS LB and x-forwarded-for
>
> > We used to make some special stuff for one of the clients, where all 
> > LB configuration work is done from outside of the ACS, i.e. python 
> > script to feed/configure VR - install latest haproxy 1.5.x for 
> > transparent proxy, since client insisted on SSL termination done on 
> > backend web SSL
> servers....
> > Not good idea, that is all I can say (custom configuration thing) - 
> > but
> the
> > LB setup is actually good - transparent mode haproxy, works on TCP 
> > level, so you can see "real client IP" on the backend servers (which 
> > must use VR as the default gtw, as per default, so the whole setup works properly).
> >
> > I'm still looking forward to see some special support of LB inside 
> > VR via ACS - proper LB setup inside VR via GUI/API -  i.e. to enable 
> > LB provisioning SCRIPT (bash, or whatever),  where all needed
> > install+configure can be done from client side  - otherwise covering 
> > install+all
> > user cases, with proper HTTP checks and similar....is impossible to 
> > do IMHO.
> >
> > Some other clients, actually have internal FW appliance (i.e. 
> > multihomed VM, acting as gtw for all VMs in all networks), and 
> > haproxy instaled on this device (with NAT configured from VR to this 
> > internal FW/VM, so
> remote
> > IP can be seen properly) - this setup is fully under customer 
> > control,
> and
> > can provide any kind of special haproxy config...
> >
> >
> >
> >
> >
> >
> > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> >
> >> Hello,
> >>
> >> Of the people running an LB (VR) with https backends, how do you 
> >> deal
> with
> >> the lack of x-forwarded-for since for port 443 there's just simple 
> >> TCP balancing?
> >>
> >> Has anyone thought of terminating SSL in the VR instead? Ideas?
> >>
> >> Cheers
> >>
> >> --
> >> Sent from the Delta quadrant using Borg technology!
> >>
> >> Nux!
> >> www.nux.ro
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
>

RE: HTTPS LB and x-forwarded-for

Posted by Paul Angus <pa...@shapeblue.com>.
Playing devils advocate,

Is this really CloudStack's job?  If an end-user wants a really high performance load-balancer, shouldn't they use a virtualised NetScaler/F5/KEMP.. whatever.
I definitely would like to see us enable the passthrough of more host processor features to guests though. 



Kind regards,

Paul Angus

paul.angus@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-----Original Message-----
From: Simon Weller [mailto:sweller@ena.com.INVALID] 
Sent: 08 November 2017 14:13
To: kmoossavi@cloudops.com; dev@cloudstack.apache.org; wstevens@cloudops.com
Cc: users@cloudstack.apache.org
Subject: RE: HTTPS LB and x-forwarded-for

I'm assuming we would have the standard openssl version with Intel TLS offload though, right? RHEL ships their FIPS compliant version that strips all the acceleration out. The cpu instruction sets should be passed through from the host, so hopefully that will make a massive difference to decryption speeds and cpu load.

Simon Weller/615-312-6068

-----Original Message-----
From: Pierre-Luc Dion [pdion@cloudops.com]
Received: Wednesday, 08 Nov 2017, 9:00AM
To: dev@cloudstack.apache.org [dev@cloudstack.apache.org]; Khosrow Moossavi [kmoossavi@cloudops.com]; Will Stevens [wstevens@cloudops.com]
CC: users [users@cloudstack.apache.org]
Subject: Re: HTTPS LB and x-forwarded-for

Same challenge here too!

Let's look at improving Load-balancing offering from cloudstack, I guest we should do a feature spec draft soon..,  from my perspective, doing SSL offload on the VR could be problematic if the VR spec if too small, and the default spec of the VR being 1vcpu@256MB, considering it can be the router of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What would be your thought about this ?

I'd be curious to have a LB offering in ACS where it would deploy a redundant traefik[1] beside the VR for doing http and https Load-balancing.
I think it would also be useful if the API of that traefik instance would be available from within the VPC or LBnetwork so is API would be accessible to other apps orchestration tools such as  kubernetes or rancher.

traefik or not, here is what I think is needed by cloudstack in the LB
improvement:

- support http, https (X-Forwarded-For)
- basic persistence tuning (API already exist)
- better backend monitoring, currently only a tcp connect validate if the webserver is up.
- ssl offload
- metric collection, more stats, maybe just export the tool status page to the private network.
- Container world support, right now if you have Rancher or kubernetes cluster, you need to deploy your own LB solution behing mostlikely a static nat., If cloudstack would deploy a traefik instance, Kub or Rancher could reuse this instance and managed it to properly do LB between containers.


What would be your prefered LB tool:
haproxy, traefik or nginx?

CloudStack already have to code to handle SSL certs per projects and accounts if not mistaking because that code was added to support NetScaler as Load-balancer in the past. so one less thing to think about :-)


[1] https://traefik.io/


PL,



On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:

> Thanks Andrija,
>
> LB outside of the VR sounds like a good idea. An appliance based on, 
> say cloud-init + ansible and so on could do the trick; alas it'd need 
> to be outside ACS.
> I guess as users we could maybe come up with a spec for an 
> improvement, at least we'd have something the devs could look at whenever it is possible.
>
> Regards,
> Lucian
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Andrija Panic" <an...@gmail.com>
> > To: "dev" <de...@cloudstack.apache.org>
> > Cc: "users" <us...@cloudstack.apache.org>
> > Sent: Thursday, 2 November, 2017 23:21:37
> > Subject: Re: HTTPS LB and x-forwarded-for
>
> > We used to make some special stuff for one of the clients, where all 
> > LB configuration work is done from outside of the ACS, i.e. python 
> > script to feed/configure VR - install latest haproxy 1.5.x for 
> > transparent proxy, since client insisted on SSL termination done on 
> > backend web SSL
> servers....
> > Not good idea, that is all I can say (custom configuration thing) - 
> > but
> the
> > LB setup is actually good - transparent mode haproxy, works on TCP 
> > level, so you can see "real client IP" on the backend servers (which 
> > must use VR as the default gtw, as per default, so the whole setup works properly).
> >
> > I'm still looking forward to see some special support of LB inside 
> > VR via ACS - proper LB setup inside VR via GUI/API -  i.e. to enable 
> > LB provisioning SCRIPT (bash, or whatever),  where all needed
> > install+configure can be done from client side  - otherwise covering 
> > install+all
> > user cases, with proper HTTP checks and similar....is impossible to 
> > do IMHO.
> >
> > Some other clients, actually have internal FW appliance (i.e. 
> > multihomed VM, acting as gtw for all VMs in all networks), and 
> > haproxy instaled on this device (with NAT configured from VR to this 
> > internal FW/VM, so
> remote
> > IP can be seen properly) - this setup is fully under customer 
> > control,
> and
> > can provide any kind of special haproxy config...
> >
> >
> >
> >
> >
> >
> > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> >
> >> Hello,
> >>
> >> Of the people running an LB (VR) with https backends, how do you 
> >> deal
> with
> >> the lack of x-forwarded-for since for port 443 there's just simple 
> >> TCP balancing?
> >>
> >> Has anyone thought of terminating SSL in the VR instead? Ideas?
> >>
> >> Cheers
> >>
> >> --
> >> Sent from the Delta quadrant using Borg technology!
> >>
> >> Nux!
> >> www.nux.ro
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
>

RE: HTTPS LB and x-forwarded-for

Posted by Simon Weller <sw...@ena.com.INVALID>.
I'm assuming we would have the standard openssl version with Intel TLS offload though, right? RHEL ships their FIPS compliant version that strips all the acceleration out. The cpu instruction sets should be passed through from the host, so hopefully that will make a massive difference to decryption speeds and cpu load.

Simon Weller/615-312-6068

-----Original Message-----
From: Pierre-Luc Dion [pdion@cloudops.com]
Received: Wednesday, 08 Nov 2017, 9:00AM
To: dev@cloudstack.apache.org [dev@cloudstack.apache.org]; Khosrow Moossavi [kmoossavi@cloudops.com]; Will Stevens [wstevens@cloudops.com]
CC: users [users@cloudstack.apache.org]
Subject: Re: HTTPS LB and x-forwarded-for

Same challenge here too!

Let's look at improving Load-balancing offering from cloudstack, I guest we
should do a feature spec draft soon..,  from my perspective, doing SSL
offload on the VR could be problematic if the VR spec if too small, and the
default spec of the VR being 1vcpu@256MB, considering it can be the router
of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What would
be your thought about this ?

I'd be curious to have a LB offering in ACS where it would deploy a
redundant traefik[1] beside the VR for doing http and https Load-balancing.
I think it would also be useful if the API of that traefik instance would
be available from within the VPC or LBnetwork so is API would be accessible
to other apps orchestration tools such as  kubernetes or rancher.

traefik or not, here is what I think is needed by cloudstack in the LB
improvement:

- support http, https (X-Forwarded-For)
- basic persistence tuning (API already exist)
- better backend monitoring, currently only a tcp connect validate if the
webserver is up.
- ssl offload
- metric collection, more stats, maybe just export the tool status page to
the private network.
- Container world support, right now if you have Rancher or kubernetes
cluster, you need to deploy your own LB solution behing mostlikely a static
nat., If cloudstack would deploy a traefik instance, Kub or Rancher could
reuse this instance and managed it to properly do LB between containers.


What would be your prefered LB tool:
haproxy, traefik or nginx?

CloudStack already have to code to handle SSL certs per projects and
accounts if not mistaking because that code was added to support NetScaler
as Load-balancer in the past. so one less thing to think about :-)


[1] https://traefik.io/


PL,



On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:

> Thanks Andrija,
>
> LB outside of the VR sounds like a good idea. An appliance based on, say
> cloud-init + ansible and so on could do the trick; alas it'd need to be
> outside ACS.
> I guess as users we could maybe come up with a spec for an improvement, at
> least we'd have something the devs could look at whenever it is possible.
>
> Regards,
> Lucian
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Andrija Panic" <an...@gmail.com>
> > To: "dev" <de...@cloudstack.apache.org>
> > Cc: "users" <us...@cloudstack.apache.org>
> > Sent: Thursday, 2 November, 2017 23:21:37
> > Subject: Re: HTTPS LB and x-forwarded-for
>
> > We used to make some special stuff for one of the clients, where all LB
> > configuration work is done from outside of the ACS, i.e. python script to
> > feed/configure VR - install latest haproxy 1.5.x for transparent proxy,
> > since client insisted on SSL termination done on backend web SSL
> servers....
> > Not good idea, that is all I can say (custom configuration thing) - but
> the
> > LB setup is actually good - transparent mode haproxy, works on TCP level,
> > so you can see "real client IP" on the backend servers (which must use VR
> > as the default gtw, as per default, so the whole setup works properly).
> >
> > I'm still looking forward to see some special support of LB inside VR via
> > ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
> > provisioning SCRIPT (bash, or whatever),  where all needed
> > install+configure can be done from client side  - otherwise covering all
> > user cases, with proper HTTP checks and similar....is impossible to do
> > IMHO.
> >
> > Some other clients, actually have internal FW appliance (i.e. multihomed
> > VM, acting as gtw for all VMs in all networks), and haproxy instaled on
> > this device (with NAT configured from VR to this internal FW/VM, so
> remote
> > IP can be seen properly) - this setup is fully under customer control,
> and
> > can provide any kind of special haproxy config...
> >
> >
> >
> >
> >
> >
> > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> >
> >> Hello,
> >>
> >> Of the people running an LB (VR) with https backends, how do you deal
> with
> >> the lack of x-forwarded-for since for port 443 there's just simple TCP
> >> balancing?
> >>
> >> Has anyone thought of terminating SSL in the VR instead? Ideas?
> >>
> >> Cheers
> >>
> >> --
> >> Sent from the Delta quadrant using Borg technology!
> >>
> >> Nux!
> >> www.nux.ro
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
>

RE: HTTPS LB and x-forwarded-for

Posted by Simon Weller <sw...@ena.com.INVALID>.
I'm assuming we would have the standard openssl version with Intel TLS offload though, right? RHEL ships their FIPS compliant version that strips all the acceleration out. The cpu instruction sets should be passed through from the host, so hopefully that will make a massive difference to decryption speeds and cpu load.

Simon Weller/615-312-6068

-----Original Message-----
From: Pierre-Luc Dion [pdion@cloudops.com]
Received: Wednesday, 08 Nov 2017, 9:00AM
To: dev@cloudstack.apache.org [dev@cloudstack.apache.org]; Khosrow Moossavi [kmoossavi@cloudops.com]; Will Stevens [wstevens@cloudops.com]
CC: users [users@cloudstack.apache.org]
Subject: Re: HTTPS LB and x-forwarded-for

Same challenge here too!

Let's look at improving Load-balancing offering from cloudstack, I guest we
should do a feature spec draft soon..,  from my perspective, doing SSL
offload on the VR could be problematic if the VR spec if too small, and the
default spec of the VR being 1vcpu@256MB, considering it can be the router
of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What would
be your thought about this ?

I'd be curious to have a LB offering in ACS where it would deploy a
redundant traefik[1] beside the VR for doing http and https Load-balancing.
I think it would also be useful if the API of that traefik instance would
be available from within the VPC or LBnetwork so is API would be accessible
to other apps orchestration tools such as  kubernetes or rancher.

traefik or not, here is what I think is needed by cloudstack in the LB
improvement:

- support http, https (X-Forwarded-For)
- basic persistence tuning (API already exist)
- better backend monitoring, currently only a tcp connect validate if the
webserver is up.
- ssl offload
- metric collection, more stats, maybe just export the tool status page to
the private network.
- Container world support, right now if you have Rancher or kubernetes
cluster, you need to deploy your own LB solution behing mostlikely a static
nat., If cloudstack would deploy a traefik instance, Kub or Rancher could
reuse this instance and managed it to properly do LB between containers.


What would be your prefered LB tool:
haproxy, traefik or nginx?

CloudStack already have to code to handle SSL certs per projects and
accounts if not mistaking because that code was added to support NetScaler
as Load-balancer in the past. so one less thing to think about :-)


[1] https://traefik.io/


PL,



On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:

> Thanks Andrija,
>
> LB outside of the VR sounds like a good idea. An appliance based on, say
> cloud-init + ansible and so on could do the trick; alas it'd need to be
> outside ACS.
> I guess as users we could maybe come up with a spec for an improvement, at
> least we'd have something the devs could look at whenever it is possible.
>
> Regards,
> Lucian
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Andrija Panic" <an...@gmail.com>
> > To: "dev" <de...@cloudstack.apache.org>
> > Cc: "users" <us...@cloudstack.apache.org>
> > Sent: Thursday, 2 November, 2017 23:21:37
> > Subject: Re: HTTPS LB and x-forwarded-for
>
> > We used to make some special stuff for one of the clients, where all LB
> > configuration work is done from outside of the ACS, i.e. python script to
> > feed/configure VR - install latest haproxy 1.5.x for transparent proxy,
> > since client insisted on SSL termination done on backend web SSL
> servers....
> > Not good idea, that is all I can say (custom configuration thing) - but
> the
> > LB setup is actually good - transparent mode haproxy, works on TCP level,
> > so you can see "real client IP" on the backend servers (which must use VR
> > as the default gtw, as per default, so the whole setup works properly).
> >
> > I'm still looking forward to see some special support of LB inside VR via
> > ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
> > provisioning SCRIPT (bash, or whatever),  where all needed
> > install+configure can be done from client side  - otherwise covering all
> > user cases, with proper HTTP checks and similar....is impossible to do
> > IMHO.
> >
> > Some other clients, actually have internal FW appliance (i.e. multihomed
> > VM, acting as gtw for all VMs in all networks), and haproxy instaled on
> > this device (with NAT configured from VR to this internal FW/VM, so
> remote
> > IP can be seen properly) - this setup is fully under customer control,
> and
> > can provide any kind of special haproxy config...
> >
> >
> >
> >
> >
> >
> > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> >
> >> Hello,
> >>
> >> Of the people running an LB (VR) with https backends, how do you deal
> with
> >> the lack of x-forwarded-for since for port 443 there's just simple TCP
> >> balancing?
> >>
> >> Has anyone thought of terminating SSL in the VR instead? Ideas?
> >>
> >> Cheers
> >>
> >> --
> >> Sent from the Delta quadrant using Borg technology!
> >>
> >> Nux!
> >> www.nux.ro
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
>

Re: HTTPS LB and x-forwarded-for

Posted by Nux! <nu...@li.nux.ro>.
lol.... good find..

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Pierre-Luc Dion" <pd...@cloudops.com>
> To: "Wido den Hollander" <wi...@widodh.nl>
> Cc: "dev" <de...@cloudstack.apache.org>, "users" <us...@cloudstack.apache.org>
> Sent: Friday, 17 November, 2017 13:56:11
> Subject: Re: HTTPS LB and x-forwarded-for

> That's funny!
> 
> Cloudstack ui does not provide lb protocol options, but the api does and
> cloudstack already support proxy proto v1!!!
> 
> So that's cool!
> 
> Le 13 nov. 2017 09 h 18, "Wido den Hollander" <wi...@widodh.nl> a écrit :
> 
>>
>> > Op 13 november 2017 om 15:14 schreef Pierre-Luc Dion <pdion@cloudops.com
>> >:
>> >
>> >
>> > Hi,
>> >
>> > This is looking quite promising, I have a colleague that tested that last
>> > Friday, look like the proxy proto v1 work out of the box with Nginx, but
>> > would need an extra package for Apache 2.4 ?
>>
>> It depends. You need HTTPd 2.4.28, see: https://httpd.apache.org/docs/
>> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
>>
>> It's there upstream, but not in all packages.
>>
>> It can from this module:
>>
>> - https://github.com/roadrunner2/mod-proxy-protocol
>> - https://roadrunner2.github.io/mod-proxy-protocol/mod_proxy_protocol.html
>>
>> They donated the code to go upstream and went into mod_remoteip but landed
>> in 2.4.28
>>
>> It will probably make it into Ubuntu 18.04 and CentOS 7.4.
>>
>> Wido
>>
>> > Otherwise, it seems to be a good way to do  https LB without complicated
>> > configuration and huge changes in CloudStack.
>> >
>> >
>> >
>> > On Fri, Nov 10, 2017 at 10:36 AM, Nux! <nu...@li.nux.ro> wrote:
>> >
>> > > Pierre-Luc,
>> > >
>> > > Haproxy docs say it should work for any kind of traffic as long as both
>> > > ends are PROXY-aware and it look like a majority of software is.
>> > > So, in short, yes.
>> > >
>> > > --
>> > > Sent from the Delta quadrant using Borg technology!
>> > >
>> > > Nux!
>> > > www.nux.ro
>> > >
>> > > ----- Original Message -----
>> > > > From: "Pierre-Luc Dion" <pd...@cloudops.com>
>> > > > To: "Wido den Hollander" <wi...@widodh.nl>
>> > > > Cc: "dev" <de...@cloudstack.apache.org>, "Khosrow Moossavi" <
>> > > kmoossavi@cloudops.com>, "Will Stevens"
>> > > > <ws...@cloudops.com>, "Nux!" <nu...@li.nux.ro>, "users" <
>> > > users@cloudstack.apache.org>
>> > > > Sent: Friday, 10 November, 2017 15:32:38
>> > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > >
>> > > > Hi Wido, do you know if this would work for https traffic too?
>> > > >
>> > > > Le 10 nov. 2017 09 h 35, "Wido den Hollander" <wi...@widodh.nl> a
>> écrit :
>> > > >
>> > > >>
>> > > >> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <
>> > > pdion@cloudops.com
>> > > >> >:
>> > > >> >
>> > > >> >
>> > > >> > I kind of like the proxy backend type, ill check on our end if
>> that
>> > > would
>> > > >> > work but definitely a simple and efficient approach!
>> > > >> >
>> > > >>
>> > > >> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
>> > > >>
>> > > >> Apache HTTPd supports PROXY since 2.4.28:
>> > > https://httpd.apache.org/docs/
>> > > >> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
>> > > >>
>> > > >> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
>> > > >>
>> > > >> Wido
>> > > >>
>> > > >> >
>> > > >> >
>> > > >> > Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wi...@widodh.nl> a
>> > > écrit :
>> > > >> >
>> > > >> > >
>> > > >> > > > Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
>> > > >> > > >
>> > > >> > > >
>> > > >> > > > Wido,
>> > > >> > > >
>> > > >> > > > Excellent suggestion with the "transparent proxy", I was not
>> > > aware of
>> > > >> > > that.
>> > > >> > > > I think that would be a great idea and wouldn't require too
>> many
>> > > >> > > modifications, especially as Haproxy comes already with the VR.
>> > > >> > > >
>> > > >> > >
>> > > >> > > It's indeed just a matter of a HAProxy config setting. We could
>> > > make it
>> > > >> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY
>> for
>> > > >> example.
>> > > >> > >
>> > > >> > > That way your problem would be solved.
>> > > >> > >
>> > > >> > > Wido
>> > > >> > >
>> > > >> > > > To Paul:
>> > > >> > > > - imho the LB solution ACS ships now is a bit handicaped since
>> > > you do
>> > > >> > > not know the remote host ip. You're flying blind unless you use
>> > > google
>> > > >> > > analytics (and these things have gotten more and more
>> aggressively
>> > > >> filtered
>> > > >> > > by adblocks).
>> > > >> > > > Enhancing Haproxy as Wido suggested would go a long way, it
>> > > wouldn't
>> > > >> > > break existing functionality and would also keep SSL processing
>> off
>> > > >> the VR.
>> > > >> > > >
>> > > >> > > > --
>> > > >> > > > Sent from the Delta quadrant using Borg technology!
>> > > >> > > >
>> > > >> > > > Nux!
>> > > >> > > > www.nux.ro
>> > > >> > > >
>> > > >> > > > ----- Original Message -----
>> > > >> > > > > From: "Andrija Panic" <an...@gmail.com>
>> > > >> > > > > To: "users" <us...@cloudstack.apache.org>
>> > > >> > > > > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will
>> > > Stevens" <
>> > > >> > > wstevens@cloudops.com>, "dev"
>> > > >> > > > > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <
>> > > pdion@cloudops.com
>> > > >> >
>> > > >> > > > > Sent: Thursday, 9 November, 2017 13:10:58
>> > > >> > > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > > >> > > >
>> > > >> > > > > Wido,
>> > > >> > > > >
>> > > >> > > > > backend servers are not Linux only, for example we have a
>> ton of
>> > > >> > > Windows
>> > > >> > > > > customers, some WEB solutions / IIS etc...
>> > > >> > > > >
>> > > >> > > > > @all - If we try to please/solve everyone's proxying
>> > > >> > > solution/requirement -
>> > > >> > > > > this is impossible IMHO - I'm thinking more about some "do
>> it as
>> > > >> you
>> > > >> > > like"
>> > > >> > > > > solution, to let customer write his own haproxy config and
>> > > upoad it
>> > > >> > > (for
>> > > >> > > > > example, or something better?).
>> > > >> > > > >
>> > > >> > > > > We can support newer version of haproxy (1.5+) which also
>> > > implement
>> > > >> > > > > "transarent proxy" (integrate with kernel so to speak)  to
>> allow
>> > > >> > > TCP-level
>> > > >> > > > > connections to backend (TCP mode, not HTTP mode) but to
>> still
>> > > >> > > "preserve"
>> > > >> > > > > remote IP by faking it (fake soruce IP = transarent proxy).
>> > > >> > > > >
>> > > >> > > > > For the rest of configuration options,  I would leave it to
>> the
>> > > >> > > customer
>> > > >> > > > > how he/she wants to configure rest of haproxy configuration,
>> > > >> inlcuding
>> > > >> > > > > custom checks, etc. Haproxy configuration is never-ending
>> story,
>> > > >> and we
>> > > >> > > > > probably should allow custom sripts/configuration instead of
>> > > >> trying to
>> > > >> > > > > provide GUI/API way to configure everything (which is
>> > > >> impossible...)
>> > > >> > > > >
>> > > >> > > > > Just my 2 cents...
>> > > >> > > > >
>> > > >> > > > > On 9 November 2017 at 08:13, Wido den Hollander <
>> wido@widodh.nl
>> > > >
>> > > >> > > wrote:
>> > > >> > > > >
>> > > >> > > > >>
>> > > >> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
>> > > >> > > pdion@cloudops.com
>> > > >> > > > >> >:
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> > Same challenge here too!
>> > > >> > > > >> >
>> > > >> > > > >> > Let's look at improving Load-balancing offering from
>> > > >> cloudstack, I
>> > > >> > > guest
>> > > >> > > > >> we
>> > > >> > > > >> > should do a feature spec draft soon..,  from my
>> perspective,
>> > > >> doing
>> > > >> > > SSL
>> > > >> > > > >> > offload on the VR could be problematic if the VR spec if
>> too
>> > > >> small,
>> > > >> > > and
>> > > >> > > > >> the
>> > > >> > > > >> > default spec of the VR being 1vcpu@256MB, considering
>> it can
>> > > >> be the
>> > > >> > > > >> router
>> > > >> > > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit
>> > > ish...
>> > > >> What
>> > > >> > > would
>> > > >> > > > >> > be your thought about this ?
>> > > >> > > > >> >
>> > > >> > > > >> > I'd be curious to have a LB offering in ACS where it
>> would
>> > > >> deploy a
>> > > >> > > > >> > redundant traefik[1] beside the VR for doing http and
>> https
>> > > >> > > > >> Load-balancing.
>> > > >> > > > >> > I think it would also be useful if the API of that
>> traefik
>> > > >> instance
>> > > >> > > would
>> > > >> > > > >> > be available from within the VPC or LBnetwork so is API
>> > > would be
>> > > >> > > > >> accessible
>> > > >> > > > >> > to other apps orchestration tools such as  kubernetes or
>> > > >> rancher.
>> > > >> > > > >> >
>> > > >> > > > >> > traefik or not, here is what I think is needed by
>> cloudstack
>> > > in
>> > > >> the
>> > > >> > > LB
>> > > >> > > > >> > improvement:
>> > > >> > > > >> >
>> > > >> > > > >> > - support http, https (X-Forwarded-For)
>> > > >> > > > >>
>> > > >> > > > >> HAProxy also supports the PROXY protocol towards the
>> backends.
>> > > >> Apache
>> > > >> > > > >> 2.4.22 supports this natively and Varnish for example can
>> also
>> > > >> talk
>> > > >> > > PROXY.
>> > > >> > > > >>
>> > > >> > > > >> It adds a littlebit of metadata to the connection so that
>> the
>> > > >> backend
>> > > >> > > > >> knows the original IP the connection came from for example:
>> > > >> > > > >> https://www.haproxy.org/download/1.8/doc/proxy-
>> protocol.txt
>> > > >> > > > >>
>> > > >> > > > >> Wido
>> > > >> > > > >>
>> > > >> > > > >> > - basic persistence tuning (API already exist)
>> > > >> > > > >> > - better backend monitoring, currently only a tcp connect
>> > > >> validate
>> > > >> > > if the
>> > > >> > > > >> > webserver is up.
>> > > >> > > > >> > - ssl offload
>> > > >> > > > >> > - metric collection, more stats, maybe just export the
>> tool
>> > > >> status
>> > > >> > > page
>> > > >> > > > >> to
>> > > >> > > > >> > the private network.
>> > > >> > > > >> > - Container world support, right now if you have Rancher
>> or
>> > > >> > > kubernetes
>> > > >> > > > >> > cluster, you need to deploy your own LB solution behing
>> > > >> mostlikely a
>> > > >> > > > >> static
>> > > >> > > > >> > nat., If cloudstack would deploy a traefik instance, Kub
>> or
>> > > >> Rancher
>> > > >> > > could
>> > > >> > > > >> > reuse this instance and managed it to properly do LB
>> between
>> > > >> > > containers.
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> > What would be your prefered LB tool:
>> > > >> > > > >> > haproxy, traefik or nginx?
>> > > >> > > > >> >
>> > > >> > > > >> > CloudStack already have to code to handle SSL certs per
>> > > >> projects and
>> > > >> > > > >> > accounts if not mistaking because that code was added to
>> > > support
>> > > >> > > > >> NetScaler
>> > > >> > > > >> > as Load-balancer in the past. so one less thing to think
>> > > about
>> > > >> :-)
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> > [1] https://traefik.io/
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> > PL,
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro>
>> wrote:
>> > > >> > > > >> >
>> > > >> > > > >> > > Thanks Andrija,
>> > > >> > > > >> > >
>> > > >> > > > >> > > LB outside of the VR sounds like a good idea. An
>> appliance
>> > > >> based
>> > > >> > > on,
>> > > >> > > > >> say
>> > > >> > > > >> > > cloud-init + ansible and so on could do the trick; alas
>> > > it'd
>> > > >> need
>> > > >> > > to be
>> > > >> > > > >> > > outside ACS.
>> > > >> > > > >> > > I guess as users we could maybe come up with a spec
>> for an
>> > > >> > > > >> improvement, at
>> > > >> > > > >> > > least we'd have something the devs could look at
>> whenever
>> > > it
>> > > >> is
>> > > >> > > > >> possible.
>> > > >> > > > >> > >
>> > > >> > > > >> > > Regards,
>> > > >> > > > >> > > Lucian
>> > > >> > > > >> > >
>> > > >> > > > >> > > --
>> > > >> > > > >> > > Sent from the Delta quadrant using Borg technology!
>> > > >> > > > >> > >
>> > > >> > > > >> > > Nux!
>> > > >> > > > >> > > www.nux.ro
>> > > >> > > > >> > >
>> > > >> > > > >> > > ----- Original Message -----
>> > > >> > > > >> > > > From: "Andrija Panic" <an...@gmail.com>
>> > > >> > > > >> > > > To: "dev" <de...@cloudstack.apache.org>
>> > > >> > > > >> > > > Cc: "users" <us...@cloudstack.apache.org>
>> > > >> > > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
>> > > >> > > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > > >> > > > >> > >
>> > > >> > > > >> > > > We used to make some special stuff for one of the
>> > > clients,
>> > > >> > > where all
>> > > >> > > > >> LB
>> > > >> > > > >> > > > configuration work is done from outside of the ACS,
>> i.e.
>> > > >> python
>> > > >> > > > >> script to
>> > > >> > > > >> > > > feed/configure VR - install latest haproxy 1.5.x for
>> > > >> transparent
>> > > >> > > > >> proxy,
>> > > >> > > > >> > > > since client insisted on SSL termination done on
>> backend
>> > > >> web SSL
>> > > >> > > > >> > > servers....
>> > > >> > > > >> > > > Not good idea, that is all I can say (custom
>> > > configuration
>> > > >> > > thing) -
>> > > >> > > > >> but
>> > > >> > > > >> > > the
>> > > >> > > > >> > > > LB setup is actually good - transparent mode haproxy,
>> > > works
>> > > >> on
>> > > >> > > TCP
>> > > >> > > > >> level,
>> > > >> > > > >> > > > so you can see "real client IP" on the backend
>> servers
>> > > >> (which
>> > > >> > > must
>> > > >> > > > >> use VR
>> > > >> > > > >> > > > as the default gtw, as per default, so the whole
>> setup
>> > > works
>> > > >> > > > >> properly).
>> > > >> > > > >> > > >
>> > > >> > > > >> > > > I'm still looking forward to see some special
>> support of
>> > > LB
>> > > >> > > inside
>> > > >> > > > >> VR via
>> > > >> > > > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e.
>> to
>> > > >> enable LB
>> > > >> > > > >> > > > provisioning SCRIPT (bash, or whatever),  where all
>> > > needed
>> > > >> > > > >> > > > install+configure can be done from client side  -
>> > > otherwise
>> > > >> > > covering
>> > > >> > > > >> all
>> > > >> > > > >> > > > user cases, with proper HTTP checks and similar....is
>> > > >> > > impossible to
>> > > >> > > > >> do
>> > > >> > > > >> > > > IMHO.
>> > > >> > > > >> > > >
>> > > >> > > > >> > > > Some other clients, actually have internal FW
>> appliance
>> > > >> (i.e.
>> > > >> > > > >> multihomed
>> > > >> > > > >> > > > VM, acting as gtw for all VMs in all networks), and
>> > > haproxy
>> > > >> > > instaled
>> > > >> > > > >> on
>> > > >> > > > >> > > > this device (with NAT configured from VR to this
>> internal
>> > > >> > > FW/VM, so
>> > > >> > > > >> > > remote
>> > > >> > > > >> > > > IP can be seen properly) - this setup is fully under
>> > > >> customer
>> > > >> > > > >> control,
>> > > >> > > > >> > > and
>> > > >> > > > >> > > > can provide any kind of special haproxy config...
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro>
>> wrote:
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >> Hello,
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >> Of the people running an LB (VR) with https
>> backends,
>> > > how
>> > > >> do
>> > > >> > > you
>> > > >> > > > >> deal
>> > > >> > > > >> > > with
>> > > >> > > > >> > > >> the lack of x-forwarded-for since for port 443
>> there's
>> > > just
>> > > >> > > simple
>> > > >> > > > >> TCP
>> > > >> > > > >> > > >> balancing?
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >> Has anyone thought of terminating SSL in the VR
>> instead?
>> > > >> Ideas?
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >> Cheers
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >> --
>> > > >> > > > >> > > >> Sent from the Delta quadrant using Borg technology!
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >> Nux!
>> > > >> > > > >> > > >> www.nux.ro
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > > --
>> > > >> > > > >> > > >
>> > > >> > > > >> > > > Andrija Panić
>> > > >> > > > >> > >
>> > > >> > > > >>
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > > > --
>> > > >> > > > >
>> > > >> > > > > Andrija Panić
>> > > >> > >
>> > >

Re: HTTPS LB and x-forwarded-for

Posted by Nux! <nu...@li.nux.ro>.
lol.... good find..

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Pierre-Luc Dion" <pd...@cloudops.com>
> To: "Wido den Hollander" <wi...@widodh.nl>
> Cc: "dev" <de...@cloudstack.apache.org>, "users" <us...@cloudstack.apache.org>
> Sent: Friday, 17 November, 2017 13:56:11
> Subject: Re: HTTPS LB and x-forwarded-for

> That's funny!
> 
> Cloudstack ui does not provide lb protocol options, but the api does and
> cloudstack already support proxy proto v1!!!
> 
> So that's cool!
> 
> Le 13 nov. 2017 09 h 18, "Wido den Hollander" <wi...@widodh.nl> a écrit :
> 
>>
>> > Op 13 november 2017 om 15:14 schreef Pierre-Luc Dion <pdion@cloudops.com
>> >:
>> >
>> >
>> > Hi,
>> >
>> > This is looking quite promising, I have a colleague that tested that last
>> > Friday, look like the proxy proto v1 work out of the box with Nginx, but
>> > would need an extra package for Apache 2.4 ?
>>
>> It depends. You need HTTPd 2.4.28, see: https://httpd.apache.org/docs/
>> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
>>
>> It's there upstream, but not in all packages.
>>
>> It can from this module:
>>
>> - https://github.com/roadrunner2/mod-proxy-protocol
>> - https://roadrunner2.github.io/mod-proxy-protocol/mod_proxy_protocol.html
>>
>> They donated the code to go upstream and went into mod_remoteip but landed
>> in 2.4.28
>>
>> It will probably make it into Ubuntu 18.04 and CentOS 7.4.
>>
>> Wido
>>
>> > Otherwise, it seems to be a good way to do  https LB without complicated
>> > configuration and huge changes in CloudStack.
>> >
>> >
>> >
>> > On Fri, Nov 10, 2017 at 10:36 AM, Nux! <nu...@li.nux.ro> wrote:
>> >
>> > > Pierre-Luc,
>> > >
>> > > Haproxy docs say it should work for any kind of traffic as long as both
>> > > ends are PROXY-aware and it look like a majority of software is.
>> > > So, in short, yes.
>> > >
>> > > --
>> > > Sent from the Delta quadrant using Borg technology!
>> > >
>> > > Nux!
>> > > www.nux.ro
>> > >
>> > > ----- Original Message -----
>> > > > From: "Pierre-Luc Dion" <pd...@cloudops.com>
>> > > > To: "Wido den Hollander" <wi...@widodh.nl>
>> > > > Cc: "dev" <de...@cloudstack.apache.org>, "Khosrow Moossavi" <
>> > > kmoossavi@cloudops.com>, "Will Stevens"
>> > > > <ws...@cloudops.com>, "Nux!" <nu...@li.nux.ro>, "users" <
>> > > users@cloudstack.apache.org>
>> > > > Sent: Friday, 10 November, 2017 15:32:38
>> > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > >
>> > > > Hi Wido, do you know if this would work for https traffic too?
>> > > >
>> > > > Le 10 nov. 2017 09 h 35, "Wido den Hollander" <wi...@widodh.nl> a
>> écrit :
>> > > >
>> > > >>
>> > > >> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <
>> > > pdion@cloudops.com
>> > > >> >:
>> > > >> >
>> > > >> >
>> > > >> > I kind of like the proxy backend type, ill check on our end if
>> that
>> > > would
>> > > >> > work but definitely a simple and efficient approach!
>> > > >> >
>> > > >>
>> > > >> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
>> > > >>
>> > > >> Apache HTTPd supports PROXY since 2.4.28:
>> > > https://httpd.apache.org/docs/
>> > > >> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
>> > > >>
>> > > >> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
>> > > >>
>> > > >> Wido
>> > > >>
>> > > >> >
>> > > >> >
>> > > >> > Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wi...@widodh.nl> a
>> > > écrit :
>> > > >> >
>> > > >> > >
>> > > >> > > > Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
>> > > >> > > >
>> > > >> > > >
>> > > >> > > > Wido,
>> > > >> > > >
>> > > >> > > > Excellent suggestion with the "transparent proxy", I was not
>> > > aware of
>> > > >> > > that.
>> > > >> > > > I think that would be a great idea and wouldn't require too
>> many
>> > > >> > > modifications, especially as Haproxy comes already with the VR.
>> > > >> > > >
>> > > >> > >
>> > > >> > > It's indeed just a matter of a HAProxy config setting. We could
>> > > make it
>> > > >> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY
>> for
>> > > >> example.
>> > > >> > >
>> > > >> > > That way your problem would be solved.
>> > > >> > >
>> > > >> > > Wido
>> > > >> > >
>> > > >> > > > To Paul:
>> > > >> > > > - imho the LB solution ACS ships now is a bit handicaped since
>> > > you do
>> > > >> > > not know the remote host ip. You're flying blind unless you use
>> > > google
>> > > >> > > analytics (and these things have gotten more and more
>> aggressively
>> > > >> filtered
>> > > >> > > by adblocks).
>> > > >> > > > Enhancing Haproxy as Wido suggested would go a long way, it
>> > > wouldn't
>> > > >> > > break existing functionality and would also keep SSL processing
>> off
>> > > >> the VR.
>> > > >> > > >
>> > > >> > > > --
>> > > >> > > > Sent from the Delta quadrant using Borg technology!
>> > > >> > > >
>> > > >> > > > Nux!
>> > > >> > > > www.nux.ro
>> > > >> > > >
>> > > >> > > > ----- Original Message -----
>> > > >> > > > > From: "Andrija Panic" <an...@gmail.com>
>> > > >> > > > > To: "users" <us...@cloudstack.apache.org>
>> > > >> > > > > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will
>> > > Stevens" <
>> > > >> > > wstevens@cloudops.com>, "dev"
>> > > >> > > > > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <
>> > > pdion@cloudops.com
>> > > >> >
>> > > >> > > > > Sent: Thursday, 9 November, 2017 13:10:58
>> > > >> > > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > > >> > > >
>> > > >> > > > > Wido,
>> > > >> > > > >
>> > > >> > > > > backend servers are not Linux only, for example we have a
>> ton of
>> > > >> > > Windows
>> > > >> > > > > customers, some WEB solutions / IIS etc...
>> > > >> > > > >
>> > > >> > > > > @all - If we try to please/solve everyone's proxying
>> > > >> > > solution/requirement -
>> > > >> > > > > this is impossible IMHO - I'm thinking more about some "do
>> it as
>> > > >> you
>> > > >> > > like"
>> > > >> > > > > solution, to let customer write his own haproxy config and
>> > > upoad it
>> > > >> > > (for
>> > > >> > > > > example, or something better?).
>> > > >> > > > >
>> > > >> > > > > We can support newer version of haproxy (1.5+) which also
>> > > implement
>> > > >> > > > > "transarent proxy" (integrate with kernel so to speak)  to
>> allow
>> > > >> > > TCP-level
>> > > >> > > > > connections to backend (TCP mode, not HTTP mode) but to
>> still
>> > > >> > > "preserve"
>> > > >> > > > > remote IP by faking it (fake soruce IP = transarent proxy).
>> > > >> > > > >
>> > > >> > > > > For the rest of configuration options,  I would leave it to
>> the
>> > > >> > > customer
>> > > >> > > > > how he/she wants to configure rest of haproxy configuration,
>> > > >> inlcuding
>> > > >> > > > > custom checks, etc. Haproxy configuration is never-ending
>> story,
>> > > >> and we
>> > > >> > > > > probably should allow custom sripts/configuration instead of
>> > > >> trying to
>> > > >> > > > > provide GUI/API way to configure everything (which is
>> > > >> impossible...)
>> > > >> > > > >
>> > > >> > > > > Just my 2 cents...
>> > > >> > > > >
>> > > >> > > > > On 9 November 2017 at 08:13, Wido den Hollander <
>> wido@widodh.nl
>> > > >
>> > > >> > > wrote:
>> > > >> > > > >
>> > > >> > > > >>
>> > > >> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
>> > > >> > > pdion@cloudops.com
>> > > >> > > > >> >:
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> > Same challenge here too!
>> > > >> > > > >> >
>> > > >> > > > >> > Let's look at improving Load-balancing offering from
>> > > >> cloudstack, I
>> > > >> > > guest
>> > > >> > > > >> we
>> > > >> > > > >> > should do a feature spec draft soon..,  from my
>> perspective,
>> > > >> doing
>> > > >> > > SSL
>> > > >> > > > >> > offload on the VR could be problematic if the VR spec if
>> too
>> > > >> small,
>> > > >> > > and
>> > > >> > > > >> the
>> > > >> > > > >> > default spec of the VR being 1vcpu@256MB, considering
>> it can
>> > > >> be the
>> > > >> > > > >> router
>> > > >> > > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit
>> > > ish...
>> > > >> What
>> > > >> > > would
>> > > >> > > > >> > be your thought about this ?
>> > > >> > > > >> >
>> > > >> > > > >> > I'd be curious to have a LB offering in ACS where it
>> would
>> > > >> deploy a
>> > > >> > > > >> > redundant traefik[1] beside the VR for doing http and
>> https
>> > > >> > > > >> Load-balancing.
>> > > >> > > > >> > I think it would also be useful if the API of that
>> traefik
>> > > >> instance
>> > > >> > > would
>> > > >> > > > >> > be available from within the VPC or LBnetwork so is API
>> > > would be
>> > > >> > > > >> accessible
>> > > >> > > > >> > to other apps orchestration tools such as  kubernetes or
>> > > >> rancher.
>> > > >> > > > >> >
>> > > >> > > > >> > traefik or not, here is what I think is needed by
>> cloudstack
>> > > in
>> > > >> the
>> > > >> > > LB
>> > > >> > > > >> > improvement:
>> > > >> > > > >> >
>> > > >> > > > >> > - support http, https (X-Forwarded-For)
>> > > >> > > > >>
>> > > >> > > > >> HAProxy also supports the PROXY protocol towards the
>> backends.
>> > > >> Apache
>> > > >> > > > >> 2.4.22 supports this natively and Varnish for example can
>> also
>> > > >> talk
>> > > >> > > PROXY.
>> > > >> > > > >>
>> > > >> > > > >> It adds a littlebit of metadata to the connection so that
>> the
>> > > >> backend
>> > > >> > > > >> knows the original IP the connection came from for example:
>> > > >> > > > >> https://www.haproxy.org/download/1.8/doc/proxy-
>> protocol.txt
>> > > >> > > > >>
>> > > >> > > > >> Wido
>> > > >> > > > >>
>> > > >> > > > >> > - basic persistence tuning (API already exist)
>> > > >> > > > >> > - better backend monitoring, currently only a tcp connect
>> > > >> validate
>> > > >> > > if the
>> > > >> > > > >> > webserver is up.
>> > > >> > > > >> > - ssl offload
>> > > >> > > > >> > - metric collection, more stats, maybe just export the
>> tool
>> > > >> status
>> > > >> > > page
>> > > >> > > > >> to
>> > > >> > > > >> > the private network.
>> > > >> > > > >> > - Container world support, right now if you have Rancher
>> or
>> > > >> > > kubernetes
>> > > >> > > > >> > cluster, you need to deploy your own LB solution behing
>> > > >> mostlikely a
>> > > >> > > > >> static
>> > > >> > > > >> > nat., If cloudstack would deploy a traefik instance, Kub
>> or
>> > > >> Rancher
>> > > >> > > could
>> > > >> > > > >> > reuse this instance and managed it to properly do LB
>> between
>> > > >> > > containers.
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> > What would be your prefered LB tool:
>> > > >> > > > >> > haproxy, traefik or nginx?
>> > > >> > > > >> >
>> > > >> > > > >> > CloudStack already have to code to handle SSL certs per
>> > > >> projects and
>> > > >> > > > >> > accounts if not mistaking because that code was added to
>> > > support
>> > > >> > > > >> NetScaler
>> > > >> > > > >> > as Load-balancer in the past. so one less thing to think
>> > > about
>> > > >> :-)
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> > [1] https://traefik.io/
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> > PL,
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro>
>> wrote:
>> > > >> > > > >> >
>> > > >> > > > >> > > Thanks Andrija,
>> > > >> > > > >> > >
>> > > >> > > > >> > > LB outside of the VR sounds like a good idea. An
>> appliance
>> > > >> based
>> > > >> > > on,
>> > > >> > > > >> say
>> > > >> > > > >> > > cloud-init + ansible and so on could do the trick; alas
>> > > it'd
>> > > >> need
>> > > >> > > to be
>> > > >> > > > >> > > outside ACS.
>> > > >> > > > >> > > I guess as users we could maybe come up with a spec
>> for an
>> > > >> > > > >> improvement, at
>> > > >> > > > >> > > least we'd have something the devs could look at
>> whenever
>> > > it
>> > > >> is
>> > > >> > > > >> possible.
>> > > >> > > > >> > >
>> > > >> > > > >> > > Regards,
>> > > >> > > > >> > > Lucian
>> > > >> > > > >> > >
>> > > >> > > > >> > > --
>> > > >> > > > >> > > Sent from the Delta quadrant using Borg technology!
>> > > >> > > > >> > >
>> > > >> > > > >> > > Nux!
>> > > >> > > > >> > > www.nux.ro
>> > > >> > > > >> > >
>> > > >> > > > >> > > ----- Original Message -----
>> > > >> > > > >> > > > From: "Andrija Panic" <an...@gmail.com>
>> > > >> > > > >> > > > To: "dev" <de...@cloudstack.apache.org>
>> > > >> > > > >> > > > Cc: "users" <us...@cloudstack.apache.org>
>> > > >> > > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
>> > > >> > > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > > >> > > > >> > >
>> > > >> > > > >> > > > We used to make some special stuff for one of the
>> > > clients,
>> > > >> > > where all
>> > > >> > > > >> LB
>> > > >> > > > >> > > > configuration work is done from outside of the ACS,
>> i.e.
>> > > >> python
>> > > >> > > > >> script to
>> > > >> > > > >> > > > feed/configure VR - install latest haproxy 1.5.x for
>> > > >> transparent
>> > > >> > > > >> proxy,
>> > > >> > > > >> > > > since client insisted on SSL termination done on
>> backend
>> > > >> web SSL
>> > > >> > > > >> > > servers....
>> > > >> > > > >> > > > Not good idea, that is all I can say (custom
>> > > configuration
>> > > >> > > thing) -
>> > > >> > > > >> but
>> > > >> > > > >> > > the
>> > > >> > > > >> > > > LB setup is actually good - transparent mode haproxy,
>> > > works
>> > > >> on
>> > > >> > > TCP
>> > > >> > > > >> level,
>> > > >> > > > >> > > > so you can see "real client IP" on the backend
>> servers
>> > > >> (which
>> > > >> > > must
>> > > >> > > > >> use VR
>> > > >> > > > >> > > > as the default gtw, as per default, so the whole
>> setup
>> > > works
>> > > >> > > > >> properly).
>> > > >> > > > >> > > >
>> > > >> > > > >> > > > I'm still looking forward to see some special
>> support of
>> > > LB
>> > > >> > > inside
>> > > >> > > > >> VR via
>> > > >> > > > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e.
>> to
>> > > >> enable LB
>> > > >> > > > >> > > > provisioning SCRIPT (bash, or whatever),  where all
>> > > needed
>> > > >> > > > >> > > > install+configure can be done from client side  -
>> > > otherwise
>> > > >> > > covering
>> > > >> > > > >> all
>> > > >> > > > >> > > > user cases, with proper HTTP checks and similar....is
>> > > >> > > impossible to
>> > > >> > > > >> do
>> > > >> > > > >> > > > IMHO.
>> > > >> > > > >> > > >
>> > > >> > > > >> > > > Some other clients, actually have internal FW
>> appliance
>> > > >> (i.e.
>> > > >> > > > >> multihomed
>> > > >> > > > >> > > > VM, acting as gtw for all VMs in all networks), and
>> > > haproxy
>> > > >> > > instaled
>> > > >> > > > >> on
>> > > >> > > > >> > > > this device (with NAT configured from VR to this
>> internal
>> > > >> > > FW/VM, so
>> > > >> > > > >> > > remote
>> > > >> > > > >> > > > IP can be seen properly) - this setup is fully under
>> > > >> customer
>> > > >> > > > >> control,
>> > > >> > > > >> > > and
>> > > >> > > > >> > > > can provide any kind of special haproxy config...
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro>
>> wrote:
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >> Hello,
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >> Of the people running an LB (VR) with https
>> backends,
>> > > how
>> > > >> do
>> > > >> > > you
>> > > >> > > > >> deal
>> > > >> > > > >> > > with
>> > > >> > > > >> > > >> the lack of x-forwarded-for since for port 443
>> there's
>> > > just
>> > > >> > > simple
>> > > >> > > > >> TCP
>> > > >> > > > >> > > >> balancing?
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >> Has anyone thought of terminating SSL in the VR
>> instead?
>> > > >> Ideas?
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >> Cheers
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >> --
>> > > >> > > > >> > > >> Sent from the Delta quadrant using Borg technology!
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >> Nux!
>> > > >> > > > >> > > >> www.nux.ro
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > > --
>> > > >> > > > >> > > >
>> > > >> > > > >> > > > Andrija Panić
>> > > >> > > > >> > >
>> > > >> > > > >>
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > > > --
>> > > >> > > > >
>> > > >> > > > > Andrija Panić
>> > > >> > >
>> > >

Re: HTTPS LB and x-forwarded-for

Posted by Pierre-Luc Dion <pd...@cloudops.com>.
That's funny!

Cloudstack ui does not provide lb protocol options, but the api does and
cloudstack already support proxy proto v1!!!

So that's cool!

Le 13 nov. 2017 09 h 18, "Wido den Hollander" <wi...@widodh.nl> a écrit :

>
> > Op 13 november 2017 om 15:14 schreef Pierre-Luc Dion <pdion@cloudops.com
> >:
> >
> >
> > Hi,
> >
> > This is looking quite promising, I have a colleague that tested that last
> > Friday, look like the proxy proto v1 work out of the box with Nginx, but
> > would need an extra package for Apache 2.4 ?
>
> It depends. You need HTTPd 2.4.28, see: https://httpd.apache.org/docs/
> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
>
> It's there upstream, but not in all packages.
>
> It can from this module:
>
> - https://github.com/roadrunner2/mod-proxy-protocol
> - https://roadrunner2.github.io/mod-proxy-protocol/mod_proxy_protocol.html
>
> They donated the code to go upstream and went into mod_remoteip but landed
> in 2.4.28
>
> It will probably make it into Ubuntu 18.04 and CentOS 7.4.
>
> Wido
>
> > Otherwise, it seems to be a good way to do  https LB without complicated
> > configuration and huge changes in CloudStack.
> >
> >
> >
> > On Fri, Nov 10, 2017 at 10:36 AM, Nux! <nu...@li.nux.ro> wrote:
> >
> > > Pierre-Luc,
> > >
> > > Haproxy docs say it should work for any kind of traffic as long as both
> > > ends are PROXY-aware and it look like a majority of software is.
> > > So, in short, yes.
> > >
> > > --
> > > Sent from the Delta quadrant using Borg technology!
> > >
> > > Nux!
> > > www.nux.ro
> > >
> > > ----- Original Message -----
> > > > From: "Pierre-Luc Dion" <pd...@cloudops.com>
> > > > To: "Wido den Hollander" <wi...@widodh.nl>
> > > > Cc: "dev" <de...@cloudstack.apache.org>, "Khosrow Moossavi" <
> > > kmoossavi@cloudops.com>, "Will Stevens"
> > > > <ws...@cloudops.com>, "Nux!" <nu...@li.nux.ro>, "users" <
> > > users@cloudstack.apache.org>
> > > > Sent: Friday, 10 November, 2017 15:32:38
> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > >
> > > > Hi Wido, do you know if this would work for https traffic too?
> > > >
> > > > Le 10 nov. 2017 09 h 35, "Wido den Hollander" <wi...@widodh.nl> a
> écrit :
> > > >
> > > >>
> > > >> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <
> > > pdion@cloudops.com
> > > >> >:
> > > >> >
> > > >> >
> > > >> > I kind of like the proxy backend type, ill check on our end if
> that
> > > would
> > > >> > work but definitely a simple and efficient approach!
> > > >> >
> > > >>
> > > >> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
> > > >>
> > > >> Apache HTTPd supports PROXY since 2.4.28:
> > > https://httpd.apache.org/docs/
> > > >> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
> > > >>
> > > >> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
> > > >>
> > > >> Wido
> > > >>
> > > >> >
> > > >> >
> > > >> > Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wi...@widodh.nl> a
> > > écrit :
> > > >> >
> > > >> > >
> > > >> > > > Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
> > > >> > > >
> > > >> > > >
> > > >> > > > Wido,
> > > >> > > >
> > > >> > > > Excellent suggestion with the "transparent proxy", I was not
> > > aware of
> > > >> > > that.
> > > >> > > > I think that would be a great idea and wouldn't require too
> many
> > > >> > > modifications, especially as Haproxy comes already with the VR.
> > > >> > > >
> > > >> > >
> > > >> > > It's indeed just a matter of a HAProxy config setting. We could
> > > make it
> > > >> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY
> for
> > > >> example.
> > > >> > >
> > > >> > > That way your problem would be solved.
> > > >> > >
> > > >> > > Wido
> > > >> > >
> > > >> > > > To Paul:
> > > >> > > > - imho the LB solution ACS ships now is a bit handicaped since
> > > you do
> > > >> > > not know the remote host ip. You're flying blind unless you use
> > > google
> > > >> > > analytics (and these things have gotten more and more
> aggressively
> > > >> filtered
> > > >> > > by adblocks).
> > > >> > > > Enhancing Haproxy as Wido suggested would go a long way, it
> > > wouldn't
> > > >> > > break existing functionality and would also keep SSL processing
> off
> > > >> the VR.
> > > >> > > >
> > > >> > > > --
> > > >> > > > Sent from the Delta quadrant using Borg technology!
> > > >> > > >
> > > >> > > > Nux!
> > > >> > > > www.nux.ro
> > > >> > > >
> > > >> > > > ----- Original Message -----
> > > >> > > > > From: "Andrija Panic" <an...@gmail.com>
> > > >> > > > > To: "users" <us...@cloudstack.apache.org>
> > > >> > > > > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will
> > > Stevens" <
> > > >> > > wstevens@cloudops.com>, "dev"
> > > >> > > > > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <
> > > pdion@cloudops.com
> > > >> >
> > > >> > > > > Sent: Thursday, 9 November, 2017 13:10:58
> > > >> > > > > Subject: Re: HTTPS LB and x-forwarded-for
> > > >> > > >
> > > >> > > > > Wido,
> > > >> > > > >
> > > >> > > > > backend servers are not Linux only, for example we have a
> ton of
> > > >> > > Windows
> > > >> > > > > customers, some WEB solutions / IIS etc...
> > > >> > > > >
> > > >> > > > > @all - If we try to please/solve everyone's proxying
> > > >> > > solution/requirement -
> > > >> > > > > this is impossible IMHO - I'm thinking more about some "do
> it as
> > > >> you
> > > >> > > like"
> > > >> > > > > solution, to let customer write his own haproxy config and
> > > upoad it
> > > >> > > (for
> > > >> > > > > example, or something better?).
> > > >> > > > >
> > > >> > > > > We can support newer version of haproxy (1.5+) which also
> > > implement
> > > >> > > > > "transarent proxy" (integrate with kernel so to speak)  to
> allow
> > > >> > > TCP-level
> > > >> > > > > connections to backend (TCP mode, not HTTP mode) but to
> still
> > > >> > > "preserve"
> > > >> > > > > remote IP by faking it (fake soruce IP = transarent proxy).
> > > >> > > > >
> > > >> > > > > For the rest of configuration options,  I would leave it to
> the
> > > >> > > customer
> > > >> > > > > how he/she wants to configure rest of haproxy configuration,
> > > >> inlcuding
> > > >> > > > > custom checks, etc. Haproxy configuration is never-ending
> story,
> > > >> and we
> > > >> > > > > probably should allow custom sripts/configuration instead of
> > > >> trying to
> > > >> > > > > provide GUI/API way to configure everything (which is
> > > >> impossible...)
> > > >> > > > >
> > > >> > > > > Just my 2 cents...
> > > >> > > > >
> > > >> > > > > On 9 November 2017 at 08:13, Wido den Hollander <
> wido@widodh.nl
> > > >
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > >>
> > > >> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
> > > >> > > pdion@cloudops.com
> > > >> > > > >> >:
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> > Same challenge here too!
> > > >> > > > >> >
> > > >> > > > >> > Let's look at improving Load-balancing offering from
> > > >> cloudstack, I
> > > >> > > guest
> > > >> > > > >> we
> > > >> > > > >> > should do a feature spec draft soon..,  from my
> perspective,
> > > >> doing
> > > >> > > SSL
> > > >> > > > >> > offload on the VR could be problematic if the VR spec if
> too
> > > >> small,
> > > >> > > and
> > > >> > > > >> the
> > > >> > > > >> > default spec of the VR being 1vcpu@256MB, considering
> it can
> > > >> be the
> > > >> > > > >> router
> > > >> > > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit
> > > ish...
> > > >> What
> > > >> > > would
> > > >> > > > >> > be your thought about this ?
> > > >> > > > >> >
> > > >> > > > >> > I'd be curious to have a LB offering in ACS where it
> would
> > > >> deploy a
> > > >> > > > >> > redundant traefik[1] beside the VR for doing http and
> https
> > > >> > > > >> Load-balancing.
> > > >> > > > >> > I think it would also be useful if the API of that
> traefik
> > > >> instance
> > > >> > > would
> > > >> > > > >> > be available from within the VPC or LBnetwork so is API
> > > would be
> > > >> > > > >> accessible
> > > >> > > > >> > to other apps orchestration tools such as  kubernetes or
> > > >> rancher.
> > > >> > > > >> >
> > > >> > > > >> > traefik or not, here is what I think is needed by
> cloudstack
> > > in
> > > >> the
> > > >> > > LB
> > > >> > > > >> > improvement:
> > > >> > > > >> >
> > > >> > > > >> > - support http, https (X-Forwarded-For)
> > > >> > > > >>
> > > >> > > > >> HAProxy also supports the PROXY protocol towards the
> backends.
> > > >> Apache
> > > >> > > > >> 2.4.22 supports this natively and Varnish for example can
> also
> > > >> talk
> > > >> > > PROXY.
> > > >> > > > >>
> > > >> > > > >> It adds a littlebit of metadata to the connection so that
> the
> > > >> backend
> > > >> > > > >> knows the original IP the connection came from for example:
> > > >> > > > >> https://www.haproxy.org/download/1.8/doc/proxy-
> protocol.txt
> > > >> > > > >>
> > > >> > > > >> Wido
> > > >> > > > >>
> > > >> > > > >> > - basic persistence tuning (API already exist)
> > > >> > > > >> > - better backend monitoring, currently only a tcp connect
> > > >> validate
> > > >> > > if the
> > > >> > > > >> > webserver is up.
> > > >> > > > >> > - ssl offload
> > > >> > > > >> > - metric collection, more stats, maybe just export the
> tool
> > > >> status
> > > >> > > page
> > > >> > > > >> to
> > > >> > > > >> > the private network.
> > > >> > > > >> > - Container world support, right now if you have Rancher
> or
> > > >> > > kubernetes
> > > >> > > > >> > cluster, you need to deploy your own LB solution behing
> > > >> mostlikely a
> > > >> > > > >> static
> > > >> > > > >> > nat., If cloudstack would deploy a traefik instance, Kub
> or
> > > >> Rancher
> > > >> > > could
> > > >> > > > >> > reuse this instance and managed it to properly do LB
> between
> > > >> > > containers.
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> > What would be your prefered LB tool:
> > > >> > > > >> > haproxy, traefik or nginx?
> > > >> > > > >> >
> > > >> > > > >> > CloudStack already have to code to handle SSL certs per
> > > >> projects and
> > > >> > > > >> > accounts if not mistaking because that code was added to
> > > support
> > > >> > > > >> NetScaler
> > > >> > > > >> > as Load-balancer in the past. so one less thing to think
> > > about
> > > >> :-)
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> > [1] https://traefik.io/
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> > PL,
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro>
> wrote:
> > > >> > > > >> >
> > > >> > > > >> > > Thanks Andrija,
> > > >> > > > >> > >
> > > >> > > > >> > > LB outside of the VR sounds like a good idea. An
> appliance
> > > >> based
> > > >> > > on,
> > > >> > > > >> say
> > > >> > > > >> > > cloud-init + ansible and so on could do the trick; alas
> > > it'd
> > > >> need
> > > >> > > to be
> > > >> > > > >> > > outside ACS.
> > > >> > > > >> > > I guess as users we could maybe come up with a spec
> for an
> > > >> > > > >> improvement, at
> > > >> > > > >> > > least we'd have something the devs could look at
> whenever
> > > it
> > > >> is
> > > >> > > > >> possible.
> > > >> > > > >> > >
> > > >> > > > >> > > Regards,
> > > >> > > > >> > > Lucian
> > > >> > > > >> > >
> > > >> > > > >> > > --
> > > >> > > > >> > > Sent from the Delta quadrant using Borg technology!
> > > >> > > > >> > >
> > > >> > > > >> > > Nux!
> > > >> > > > >> > > www.nux.ro
> > > >> > > > >> > >
> > > >> > > > >> > > ----- Original Message -----
> > > >> > > > >> > > > From: "Andrija Panic" <an...@gmail.com>
> > > >> > > > >> > > > To: "dev" <de...@cloudstack.apache.org>
> > > >> > > > >> > > > Cc: "users" <us...@cloudstack.apache.org>
> > > >> > > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
> > > >> > > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > > >> > > > >> > >
> > > >> > > > >> > > > We used to make some special stuff for one of the
> > > clients,
> > > >> > > where all
> > > >> > > > >> LB
> > > >> > > > >> > > > configuration work is done from outside of the ACS,
> i.e.
> > > >> python
> > > >> > > > >> script to
> > > >> > > > >> > > > feed/configure VR - install latest haproxy 1.5.x for
> > > >> transparent
> > > >> > > > >> proxy,
> > > >> > > > >> > > > since client insisted on SSL termination done on
> backend
> > > >> web SSL
> > > >> > > > >> > > servers....
> > > >> > > > >> > > > Not good idea, that is all I can say (custom
> > > configuration
> > > >> > > thing) -
> > > >> > > > >> but
> > > >> > > > >> > > the
> > > >> > > > >> > > > LB setup is actually good - transparent mode haproxy,
> > > works
> > > >> on
> > > >> > > TCP
> > > >> > > > >> level,
> > > >> > > > >> > > > so you can see "real client IP" on the backend
> servers
> > > >> (which
> > > >> > > must
> > > >> > > > >> use VR
> > > >> > > > >> > > > as the default gtw, as per default, so the whole
> setup
> > > works
> > > >> > > > >> properly).
> > > >> > > > >> > > >
> > > >> > > > >> > > > I'm still looking forward to see some special
> support of
> > > LB
> > > >> > > inside
> > > >> > > > >> VR via
> > > >> > > > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e.
> to
> > > >> enable LB
> > > >> > > > >> > > > provisioning SCRIPT (bash, or whatever),  where all
> > > needed
> > > >> > > > >> > > > install+configure can be done from client side  -
> > > otherwise
> > > >> > > covering
> > > >> > > > >> all
> > > >> > > > >> > > > user cases, with proper HTTP checks and similar....is
> > > >> > > impossible to
> > > >> > > > >> do
> > > >> > > > >> > > > IMHO.
> > > >> > > > >> > > >
> > > >> > > > >> > > > Some other clients, actually have internal FW
> appliance
> > > >> (i.e.
> > > >> > > > >> multihomed
> > > >> > > > >> > > > VM, acting as gtw for all VMs in all networks), and
> > > haproxy
> > > >> > > instaled
> > > >> > > > >> on
> > > >> > > > >> > > > this device (with NAT configured from VR to this
> internal
> > > >> > > FW/VM, so
> > > >> > > > >> > > remote
> > > >> > > > >> > > > IP can be seen properly) - this setup is fully under
> > > >> customer
> > > >> > > > >> control,
> > > >> > > > >> > > and
> > > >> > > > >> > > > can provide any kind of special haproxy config...
> > > >> > > > >> > > >
> > > >> > > > >> > > >
> > > >> > > > >> > > >
> > > >> > > > >> > > >
> > > >> > > > >> > > >
> > > >> > > > >> > > >
> > > >> > > > >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro>
> wrote:
> > > >> > > > >> > > >
> > > >> > > > >> > > >> Hello,
> > > >> > > > >> > > >>
> > > >> > > > >> > > >> Of the people running an LB (VR) with https
> backends,
> > > how
> > > >> do
> > > >> > > you
> > > >> > > > >> deal
> > > >> > > > >> > > with
> > > >> > > > >> > > >> the lack of x-forwarded-for since for port 443
> there's
> > > just
> > > >> > > simple
> > > >> > > > >> TCP
> > > >> > > > >> > > >> balancing?
> > > >> > > > >> > > >>
> > > >> > > > >> > > >> Has anyone thought of terminating SSL in the VR
> instead?
> > > >> Ideas?
> > > >> > > > >> > > >>
> > > >> > > > >> > > >> Cheers
> > > >> > > > >> > > >>
> > > >> > > > >> > > >> --
> > > >> > > > >> > > >> Sent from the Delta quadrant using Borg technology!
> > > >> > > > >> > > >>
> > > >> > > > >> > > >> Nux!
> > > >> > > > >> > > >> www.nux.ro
> > > >> > > > >> > > >>
> > > >> > > > >> > > >
> > > >> > > > >> > > >
> > > >> > > > >> > > >
> > > >> > > > >> > > > --
> > > >> > > > >> > > >
> > > >> > > > >> > > > Andrija Panić
> > > >> > > > >> > >
> > > >> > > > >>
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > --
> > > >> > > > >
> > > >> > > > > Andrija Panić
> > > >> > >
> > >
>

Re: HTTPS LB and x-forwarded-for

Posted by Pierre-Luc Dion <pd...@cloudops.com>.
That's funny!

Cloudstack ui does not provide lb protocol options, but the api does and
cloudstack already support proxy proto v1!!!

So that's cool!

Le 13 nov. 2017 09 h 18, "Wido den Hollander" <wi...@widodh.nl> a écrit :

>
> > Op 13 november 2017 om 15:14 schreef Pierre-Luc Dion <pdion@cloudops.com
> >:
> >
> >
> > Hi,
> >
> > This is looking quite promising, I have a colleague that tested that last
> > Friday, look like the proxy proto v1 work out of the box with Nginx, but
> > would need an extra package for Apache 2.4 ?
>
> It depends. You need HTTPd 2.4.28, see: https://httpd.apache.org/docs/
> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
>
> It's there upstream, but not in all packages.
>
> It can from this module:
>
> - https://github.com/roadrunner2/mod-proxy-protocol
> - https://roadrunner2.github.io/mod-proxy-protocol/mod_proxy_protocol.html
>
> They donated the code to go upstream and went into mod_remoteip but landed
> in 2.4.28
>
> It will probably make it into Ubuntu 18.04 and CentOS 7.4.
>
> Wido
>
> > Otherwise, it seems to be a good way to do  https LB without complicated
> > configuration and huge changes in CloudStack.
> >
> >
> >
> > On Fri, Nov 10, 2017 at 10:36 AM, Nux! <nu...@li.nux.ro> wrote:
> >
> > > Pierre-Luc,
> > >
> > > Haproxy docs say it should work for any kind of traffic as long as both
> > > ends are PROXY-aware and it look like a majority of software is.
> > > So, in short, yes.
> > >
> > > --
> > > Sent from the Delta quadrant using Borg technology!
> > >
> > > Nux!
> > > www.nux.ro
> > >
> > > ----- Original Message -----
> > > > From: "Pierre-Luc Dion" <pd...@cloudops.com>
> > > > To: "Wido den Hollander" <wi...@widodh.nl>
> > > > Cc: "dev" <de...@cloudstack.apache.org>, "Khosrow Moossavi" <
> > > kmoossavi@cloudops.com>, "Will Stevens"
> > > > <ws...@cloudops.com>, "Nux!" <nu...@li.nux.ro>, "users" <
> > > users@cloudstack.apache.org>
> > > > Sent: Friday, 10 November, 2017 15:32:38
> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > >
> > > > Hi Wido, do you know if this would work for https traffic too?
> > > >
> > > > Le 10 nov. 2017 09 h 35, "Wido den Hollander" <wi...@widodh.nl> a
> écrit :
> > > >
> > > >>
> > > >> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <
> > > pdion@cloudops.com
> > > >> >:
> > > >> >
> > > >> >
> > > >> > I kind of like the proxy backend type, ill check on our end if
> that
> > > would
> > > >> > work but definitely a simple and efficient approach!
> > > >> >
> > > >>
> > > >> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
> > > >>
> > > >> Apache HTTPd supports PROXY since 2.4.28:
> > > https://httpd.apache.org/docs/
> > > >> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
> > > >>
> > > >> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
> > > >>
> > > >> Wido
> > > >>
> > > >> >
> > > >> >
> > > >> > Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wi...@widodh.nl> a
> > > écrit :
> > > >> >
> > > >> > >
> > > >> > > > Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
> > > >> > > >
> > > >> > > >
> > > >> > > > Wido,
> > > >> > > >
> > > >> > > > Excellent suggestion with the "transparent proxy", I was not
> > > aware of
> > > >> > > that.
> > > >> > > > I think that would be a great idea and wouldn't require too
> many
> > > >> > > modifications, especially as Haproxy comes already with the VR.
> > > >> > > >
> > > >> > >
> > > >> > > It's indeed just a matter of a HAProxy config setting. We could
> > > make it
> > > >> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY
> for
> > > >> example.
> > > >> > >
> > > >> > > That way your problem would be solved.
> > > >> > >
> > > >> > > Wido
> > > >> > >
> > > >> > > > To Paul:
> > > >> > > > - imho the LB solution ACS ships now is a bit handicaped since
> > > you do
> > > >> > > not know the remote host ip. You're flying blind unless you use
> > > google
> > > >> > > analytics (and these things have gotten more and more
> aggressively
> > > >> filtered
> > > >> > > by adblocks).
> > > >> > > > Enhancing Haproxy as Wido suggested would go a long way, it
> > > wouldn't
> > > >> > > break existing functionality and would also keep SSL processing
> off
> > > >> the VR.
> > > >> > > >
> > > >> > > > --
> > > >> > > > Sent from the Delta quadrant using Borg technology!
> > > >> > > >
> > > >> > > > Nux!
> > > >> > > > www.nux.ro
> > > >> > > >
> > > >> > > > ----- Original Message -----
> > > >> > > > > From: "Andrija Panic" <an...@gmail.com>
> > > >> > > > > To: "users" <us...@cloudstack.apache.org>
> > > >> > > > > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will
> > > Stevens" <
> > > >> > > wstevens@cloudops.com>, "dev"
> > > >> > > > > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <
> > > pdion@cloudops.com
> > > >> >
> > > >> > > > > Sent: Thursday, 9 November, 2017 13:10:58
> > > >> > > > > Subject: Re: HTTPS LB and x-forwarded-for
> > > >> > > >
> > > >> > > > > Wido,
> > > >> > > > >
> > > >> > > > > backend servers are not Linux only, for example we have a
> ton of
> > > >> > > Windows
> > > >> > > > > customers, some WEB solutions / IIS etc...
> > > >> > > > >
> > > >> > > > > @all - If we try to please/solve everyone's proxying
> > > >> > > solution/requirement -
> > > >> > > > > this is impossible IMHO - I'm thinking more about some "do
> it as
> > > >> you
> > > >> > > like"
> > > >> > > > > solution, to let customer write his own haproxy config and
> > > upoad it
> > > >> > > (for
> > > >> > > > > example, or something better?).
> > > >> > > > >
> > > >> > > > > We can support newer version of haproxy (1.5+) which also
> > > implement
> > > >> > > > > "transarent proxy" (integrate with kernel so to speak)  to
> allow
> > > >> > > TCP-level
> > > >> > > > > connections to backend (TCP mode, not HTTP mode) but to
> still
> > > >> > > "preserve"
> > > >> > > > > remote IP by faking it (fake soruce IP = transarent proxy).
> > > >> > > > >
> > > >> > > > > For the rest of configuration options,  I would leave it to
> the
> > > >> > > customer
> > > >> > > > > how he/she wants to configure rest of haproxy configuration,
> > > >> inlcuding
> > > >> > > > > custom checks, etc. Haproxy configuration is never-ending
> story,
> > > >> and we
> > > >> > > > > probably should allow custom sripts/configuration instead of
> > > >> trying to
> > > >> > > > > provide GUI/API way to configure everything (which is
> > > >> impossible...)
> > > >> > > > >
> > > >> > > > > Just my 2 cents...
> > > >> > > > >
> > > >> > > > > On 9 November 2017 at 08:13, Wido den Hollander <
> wido@widodh.nl
> > > >
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > >>
> > > >> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
> > > >> > > pdion@cloudops.com
> > > >> > > > >> >:
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> > Same challenge here too!
> > > >> > > > >> >
> > > >> > > > >> > Let's look at improving Load-balancing offering from
> > > >> cloudstack, I
> > > >> > > guest
> > > >> > > > >> we
> > > >> > > > >> > should do a feature spec draft soon..,  from my
> perspective,
> > > >> doing
> > > >> > > SSL
> > > >> > > > >> > offload on the VR could be problematic if the VR spec if
> too
> > > >> small,
> > > >> > > and
> > > >> > > > >> the
> > > >> > > > >> > default spec of the VR being 1vcpu@256MB, considering
> it can
> > > >> be the
> > > >> > > > >> router
> > > >> > > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit
> > > ish...
> > > >> What
> > > >> > > would
> > > >> > > > >> > be your thought about this ?
> > > >> > > > >> >
> > > >> > > > >> > I'd be curious to have a LB offering in ACS where it
> would
> > > >> deploy a
> > > >> > > > >> > redundant traefik[1] beside the VR for doing http and
> https
> > > >> > > > >> Load-balancing.
> > > >> > > > >> > I think it would also be useful if the API of that
> traefik
> > > >> instance
> > > >> > > would
> > > >> > > > >> > be available from within the VPC or LBnetwork so is API
> > > would be
> > > >> > > > >> accessible
> > > >> > > > >> > to other apps orchestration tools such as  kubernetes or
> > > >> rancher.
> > > >> > > > >> >
> > > >> > > > >> > traefik or not, here is what I think is needed by
> cloudstack
> > > in
> > > >> the
> > > >> > > LB
> > > >> > > > >> > improvement:
> > > >> > > > >> >
> > > >> > > > >> > - support http, https (X-Forwarded-For)
> > > >> > > > >>
> > > >> > > > >> HAProxy also supports the PROXY protocol towards the
> backends.
> > > >> Apache
> > > >> > > > >> 2.4.22 supports this natively and Varnish for example can
> also
> > > >> talk
> > > >> > > PROXY.
> > > >> > > > >>
> > > >> > > > >> It adds a littlebit of metadata to the connection so that
> the
> > > >> backend
> > > >> > > > >> knows the original IP the connection came from for example:
> > > >> > > > >> https://www.haproxy.org/download/1.8/doc/proxy-
> protocol.txt
> > > >> > > > >>
> > > >> > > > >> Wido
> > > >> > > > >>
> > > >> > > > >> > - basic persistence tuning (API already exist)
> > > >> > > > >> > - better backend monitoring, currently only a tcp connect
> > > >> validate
> > > >> > > if the
> > > >> > > > >> > webserver is up.
> > > >> > > > >> > - ssl offload
> > > >> > > > >> > - metric collection, more stats, maybe just export the
> tool
> > > >> status
> > > >> > > page
> > > >> > > > >> to
> > > >> > > > >> > the private network.
> > > >> > > > >> > - Container world support, right now if you have Rancher
> or
> > > >> > > kubernetes
> > > >> > > > >> > cluster, you need to deploy your own LB solution behing
> > > >> mostlikely a
> > > >> > > > >> static
> > > >> > > > >> > nat., If cloudstack would deploy a traefik instance, Kub
> or
> > > >> Rancher
> > > >> > > could
> > > >> > > > >> > reuse this instance and managed it to properly do LB
> between
> > > >> > > containers.
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> > What would be your prefered LB tool:
> > > >> > > > >> > haproxy, traefik or nginx?
> > > >> > > > >> >
> > > >> > > > >> > CloudStack already have to code to handle SSL certs per
> > > >> projects and
> > > >> > > > >> > accounts if not mistaking because that code was added to
> > > support
> > > >> > > > >> NetScaler
> > > >> > > > >> > as Load-balancer in the past. so one less thing to think
> > > about
> > > >> :-)
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> > [1] https://traefik.io/
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> > PL,
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro>
> wrote:
> > > >> > > > >> >
> > > >> > > > >> > > Thanks Andrija,
> > > >> > > > >> > >
> > > >> > > > >> > > LB outside of the VR sounds like a good idea. An
> appliance
> > > >> based
> > > >> > > on,
> > > >> > > > >> say
> > > >> > > > >> > > cloud-init + ansible and so on could do the trick; alas
> > > it'd
> > > >> need
> > > >> > > to be
> > > >> > > > >> > > outside ACS.
> > > >> > > > >> > > I guess as users we could maybe come up with a spec
> for an
> > > >> > > > >> improvement, at
> > > >> > > > >> > > least we'd have something the devs could look at
> whenever
> > > it
> > > >> is
> > > >> > > > >> possible.
> > > >> > > > >> > >
> > > >> > > > >> > > Regards,
> > > >> > > > >> > > Lucian
> > > >> > > > >> > >
> > > >> > > > >> > > --
> > > >> > > > >> > > Sent from the Delta quadrant using Borg technology!
> > > >> > > > >> > >
> > > >> > > > >> > > Nux!
> > > >> > > > >> > > www.nux.ro
> > > >> > > > >> > >
> > > >> > > > >> > > ----- Original Message -----
> > > >> > > > >> > > > From: "Andrija Panic" <an...@gmail.com>
> > > >> > > > >> > > > To: "dev" <de...@cloudstack.apache.org>
> > > >> > > > >> > > > Cc: "users" <us...@cloudstack.apache.org>
> > > >> > > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
> > > >> > > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > > >> > > > >> > >
> > > >> > > > >> > > > We used to make some special stuff for one of the
> > > clients,
> > > >> > > where all
> > > >> > > > >> LB
> > > >> > > > >> > > > configuration work is done from outside of the ACS,
> i.e.
> > > >> python
> > > >> > > > >> script to
> > > >> > > > >> > > > feed/configure VR - install latest haproxy 1.5.x for
> > > >> transparent
> > > >> > > > >> proxy,
> > > >> > > > >> > > > since client insisted on SSL termination done on
> backend
> > > >> web SSL
> > > >> > > > >> > > servers....
> > > >> > > > >> > > > Not good idea, that is all I can say (custom
> > > configuration
> > > >> > > thing) -
> > > >> > > > >> but
> > > >> > > > >> > > the
> > > >> > > > >> > > > LB setup is actually good - transparent mode haproxy,
> > > works
> > > >> on
> > > >> > > TCP
> > > >> > > > >> level,
> > > >> > > > >> > > > so you can see "real client IP" on the backend
> servers
> > > >> (which
> > > >> > > must
> > > >> > > > >> use VR
> > > >> > > > >> > > > as the default gtw, as per default, so the whole
> setup
> > > works
> > > >> > > > >> properly).
> > > >> > > > >> > > >
> > > >> > > > >> > > > I'm still looking forward to see some special
> support of
> > > LB
> > > >> > > inside
> > > >> > > > >> VR via
> > > >> > > > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e.
> to
> > > >> enable LB
> > > >> > > > >> > > > provisioning SCRIPT (bash, or whatever),  where all
> > > needed
> > > >> > > > >> > > > install+configure can be done from client side  -
> > > otherwise
> > > >> > > covering
> > > >> > > > >> all
> > > >> > > > >> > > > user cases, with proper HTTP checks and similar....is
> > > >> > > impossible to
> > > >> > > > >> do
> > > >> > > > >> > > > IMHO.
> > > >> > > > >> > > >
> > > >> > > > >> > > > Some other clients, actually have internal FW
> appliance
> > > >> (i.e.
> > > >> > > > >> multihomed
> > > >> > > > >> > > > VM, acting as gtw for all VMs in all networks), and
> > > haproxy
> > > >> > > instaled
> > > >> > > > >> on
> > > >> > > > >> > > > this device (with NAT configured from VR to this
> internal
> > > >> > > FW/VM, so
> > > >> > > > >> > > remote
> > > >> > > > >> > > > IP can be seen properly) - this setup is fully under
> > > >> customer
> > > >> > > > >> control,
> > > >> > > > >> > > and
> > > >> > > > >> > > > can provide any kind of special haproxy config...
> > > >> > > > >> > > >
> > > >> > > > >> > > >
> > > >> > > > >> > > >
> > > >> > > > >> > > >
> > > >> > > > >> > > >
> > > >> > > > >> > > >
> > > >> > > > >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro>
> wrote:
> > > >> > > > >> > > >
> > > >> > > > >> > > >> Hello,
> > > >> > > > >> > > >>
> > > >> > > > >> > > >> Of the people running an LB (VR) with https
> backends,
> > > how
> > > >> do
> > > >> > > you
> > > >> > > > >> deal
> > > >> > > > >> > > with
> > > >> > > > >> > > >> the lack of x-forwarded-for since for port 443
> there's
> > > just
> > > >> > > simple
> > > >> > > > >> TCP
> > > >> > > > >> > > >> balancing?
> > > >> > > > >> > > >>
> > > >> > > > >> > > >> Has anyone thought of terminating SSL in the VR
> instead?
> > > >> Ideas?
> > > >> > > > >> > > >>
> > > >> > > > >> > > >> Cheers
> > > >> > > > >> > > >>
> > > >> > > > >> > > >> --
> > > >> > > > >> > > >> Sent from the Delta quadrant using Borg technology!
> > > >> > > > >> > > >>
> > > >> > > > >> > > >> Nux!
> > > >> > > > >> > > >> www.nux.ro
> > > >> > > > >> > > >>
> > > >> > > > >> > > >
> > > >> > > > >> > > >
> > > >> > > > >> > > >
> > > >> > > > >> > > > --
> > > >> > > > >> > > >
> > > >> > > > >> > > > Andrija Panić
> > > >> > > > >> > >
> > > >> > > > >>
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > --
> > > >> > > > >
> > > >> > > > > Andrija Panić
> > > >> > >
> > >
>

Re: HTTPS LB and x-forwarded-for

Posted by Wido den Hollander <wi...@widodh.nl>.
> Op 13 november 2017 om 15:14 schreef Pierre-Luc Dion <pd...@cloudops.com>:
> 
> 
> Hi,
> 
> This is looking quite promising, I have a colleague that tested that last
> Friday, look like the proxy proto v1 work out of the box with Nginx, but
> would need an extra package for Apache 2.4 ?

It depends. You need HTTPd 2.4.28, see: https://httpd.apache.org/docs/trunk/mod/mod_remoteip.html#remoteipproxyprotocol

It's there upstream, but not in all packages. 

It can from this module:

- https://github.com/roadrunner2/mod-proxy-protocol
- https://roadrunner2.github.io/mod-proxy-protocol/mod_proxy_protocol.html

They donated the code to go upstream and went into mod_remoteip but landed in 2.4.28

It will probably make it into Ubuntu 18.04 and CentOS 7.4.

Wido

> Otherwise, it seems to be a good way to do  https LB without complicated
> configuration and huge changes in CloudStack.
> 
> 
> 
> On Fri, Nov 10, 2017 at 10:36 AM, Nux! <nu...@li.nux.ro> wrote:
> 
> > Pierre-Luc,
> >
> > Haproxy docs say it should work for any kind of traffic as long as both
> > ends are PROXY-aware and it look like a majority of software is.
> > So, in short, yes.
> >
> > --
> > Sent from the Delta quadrant using Borg technology!
> >
> > Nux!
> > www.nux.ro
> >
> > ----- Original Message -----
> > > From: "Pierre-Luc Dion" <pd...@cloudops.com>
> > > To: "Wido den Hollander" <wi...@widodh.nl>
> > > Cc: "dev" <de...@cloudstack.apache.org>, "Khosrow Moossavi" <
> > kmoossavi@cloudops.com>, "Will Stevens"
> > > <ws...@cloudops.com>, "Nux!" <nu...@li.nux.ro>, "users" <
> > users@cloudstack.apache.org>
> > > Sent: Friday, 10 November, 2017 15:32:38
> > > Subject: Re: HTTPS LB and x-forwarded-for
> >
> > > Hi Wido, do you know if this would work for https traffic too?
> > >
> > > Le 10 nov. 2017 09 h 35, "Wido den Hollander" <wi...@widodh.nl> a écrit :
> > >
> > >>
> > >> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <
> > pdion@cloudops.com
> > >> >:
> > >> >
> > >> >
> > >> > I kind of like the proxy backend type, ill check on our end if that
> > would
> > >> > work but definitely a simple and efficient approach!
> > >> >
> > >>
> > >> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
> > >>
> > >> Apache HTTPd supports PROXY since 2.4.28:
> > https://httpd.apache.org/docs/
> > >> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
> > >>
> > >> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
> > >>
> > >> Wido
> > >>
> > >> >
> > >> >
> > >> > Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wi...@widodh.nl> a
> > écrit :
> > >> >
> > >> > >
> > >> > > > Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
> > >> > > >
> > >> > > >
> > >> > > > Wido,
> > >> > > >
> > >> > > > Excellent suggestion with the "transparent proxy", I was not
> > aware of
> > >> > > that.
> > >> > > > I think that would be a great idea and wouldn't require too many
> > >> > > modifications, especially as Haproxy comes already with the VR.
> > >> > > >
> > >> > >
> > >> > > It's indeed just a matter of a HAProxy config setting. We could
> > make it
> > >> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for
> > >> example.
> > >> > >
> > >> > > That way your problem would be solved.
> > >> > >
> > >> > > Wido
> > >> > >
> > >> > > > To Paul:
> > >> > > > - imho the LB solution ACS ships now is a bit handicaped since
> > you do
> > >> > > not know the remote host ip. You're flying blind unless you use
> > google
> > >> > > analytics (and these things have gotten more and more aggressively
> > >> filtered
> > >> > > by adblocks).
> > >> > > > Enhancing Haproxy as Wido suggested would go a long way, it
> > wouldn't
> > >> > > break existing functionality and would also keep SSL processing off
> > >> the VR.
> > >> > > >
> > >> > > > --
> > >> > > > Sent from the Delta quadrant using Borg technology!
> > >> > > >
> > >> > > > Nux!
> > >> > > > www.nux.ro
> > >> > > >
> > >> > > > ----- Original Message -----
> > >> > > > > From: "Andrija Panic" <an...@gmail.com>
> > >> > > > > To: "users" <us...@cloudstack.apache.org>
> > >> > > > > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will
> > Stevens" <
> > >> > > wstevens@cloudops.com>, "dev"
> > >> > > > > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <
> > pdion@cloudops.com
> > >> >
> > >> > > > > Sent: Thursday, 9 November, 2017 13:10:58
> > >> > > > > Subject: Re: HTTPS LB and x-forwarded-for
> > >> > > >
> > >> > > > > Wido,
> > >> > > > >
> > >> > > > > backend servers are not Linux only, for example we have a ton of
> > >> > > Windows
> > >> > > > > customers, some WEB solutions / IIS etc...
> > >> > > > >
> > >> > > > > @all - If we try to please/solve everyone's proxying
> > >> > > solution/requirement -
> > >> > > > > this is impossible IMHO - I'm thinking more about some "do it as
> > >> you
> > >> > > like"
> > >> > > > > solution, to let customer write his own haproxy config and
> > upoad it
> > >> > > (for
> > >> > > > > example, or something better?).
> > >> > > > >
> > >> > > > > We can support newer version of haproxy (1.5+) which also
> > implement
> > >> > > > > "transarent proxy" (integrate with kernel so to speak)  to allow
> > >> > > TCP-level
> > >> > > > > connections to backend (TCP mode, not HTTP mode) but to still
> > >> > > "preserve"
> > >> > > > > remote IP by faking it (fake soruce IP = transarent proxy).
> > >> > > > >
> > >> > > > > For the rest of configuration options,  I would leave it to the
> > >> > > customer
> > >> > > > > how he/she wants to configure rest of haproxy configuration,
> > >> inlcuding
> > >> > > > > custom checks, etc. Haproxy configuration is never-ending story,
> > >> and we
> > >> > > > > probably should allow custom sripts/configuration instead of
> > >> trying to
> > >> > > > > provide GUI/API way to configure everything (which is
> > >> impossible...)
> > >> > > > >
> > >> > > > > Just my 2 cents...
> > >> > > > >
> > >> > > > > On 9 November 2017 at 08:13, Wido den Hollander <wido@widodh.nl
> > >
> > >> > > wrote:
> > >> > > > >
> > >> > > > >>
> > >> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
> > >> > > pdion@cloudops.com
> > >> > > > >> >:
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > Same challenge here too!
> > >> > > > >> >
> > >> > > > >> > Let's look at improving Load-balancing offering from
> > >> cloudstack, I
> > >> > > guest
> > >> > > > >> we
> > >> > > > >> > should do a feature spec draft soon..,  from my perspective,
> > >> doing
> > >> > > SSL
> > >> > > > >> > offload on the VR could be problematic if the VR spec if too
> > >> small,
> > >> > > and
> > >> > > > >> the
> > >> > > > >> > default spec of the VR being 1vcpu@256MB, considering it can
> > >> be the
> > >> > > > >> router
> > >> > > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit
> > ish...
> > >> What
> > >> > > would
> > >> > > > >> > be your thought about this ?
> > >> > > > >> >
> > >> > > > >> > I'd be curious to have a LB offering in ACS where it would
> > >> deploy a
> > >> > > > >> > redundant traefik[1] beside the VR for doing http and https
> > >> > > > >> Load-balancing.
> > >> > > > >> > I think it would also be useful if the API of that traefik
> > >> instance
> > >> > > would
> > >> > > > >> > be available from within the VPC or LBnetwork so is API
> > would be
> > >> > > > >> accessible
> > >> > > > >> > to other apps orchestration tools such as  kubernetes or
> > >> rancher.
> > >> > > > >> >
> > >> > > > >> > traefik or not, here is what I think is needed by cloudstack
> > in
> > >> the
> > >> > > LB
> > >> > > > >> > improvement:
> > >> > > > >> >
> > >> > > > >> > - support http, https (X-Forwarded-For)
> > >> > > > >>
> > >> > > > >> HAProxy also supports the PROXY protocol towards the backends.
> > >> Apache
> > >> > > > >> 2.4.22 supports this natively and Varnish for example can also
> > >> talk
> > >> > > PROXY.
> > >> > > > >>
> > >> > > > >> It adds a littlebit of metadata to the connection so that the
> > >> backend
> > >> > > > >> knows the original IP the connection came from for example:
> > >> > > > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
> > >> > > > >>
> > >> > > > >> Wido
> > >> > > > >>
> > >> > > > >> > - basic persistence tuning (API already exist)
> > >> > > > >> > - better backend monitoring, currently only a tcp connect
> > >> validate
> > >> > > if the
> > >> > > > >> > webserver is up.
> > >> > > > >> > - ssl offload
> > >> > > > >> > - metric collection, more stats, maybe just export the tool
> > >> status
> > >> > > page
> > >> > > > >> to
> > >> > > > >> > the private network.
> > >> > > > >> > - Container world support, right now if you have Rancher or
> > >> > > kubernetes
> > >> > > > >> > cluster, you need to deploy your own LB solution behing
> > >> mostlikely a
> > >> > > > >> static
> > >> > > > >> > nat., If cloudstack would deploy a traefik instance, Kub or
> > >> Rancher
> > >> > > could
> > >> > > > >> > reuse this instance and managed it to properly do LB between
> > >> > > containers.
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > What would be your prefered LB tool:
> > >> > > > >> > haproxy, traefik or nginx?
> > >> > > > >> >
> > >> > > > >> > CloudStack already have to code to handle SSL certs per
> > >> projects and
> > >> > > > >> > accounts if not mistaking because that code was added to
> > support
> > >> > > > >> NetScaler
> > >> > > > >> > as Load-balancer in the past. so one less thing to think
> > about
> > >> :-)
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > [1] https://traefik.io/
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > PL,
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
> > >> > > > >> >
> > >> > > > >> > > Thanks Andrija,
> > >> > > > >> > >
> > >> > > > >> > > LB outside of the VR sounds like a good idea. An appliance
> > >> based
> > >> > > on,
> > >> > > > >> say
> > >> > > > >> > > cloud-init + ansible and so on could do the trick; alas
> > it'd
> > >> need
> > >> > > to be
> > >> > > > >> > > outside ACS.
> > >> > > > >> > > I guess as users we could maybe come up with a spec for an
> > >> > > > >> improvement, at
> > >> > > > >> > > least we'd have something the devs could look at whenever
> > it
> > >> is
> > >> > > > >> possible.
> > >> > > > >> > >
> > >> > > > >> > > Regards,
> > >> > > > >> > > Lucian
> > >> > > > >> > >
> > >> > > > >> > > --
> > >> > > > >> > > Sent from the Delta quadrant using Borg technology!
> > >> > > > >> > >
> > >> > > > >> > > Nux!
> > >> > > > >> > > www.nux.ro
> > >> > > > >> > >
> > >> > > > >> > > ----- Original Message -----
> > >> > > > >> > > > From: "Andrija Panic" <an...@gmail.com>
> > >> > > > >> > > > To: "dev" <de...@cloudstack.apache.org>
> > >> > > > >> > > > Cc: "users" <us...@cloudstack.apache.org>
> > >> > > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
> > >> > > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > >> > > > >> > >
> > >> > > > >> > > > We used to make some special stuff for one of the
> > clients,
> > >> > > where all
> > >> > > > >> LB
> > >> > > > >> > > > configuration work is done from outside of the ACS, i.e.
> > >> python
> > >> > > > >> script to
> > >> > > > >> > > > feed/configure VR - install latest haproxy 1.5.x for
> > >> transparent
> > >> > > > >> proxy,
> > >> > > > >> > > > since client insisted on SSL termination done on backend
> > >> web SSL
> > >> > > > >> > > servers....
> > >> > > > >> > > > Not good idea, that is all I can say (custom
> > configuration
> > >> > > thing) -
> > >> > > > >> but
> > >> > > > >> > > the
> > >> > > > >> > > > LB setup is actually good - transparent mode haproxy,
> > works
> > >> on
> > >> > > TCP
> > >> > > > >> level,
> > >> > > > >> > > > so you can see "real client IP" on the backend servers
> > >> (which
> > >> > > must
> > >> > > > >> use VR
> > >> > > > >> > > > as the default gtw, as per default, so the whole setup
> > works
> > >> > > > >> properly).
> > >> > > > >> > > >
> > >> > > > >> > > > I'm still looking forward to see some special support of
> > LB
> > >> > > inside
> > >> > > > >> VR via
> > >> > > > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to
> > >> enable LB
> > >> > > > >> > > > provisioning SCRIPT (bash, or whatever),  where all
> > needed
> > >> > > > >> > > > install+configure can be done from client side  -
> > otherwise
> > >> > > covering
> > >> > > > >> all
> > >> > > > >> > > > user cases, with proper HTTP checks and similar....is
> > >> > > impossible to
> > >> > > > >> do
> > >> > > > >> > > > IMHO.
> > >> > > > >> > > >
> > >> > > > >> > > > Some other clients, actually have internal FW appliance
> > >> (i.e.
> > >> > > > >> multihomed
> > >> > > > >> > > > VM, acting as gtw for all VMs in all networks), and
> > haproxy
> > >> > > instaled
> > >> > > > >> on
> > >> > > > >> > > > this device (with NAT configured from VR to this internal
> > >> > > FW/VM, so
> > >> > > > >> > > remote
> > >> > > > >> > > > IP can be seen properly) - this setup is fully under
> > >> customer
> > >> > > > >> control,
> > >> > > > >> > > and
> > >> > > > >> > > > can provide any kind of special haproxy config...
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> > >> > > > >> > > >
> > >> > > > >> > > >> Hello,
> > >> > > > >> > > >>
> > >> > > > >> > > >> Of the people running an LB (VR) with https backends,
> > how
> > >> do
> > >> > > you
> > >> > > > >> deal
> > >> > > > >> > > with
> > >> > > > >> > > >> the lack of x-forwarded-for since for port 443 there's
> > just
> > >> > > simple
> > >> > > > >> TCP
> > >> > > > >> > > >> balancing?
> > >> > > > >> > > >>
> > >> > > > >> > > >> Has anyone thought of terminating SSL in the VR instead?
> > >> Ideas?
> > >> > > > >> > > >>
> > >> > > > >> > > >> Cheers
> > >> > > > >> > > >>
> > >> > > > >> > > >> --
> > >> > > > >> > > >> Sent from the Delta quadrant using Borg technology!
> > >> > > > >> > > >>
> > >> > > > >> > > >> Nux!
> > >> > > > >> > > >> www.nux.ro
> > >> > > > >> > > >>
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > > --
> > >> > > > >> > > >
> > >> > > > >> > > > Andrija Panić
> > >> > > > >> > >
> > >> > > > >>
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > >
> > >> > > > > Andrija Panić
> > >> > >
> >

Re: HTTPS LB and x-forwarded-for

Posted by Wido den Hollander <wi...@widodh.nl>.
> Op 13 november 2017 om 15:14 schreef Pierre-Luc Dion <pd...@cloudops.com>:
> 
> 
> Hi,
> 
> This is looking quite promising, I have a colleague that tested that last
> Friday, look like the proxy proto v1 work out of the box with Nginx, but
> would need an extra package for Apache 2.4 ?

It depends. You need HTTPd 2.4.28, see: https://httpd.apache.org/docs/trunk/mod/mod_remoteip.html#remoteipproxyprotocol

It's there upstream, but not in all packages. 

It can from this module:

- https://github.com/roadrunner2/mod-proxy-protocol
- https://roadrunner2.github.io/mod-proxy-protocol/mod_proxy_protocol.html

They donated the code to go upstream and went into mod_remoteip but landed in 2.4.28

It will probably make it into Ubuntu 18.04 and CentOS 7.4.

Wido

> Otherwise, it seems to be a good way to do  https LB without complicated
> configuration and huge changes in CloudStack.
> 
> 
> 
> On Fri, Nov 10, 2017 at 10:36 AM, Nux! <nu...@li.nux.ro> wrote:
> 
> > Pierre-Luc,
> >
> > Haproxy docs say it should work for any kind of traffic as long as both
> > ends are PROXY-aware and it look like a majority of software is.
> > So, in short, yes.
> >
> > --
> > Sent from the Delta quadrant using Borg technology!
> >
> > Nux!
> > www.nux.ro
> >
> > ----- Original Message -----
> > > From: "Pierre-Luc Dion" <pd...@cloudops.com>
> > > To: "Wido den Hollander" <wi...@widodh.nl>
> > > Cc: "dev" <de...@cloudstack.apache.org>, "Khosrow Moossavi" <
> > kmoossavi@cloudops.com>, "Will Stevens"
> > > <ws...@cloudops.com>, "Nux!" <nu...@li.nux.ro>, "users" <
> > users@cloudstack.apache.org>
> > > Sent: Friday, 10 November, 2017 15:32:38
> > > Subject: Re: HTTPS LB and x-forwarded-for
> >
> > > Hi Wido, do you know if this would work for https traffic too?
> > >
> > > Le 10 nov. 2017 09 h 35, "Wido den Hollander" <wi...@widodh.nl> a écrit :
> > >
> > >>
> > >> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <
> > pdion@cloudops.com
> > >> >:
> > >> >
> > >> >
> > >> > I kind of like the proxy backend type, ill check on our end if that
> > would
> > >> > work but definitely a simple and efficient approach!
> > >> >
> > >>
> > >> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
> > >>
> > >> Apache HTTPd supports PROXY since 2.4.28:
> > https://httpd.apache.org/docs/
> > >> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
> > >>
> > >> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
> > >>
> > >> Wido
> > >>
> > >> >
> > >> >
> > >> > Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wi...@widodh.nl> a
> > écrit :
> > >> >
> > >> > >
> > >> > > > Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
> > >> > > >
> > >> > > >
> > >> > > > Wido,
> > >> > > >
> > >> > > > Excellent suggestion with the "transparent proxy", I was not
> > aware of
> > >> > > that.
> > >> > > > I think that would be a great idea and wouldn't require too many
> > >> > > modifications, especially as Haproxy comes already with the VR.
> > >> > > >
> > >> > >
> > >> > > It's indeed just a matter of a HAProxy config setting. We could
> > make it
> > >> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for
> > >> example.
> > >> > >
> > >> > > That way your problem would be solved.
> > >> > >
> > >> > > Wido
> > >> > >
> > >> > > > To Paul:
> > >> > > > - imho the LB solution ACS ships now is a bit handicaped since
> > you do
> > >> > > not know the remote host ip. You're flying blind unless you use
> > google
> > >> > > analytics (and these things have gotten more and more aggressively
> > >> filtered
> > >> > > by adblocks).
> > >> > > > Enhancing Haproxy as Wido suggested would go a long way, it
> > wouldn't
> > >> > > break existing functionality and would also keep SSL processing off
> > >> the VR.
> > >> > > >
> > >> > > > --
> > >> > > > Sent from the Delta quadrant using Borg technology!
> > >> > > >
> > >> > > > Nux!
> > >> > > > www.nux.ro
> > >> > > >
> > >> > > > ----- Original Message -----
> > >> > > > > From: "Andrija Panic" <an...@gmail.com>
> > >> > > > > To: "users" <us...@cloudstack.apache.org>
> > >> > > > > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will
> > Stevens" <
> > >> > > wstevens@cloudops.com>, "dev"
> > >> > > > > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <
> > pdion@cloudops.com
> > >> >
> > >> > > > > Sent: Thursday, 9 November, 2017 13:10:58
> > >> > > > > Subject: Re: HTTPS LB and x-forwarded-for
> > >> > > >
> > >> > > > > Wido,
> > >> > > > >
> > >> > > > > backend servers are not Linux only, for example we have a ton of
> > >> > > Windows
> > >> > > > > customers, some WEB solutions / IIS etc...
> > >> > > > >
> > >> > > > > @all - If we try to please/solve everyone's proxying
> > >> > > solution/requirement -
> > >> > > > > this is impossible IMHO - I'm thinking more about some "do it as
> > >> you
> > >> > > like"
> > >> > > > > solution, to let customer write his own haproxy config and
> > upoad it
> > >> > > (for
> > >> > > > > example, or something better?).
> > >> > > > >
> > >> > > > > We can support newer version of haproxy (1.5+) which also
> > implement
> > >> > > > > "transarent proxy" (integrate with kernel so to speak)  to allow
> > >> > > TCP-level
> > >> > > > > connections to backend (TCP mode, not HTTP mode) but to still
> > >> > > "preserve"
> > >> > > > > remote IP by faking it (fake soruce IP = transarent proxy).
> > >> > > > >
> > >> > > > > For the rest of configuration options,  I would leave it to the
> > >> > > customer
> > >> > > > > how he/she wants to configure rest of haproxy configuration,
> > >> inlcuding
> > >> > > > > custom checks, etc. Haproxy configuration is never-ending story,
> > >> and we
> > >> > > > > probably should allow custom sripts/configuration instead of
> > >> trying to
> > >> > > > > provide GUI/API way to configure everything (which is
> > >> impossible...)
> > >> > > > >
> > >> > > > > Just my 2 cents...
> > >> > > > >
> > >> > > > > On 9 November 2017 at 08:13, Wido den Hollander <wido@widodh.nl
> > >
> > >> > > wrote:
> > >> > > > >
> > >> > > > >>
> > >> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
> > >> > > pdion@cloudops.com
> > >> > > > >> >:
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > Same challenge here too!
> > >> > > > >> >
> > >> > > > >> > Let's look at improving Load-balancing offering from
> > >> cloudstack, I
> > >> > > guest
> > >> > > > >> we
> > >> > > > >> > should do a feature spec draft soon..,  from my perspective,
> > >> doing
> > >> > > SSL
> > >> > > > >> > offload on the VR could be problematic if the VR spec if too
> > >> small,
> > >> > > and
> > >> > > > >> the
> > >> > > > >> > default spec of the VR being 1vcpu@256MB, considering it can
> > >> be the
> > >> > > > >> router
> > >> > > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit
> > ish...
> > >> What
> > >> > > would
> > >> > > > >> > be your thought about this ?
> > >> > > > >> >
> > >> > > > >> > I'd be curious to have a LB offering in ACS where it would
> > >> deploy a
> > >> > > > >> > redundant traefik[1] beside the VR for doing http and https
> > >> > > > >> Load-balancing.
> > >> > > > >> > I think it would also be useful if the API of that traefik
> > >> instance
> > >> > > would
> > >> > > > >> > be available from within the VPC or LBnetwork so is API
> > would be
> > >> > > > >> accessible
> > >> > > > >> > to other apps orchestration tools such as  kubernetes or
> > >> rancher.
> > >> > > > >> >
> > >> > > > >> > traefik or not, here is what I think is needed by cloudstack
> > in
> > >> the
> > >> > > LB
> > >> > > > >> > improvement:
> > >> > > > >> >
> > >> > > > >> > - support http, https (X-Forwarded-For)
> > >> > > > >>
> > >> > > > >> HAProxy also supports the PROXY protocol towards the backends.
> > >> Apache
> > >> > > > >> 2.4.22 supports this natively and Varnish for example can also
> > >> talk
> > >> > > PROXY.
> > >> > > > >>
> > >> > > > >> It adds a littlebit of metadata to the connection so that the
> > >> backend
> > >> > > > >> knows the original IP the connection came from for example:
> > >> > > > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
> > >> > > > >>
> > >> > > > >> Wido
> > >> > > > >>
> > >> > > > >> > - basic persistence tuning (API already exist)
> > >> > > > >> > - better backend monitoring, currently only a tcp connect
> > >> validate
> > >> > > if the
> > >> > > > >> > webserver is up.
> > >> > > > >> > - ssl offload
> > >> > > > >> > - metric collection, more stats, maybe just export the tool
> > >> status
> > >> > > page
> > >> > > > >> to
> > >> > > > >> > the private network.
> > >> > > > >> > - Container world support, right now if you have Rancher or
> > >> > > kubernetes
> > >> > > > >> > cluster, you need to deploy your own LB solution behing
> > >> mostlikely a
> > >> > > > >> static
> > >> > > > >> > nat., If cloudstack would deploy a traefik instance, Kub or
> > >> Rancher
> > >> > > could
> > >> > > > >> > reuse this instance and managed it to properly do LB between
> > >> > > containers.
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > What would be your prefered LB tool:
> > >> > > > >> > haproxy, traefik or nginx?
> > >> > > > >> >
> > >> > > > >> > CloudStack already have to code to handle SSL certs per
> > >> projects and
> > >> > > > >> > accounts if not mistaking because that code was added to
> > support
> > >> > > > >> NetScaler
> > >> > > > >> > as Load-balancer in the past. so one less thing to think
> > about
> > >> :-)
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > [1] https://traefik.io/
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > PL,
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
> > >> > > > >> >
> > >> > > > >> > > Thanks Andrija,
> > >> > > > >> > >
> > >> > > > >> > > LB outside of the VR sounds like a good idea. An appliance
> > >> based
> > >> > > on,
> > >> > > > >> say
> > >> > > > >> > > cloud-init + ansible and so on could do the trick; alas
> > it'd
> > >> need
> > >> > > to be
> > >> > > > >> > > outside ACS.
> > >> > > > >> > > I guess as users we could maybe come up with a spec for an
> > >> > > > >> improvement, at
> > >> > > > >> > > least we'd have something the devs could look at whenever
> > it
> > >> is
> > >> > > > >> possible.
> > >> > > > >> > >
> > >> > > > >> > > Regards,
> > >> > > > >> > > Lucian
> > >> > > > >> > >
> > >> > > > >> > > --
> > >> > > > >> > > Sent from the Delta quadrant using Borg technology!
> > >> > > > >> > >
> > >> > > > >> > > Nux!
> > >> > > > >> > > www.nux.ro
> > >> > > > >> > >
> > >> > > > >> > > ----- Original Message -----
> > >> > > > >> > > > From: "Andrija Panic" <an...@gmail.com>
> > >> > > > >> > > > To: "dev" <de...@cloudstack.apache.org>
> > >> > > > >> > > > Cc: "users" <us...@cloudstack.apache.org>
> > >> > > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
> > >> > > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > >> > > > >> > >
> > >> > > > >> > > > We used to make some special stuff for one of the
> > clients,
> > >> > > where all
> > >> > > > >> LB
> > >> > > > >> > > > configuration work is done from outside of the ACS, i.e.
> > >> python
> > >> > > > >> script to
> > >> > > > >> > > > feed/configure VR - install latest haproxy 1.5.x for
> > >> transparent
> > >> > > > >> proxy,
> > >> > > > >> > > > since client insisted on SSL termination done on backend
> > >> web SSL
> > >> > > > >> > > servers....
> > >> > > > >> > > > Not good idea, that is all I can say (custom
> > configuration
> > >> > > thing) -
> > >> > > > >> but
> > >> > > > >> > > the
> > >> > > > >> > > > LB setup is actually good - transparent mode haproxy,
> > works
> > >> on
> > >> > > TCP
> > >> > > > >> level,
> > >> > > > >> > > > so you can see "real client IP" on the backend servers
> > >> (which
> > >> > > must
> > >> > > > >> use VR
> > >> > > > >> > > > as the default gtw, as per default, so the whole setup
> > works
> > >> > > > >> properly).
> > >> > > > >> > > >
> > >> > > > >> > > > I'm still looking forward to see some special support of
> > LB
> > >> > > inside
> > >> > > > >> VR via
> > >> > > > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to
> > >> enable LB
> > >> > > > >> > > > provisioning SCRIPT (bash, or whatever),  where all
> > needed
> > >> > > > >> > > > install+configure can be done from client side  -
> > otherwise
> > >> > > covering
> > >> > > > >> all
> > >> > > > >> > > > user cases, with proper HTTP checks and similar....is
> > >> > > impossible to
> > >> > > > >> do
> > >> > > > >> > > > IMHO.
> > >> > > > >> > > >
> > >> > > > >> > > > Some other clients, actually have internal FW appliance
> > >> (i.e.
> > >> > > > >> multihomed
> > >> > > > >> > > > VM, acting as gtw for all VMs in all networks), and
> > haproxy
> > >> > > instaled
> > >> > > > >> on
> > >> > > > >> > > > this device (with NAT configured from VR to this internal
> > >> > > FW/VM, so
> > >> > > > >> > > remote
> > >> > > > >> > > > IP can be seen properly) - this setup is fully under
> > >> customer
> > >> > > > >> control,
> > >> > > > >> > > and
> > >> > > > >> > > > can provide any kind of special haproxy config...
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> > >> > > > >> > > >
> > >> > > > >> > > >> Hello,
> > >> > > > >> > > >>
> > >> > > > >> > > >> Of the people running an LB (VR) with https backends,
> > how
> > >> do
> > >> > > you
> > >> > > > >> deal
> > >> > > > >> > > with
> > >> > > > >> > > >> the lack of x-forwarded-for since for port 443 there's
> > just
> > >> > > simple
> > >> > > > >> TCP
> > >> > > > >> > > >> balancing?
> > >> > > > >> > > >>
> > >> > > > >> > > >> Has anyone thought of terminating SSL in the VR instead?
> > >> Ideas?
> > >> > > > >> > > >>
> > >> > > > >> > > >> Cheers
> > >> > > > >> > > >>
> > >> > > > >> > > >> --
> > >> > > > >> > > >> Sent from the Delta quadrant using Borg technology!
> > >> > > > >> > > >>
> > >> > > > >> > > >> Nux!
> > >> > > > >> > > >> www.nux.ro
> > >> > > > >> > > >>
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > > --
> > >> > > > >> > > >
> > >> > > > >> > > > Andrija Panić
> > >> > > > >> > >
> > >> > > > >>
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > >
> > >> > > > > Andrija Panić
> > >> > >
> >

Re: HTTPS LB and x-forwarded-for

Posted by Pierre-Luc Dion <pd...@cloudops.com>.
Hi,

This is looking quite promising, I have a colleague that tested that last
Friday, look like the proxy proto v1 work out of the box with Nginx, but
would need an extra package for Apache 2.4 ?
Otherwise, it seems to be a good way to do  https LB without complicated
configuration and huge changes in CloudStack.



On Fri, Nov 10, 2017 at 10:36 AM, Nux! <nu...@li.nux.ro> wrote:

> Pierre-Luc,
>
> Haproxy docs say it should work for any kind of traffic as long as both
> ends are PROXY-aware and it look like a majority of software is.
> So, in short, yes.
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Pierre-Luc Dion" <pd...@cloudops.com>
> > To: "Wido den Hollander" <wi...@widodh.nl>
> > Cc: "dev" <de...@cloudstack.apache.org>, "Khosrow Moossavi" <
> kmoossavi@cloudops.com>, "Will Stevens"
> > <ws...@cloudops.com>, "Nux!" <nu...@li.nux.ro>, "users" <
> users@cloudstack.apache.org>
> > Sent: Friday, 10 November, 2017 15:32:38
> > Subject: Re: HTTPS LB and x-forwarded-for
>
> > Hi Wido, do you know if this would work for https traffic too?
> >
> > Le 10 nov. 2017 09 h 35, "Wido den Hollander" <wi...@widodh.nl> a écrit :
> >
> >>
> >> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <
> pdion@cloudops.com
> >> >:
> >> >
> >> >
> >> > I kind of like the proxy backend type, ill check on our end if that
> would
> >> > work but definitely a simple and efficient approach!
> >> >
> >>
> >> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
> >>
> >> Apache HTTPd supports PROXY since 2.4.28:
> https://httpd.apache.org/docs/
> >> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
> >>
> >> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
> >>
> >> Wido
> >>
> >> >
> >> >
> >> > Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wi...@widodh.nl> a
> écrit :
> >> >
> >> > >
> >> > > > Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
> >> > > >
> >> > > >
> >> > > > Wido,
> >> > > >
> >> > > > Excellent suggestion with the "transparent proxy", I was not
> aware of
> >> > > that.
> >> > > > I think that would be a great idea and wouldn't require too many
> >> > > modifications, especially as Haproxy comes already with the VR.
> >> > > >
> >> > >
> >> > > It's indeed just a matter of a HAProxy config setting. We could
> make it
> >> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for
> >> example.
> >> > >
> >> > > That way your problem would be solved.
> >> > >
> >> > > Wido
> >> > >
> >> > > > To Paul:
> >> > > > - imho the LB solution ACS ships now is a bit handicaped since
> you do
> >> > > not know the remote host ip. You're flying blind unless you use
> google
> >> > > analytics (and these things have gotten more and more aggressively
> >> filtered
> >> > > by adblocks).
> >> > > > Enhancing Haproxy as Wido suggested would go a long way, it
> wouldn't
> >> > > break existing functionality and would also keep SSL processing off
> >> the VR.
> >> > > >
> >> > > > --
> >> > > > Sent from the Delta quadrant using Borg technology!
> >> > > >
> >> > > > Nux!
> >> > > > www.nux.ro
> >> > > >
> >> > > > ----- Original Message -----
> >> > > > > From: "Andrija Panic" <an...@gmail.com>
> >> > > > > To: "users" <us...@cloudstack.apache.org>
> >> > > > > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will
> Stevens" <
> >> > > wstevens@cloudops.com>, "dev"
> >> > > > > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <
> pdion@cloudops.com
> >> >
> >> > > > > Sent: Thursday, 9 November, 2017 13:10:58
> >> > > > > Subject: Re: HTTPS LB and x-forwarded-for
> >> > > >
> >> > > > > Wido,
> >> > > > >
> >> > > > > backend servers are not Linux only, for example we have a ton of
> >> > > Windows
> >> > > > > customers, some WEB solutions / IIS etc...
> >> > > > >
> >> > > > > @all - If we try to please/solve everyone's proxying
> >> > > solution/requirement -
> >> > > > > this is impossible IMHO - I'm thinking more about some "do it as
> >> you
> >> > > like"
> >> > > > > solution, to let customer write his own haproxy config and
> upoad it
> >> > > (for
> >> > > > > example, or something better?).
> >> > > > >
> >> > > > > We can support newer version of haproxy (1.5+) which also
> implement
> >> > > > > "transarent proxy" (integrate with kernel so to speak)  to allow
> >> > > TCP-level
> >> > > > > connections to backend (TCP mode, not HTTP mode) but to still
> >> > > "preserve"
> >> > > > > remote IP by faking it (fake soruce IP = transarent proxy).
> >> > > > >
> >> > > > > For the rest of configuration options,  I would leave it to the
> >> > > customer
> >> > > > > how he/she wants to configure rest of haproxy configuration,
> >> inlcuding
> >> > > > > custom checks, etc. Haproxy configuration is never-ending story,
> >> and we
> >> > > > > probably should allow custom sripts/configuration instead of
> >> trying to
> >> > > > > provide GUI/API way to configure everything (which is
> >> impossible...)
> >> > > > >
> >> > > > > Just my 2 cents...
> >> > > > >
> >> > > > > On 9 November 2017 at 08:13, Wido den Hollander <wido@widodh.nl
> >
> >> > > wrote:
> >> > > > >
> >> > > > >>
> >> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
> >> > > pdion@cloudops.com
> >> > > > >> >:
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > Same challenge here too!
> >> > > > >> >
> >> > > > >> > Let's look at improving Load-balancing offering from
> >> cloudstack, I
> >> > > guest
> >> > > > >> we
> >> > > > >> > should do a feature spec draft soon..,  from my perspective,
> >> doing
> >> > > SSL
> >> > > > >> > offload on the VR could be problematic if the VR spec if too
> >> small,
> >> > > and
> >> > > > >> the
> >> > > > >> > default spec of the VR being 1vcpu@256MB, considering it can
> >> be the
> >> > > > >> router
> >> > > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit
> ish...
> >> What
> >> > > would
> >> > > > >> > be your thought about this ?
> >> > > > >> >
> >> > > > >> > I'd be curious to have a LB offering in ACS where it would
> >> deploy a
> >> > > > >> > redundant traefik[1] beside the VR for doing http and https
> >> > > > >> Load-balancing.
> >> > > > >> > I think it would also be useful if the API of that traefik
> >> instance
> >> > > would
> >> > > > >> > be available from within the VPC or LBnetwork so is API
> would be
> >> > > > >> accessible
> >> > > > >> > to other apps orchestration tools such as  kubernetes or
> >> rancher.
> >> > > > >> >
> >> > > > >> > traefik or not, here is what I think is needed by cloudstack
> in
> >> the
> >> > > LB
> >> > > > >> > improvement:
> >> > > > >> >
> >> > > > >> > - support http, https (X-Forwarded-For)
> >> > > > >>
> >> > > > >> HAProxy also supports the PROXY protocol towards the backends.
> >> Apache
> >> > > > >> 2.4.22 supports this natively and Varnish for example can also
> >> talk
> >> > > PROXY.
> >> > > > >>
> >> > > > >> It adds a littlebit of metadata to the connection so that the
> >> backend
> >> > > > >> knows the original IP the connection came from for example:
> >> > > > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
> >> > > > >>
> >> > > > >> Wido
> >> > > > >>
> >> > > > >> > - basic persistence tuning (API already exist)
> >> > > > >> > - better backend monitoring, currently only a tcp connect
> >> validate
> >> > > if the
> >> > > > >> > webserver is up.
> >> > > > >> > - ssl offload
> >> > > > >> > - metric collection, more stats, maybe just export the tool
> >> status
> >> > > page
> >> > > > >> to
> >> > > > >> > the private network.
> >> > > > >> > - Container world support, right now if you have Rancher or
> >> > > kubernetes
> >> > > > >> > cluster, you need to deploy your own LB solution behing
> >> mostlikely a
> >> > > > >> static
> >> > > > >> > nat., If cloudstack would deploy a traefik instance, Kub or
> >> Rancher
> >> > > could
> >> > > > >> > reuse this instance and managed it to properly do LB between
> >> > > containers.
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > What would be your prefered LB tool:
> >> > > > >> > haproxy, traefik or nginx?
> >> > > > >> >
> >> > > > >> > CloudStack already have to code to handle SSL certs per
> >> projects and
> >> > > > >> > accounts if not mistaking because that code was added to
> support
> >> > > > >> NetScaler
> >> > > > >> > as Load-balancer in the past. so one less thing to think
> about
> >> :-)
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > [1] https://traefik.io/
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > PL,
> >> > > > >> >
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
> >> > > > >> >
> >> > > > >> > > Thanks Andrija,
> >> > > > >> > >
> >> > > > >> > > LB outside of the VR sounds like a good idea. An appliance
> >> based
> >> > > on,
> >> > > > >> say
> >> > > > >> > > cloud-init + ansible and so on could do the trick; alas
> it'd
> >> need
> >> > > to be
> >> > > > >> > > outside ACS.
> >> > > > >> > > I guess as users we could maybe come up with a spec for an
> >> > > > >> improvement, at
> >> > > > >> > > least we'd have something the devs could look at whenever
> it
> >> is
> >> > > > >> possible.
> >> > > > >> > >
> >> > > > >> > > Regards,
> >> > > > >> > > Lucian
> >> > > > >> > >
> >> > > > >> > > --
> >> > > > >> > > Sent from the Delta quadrant using Borg technology!
> >> > > > >> > >
> >> > > > >> > > Nux!
> >> > > > >> > > www.nux.ro
> >> > > > >> > >
> >> > > > >> > > ----- Original Message -----
> >> > > > >> > > > From: "Andrija Panic" <an...@gmail.com>
> >> > > > >> > > > To: "dev" <de...@cloudstack.apache.org>
> >> > > > >> > > > Cc: "users" <us...@cloudstack.apache.org>
> >> > > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
> >> > > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
> >> > > > >> > >
> >> > > > >> > > > We used to make some special stuff for one of the
> clients,
> >> > > where all
> >> > > > >> LB
> >> > > > >> > > > configuration work is done from outside of the ACS, i.e.
> >> python
> >> > > > >> script to
> >> > > > >> > > > feed/configure VR - install latest haproxy 1.5.x for
> >> transparent
> >> > > > >> proxy,
> >> > > > >> > > > since client insisted on SSL termination done on backend
> >> web SSL
> >> > > > >> > > servers....
> >> > > > >> > > > Not good idea, that is all I can say (custom
> configuration
> >> > > thing) -
> >> > > > >> but
> >> > > > >> > > the
> >> > > > >> > > > LB setup is actually good - transparent mode haproxy,
> works
> >> on
> >> > > TCP
> >> > > > >> level,
> >> > > > >> > > > so you can see "real client IP" on the backend servers
> >> (which
> >> > > must
> >> > > > >> use VR
> >> > > > >> > > > as the default gtw, as per default, so the whole setup
> works
> >> > > > >> properly).
> >> > > > >> > > >
> >> > > > >> > > > I'm still looking forward to see some special support of
> LB
> >> > > inside
> >> > > > >> VR via
> >> > > > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to
> >> enable LB
> >> > > > >> > > > provisioning SCRIPT (bash, or whatever),  where all
> needed
> >> > > > >> > > > install+configure can be done from client side  -
> otherwise
> >> > > covering
> >> > > > >> all
> >> > > > >> > > > user cases, with proper HTTP checks and similar....is
> >> > > impossible to
> >> > > > >> do
> >> > > > >> > > > IMHO.
> >> > > > >> > > >
> >> > > > >> > > > Some other clients, actually have internal FW appliance
> >> (i.e.
> >> > > > >> multihomed
> >> > > > >> > > > VM, acting as gtw for all VMs in all networks), and
> haproxy
> >> > > instaled
> >> > > > >> on
> >> > > > >> > > > this device (with NAT configured from VR to this internal
> >> > > FW/VM, so
> >> > > > >> > > remote
> >> > > > >> > > > IP can be seen properly) - this setup is fully under
> >> customer
> >> > > > >> control,
> >> > > > >> > > and
> >> > > > >> > > > can provide any kind of special haproxy config...
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> >> > > > >> > > >
> >> > > > >> > > >> Hello,
> >> > > > >> > > >>
> >> > > > >> > > >> Of the people running an LB (VR) with https backends,
> how
> >> do
> >> > > you
> >> > > > >> deal
> >> > > > >> > > with
> >> > > > >> > > >> the lack of x-forwarded-for since for port 443 there's
> just
> >> > > simple
> >> > > > >> TCP
> >> > > > >> > > >> balancing?
> >> > > > >> > > >>
> >> > > > >> > > >> Has anyone thought of terminating SSL in the VR instead?
> >> Ideas?
> >> > > > >> > > >>
> >> > > > >> > > >> Cheers
> >> > > > >> > > >>
> >> > > > >> > > >> --
> >> > > > >> > > >> Sent from the Delta quadrant using Borg technology!
> >> > > > >> > > >>
> >> > > > >> > > >> Nux!
> >> > > > >> > > >> www.nux.ro
> >> > > > >> > > >>
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > > --
> >> > > > >> > > >
> >> > > > >> > > > Andrija Panić
> >> > > > >> > >
> >> > > > >>
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > >
> >> > > > > Andrija Panić
> >> > >
>

Re: HTTPS LB and x-forwarded-for

Posted by Pierre-Luc Dion <pd...@cloudops.com>.
Hi,

This is looking quite promising, I have a colleague that tested that last
Friday, look like the proxy proto v1 work out of the box with Nginx, but
would need an extra package for Apache 2.4 ?
Otherwise, it seems to be a good way to do  https LB without complicated
configuration and huge changes in CloudStack.



On Fri, Nov 10, 2017 at 10:36 AM, Nux! <nu...@li.nux.ro> wrote:

> Pierre-Luc,
>
> Haproxy docs say it should work for any kind of traffic as long as both
> ends are PROXY-aware and it look like a majority of software is.
> So, in short, yes.
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Pierre-Luc Dion" <pd...@cloudops.com>
> > To: "Wido den Hollander" <wi...@widodh.nl>
> > Cc: "dev" <de...@cloudstack.apache.org>, "Khosrow Moossavi" <
> kmoossavi@cloudops.com>, "Will Stevens"
> > <ws...@cloudops.com>, "Nux!" <nu...@li.nux.ro>, "users" <
> users@cloudstack.apache.org>
> > Sent: Friday, 10 November, 2017 15:32:38
> > Subject: Re: HTTPS LB and x-forwarded-for
>
> > Hi Wido, do you know if this would work for https traffic too?
> >
> > Le 10 nov. 2017 09 h 35, "Wido den Hollander" <wi...@widodh.nl> a écrit :
> >
> >>
> >> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <
> pdion@cloudops.com
> >> >:
> >> >
> >> >
> >> > I kind of like the proxy backend type, ill check on our end if that
> would
> >> > work but definitely a simple and efficient approach!
> >> >
> >>
> >> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
> >>
> >> Apache HTTPd supports PROXY since 2.4.28:
> https://httpd.apache.org/docs/
> >> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
> >>
> >> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
> >>
> >> Wido
> >>
> >> >
> >> >
> >> > Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wi...@widodh.nl> a
> écrit :
> >> >
> >> > >
> >> > > > Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
> >> > > >
> >> > > >
> >> > > > Wido,
> >> > > >
> >> > > > Excellent suggestion with the "transparent proxy", I was not
> aware of
> >> > > that.
> >> > > > I think that would be a great idea and wouldn't require too many
> >> > > modifications, especially as Haproxy comes already with the VR.
> >> > > >
> >> > >
> >> > > It's indeed just a matter of a HAProxy config setting. We could
> make it
> >> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for
> >> example.
> >> > >
> >> > > That way your problem would be solved.
> >> > >
> >> > > Wido
> >> > >
> >> > > > To Paul:
> >> > > > - imho the LB solution ACS ships now is a bit handicaped since
> you do
> >> > > not know the remote host ip. You're flying blind unless you use
> google
> >> > > analytics (and these things have gotten more and more aggressively
> >> filtered
> >> > > by adblocks).
> >> > > > Enhancing Haproxy as Wido suggested would go a long way, it
> wouldn't
> >> > > break existing functionality and would also keep SSL processing off
> >> the VR.
> >> > > >
> >> > > > --
> >> > > > Sent from the Delta quadrant using Borg technology!
> >> > > >
> >> > > > Nux!
> >> > > > www.nux.ro
> >> > > >
> >> > > > ----- Original Message -----
> >> > > > > From: "Andrija Panic" <an...@gmail.com>
> >> > > > > To: "users" <us...@cloudstack.apache.org>
> >> > > > > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will
> Stevens" <
> >> > > wstevens@cloudops.com>, "dev"
> >> > > > > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <
> pdion@cloudops.com
> >> >
> >> > > > > Sent: Thursday, 9 November, 2017 13:10:58
> >> > > > > Subject: Re: HTTPS LB and x-forwarded-for
> >> > > >
> >> > > > > Wido,
> >> > > > >
> >> > > > > backend servers are not Linux only, for example we have a ton of
> >> > > Windows
> >> > > > > customers, some WEB solutions / IIS etc...
> >> > > > >
> >> > > > > @all - If we try to please/solve everyone's proxying
> >> > > solution/requirement -
> >> > > > > this is impossible IMHO - I'm thinking more about some "do it as
> >> you
> >> > > like"
> >> > > > > solution, to let customer write his own haproxy config and
> upoad it
> >> > > (for
> >> > > > > example, or something better?).
> >> > > > >
> >> > > > > We can support newer version of haproxy (1.5+) which also
> implement
> >> > > > > "transarent proxy" (integrate with kernel so to speak)  to allow
> >> > > TCP-level
> >> > > > > connections to backend (TCP mode, not HTTP mode) but to still
> >> > > "preserve"
> >> > > > > remote IP by faking it (fake soruce IP = transarent proxy).
> >> > > > >
> >> > > > > For the rest of configuration options,  I would leave it to the
> >> > > customer
> >> > > > > how he/she wants to configure rest of haproxy configuration,
> >> inlcuding
> >> > > > > custom checks, etc. Haproxy configuration is never-ending story,
> >> and we
> >> > > > > probably should allow custom sripts/configuration instead of
> >> trying to
> >> > > > > provide GUI/API way to configure everything (which is
> >> impossible...)
> >> > > > >
> >> > > > > Just my 2 cents...
> >> > > > >
> >> > > > > On 9 November 2017 at 08:13, Wido den Hollander <wido@widodh.nl
> >
> >> > > wrote:
> >> > > > >
> >> > > > >>
> >> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
> >> > > pdion@cloudops.com
> >> > > > >> >:
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > Same challenge here too!
> >> > > > >> >
> >> > > > >> > Let's look at improving Load-balancing offering from
> >> cloudstack, I
> >> > > guest
> >> > > > >> we
> >> > > > >> > should do a feature spec draft soon..,  from my perspective,
> >> doing
> >> > > SSL
> >> > > > >> > offload on the VR could be problematic if the VR spec if too
> >> small,
> >> > > and
> >> > > > >> the
> >> > > > >> > default spec of the VR being 1vcpu@256MB, considering it can
> >> be the
> >> > > > >> router
> >> > > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit
> ish...
> >> What
> >> > > would
> >> > > > >> > be your thought about this ?
> >> > > > >> >
> >> > > > >> > I'd be curious to have a LB offering in ACS where it would
> >> deploy a
> >> > > > >> > redundant traefik[1] beside the VR for doing http and https
> >> > > > >> Load-balancing.
> >> > > > >> > I think it would also be useful if the API of that traefik
> >> instance
> >> > > would
> >> > > > >> > be available from within the VPC or LBnetwork so is API
> would be
> >> > > > >> accessible
> >> > > > >> > to other apps orchestration tools such as  kubernetes or
> >> rancher.
> >> > > > >> >
> >> > > > >> > traefik or not, here is what I think is needed by cloudstack
> in
> >> the
> >> > > LB
> >> > > > >> > improvement:
> >> > > > >> >
> >> > > > >> > - support http, https (X-Forwarded-For)
> >> > > > >>
> >> > > > >> HAProxy also supports the PROXY protocol towards the backends.
> >> Apache
> >> > > > >> 2.4.22 supports this natively and Varnish for example can also
> >> talk
> >> > > PROXY.
> >> > > > >>
> >> > > > >> It adds a littlebit of metadata to the connection so that the
> >> backend
> >> > > > >> knows the original IP the connection came from for example:
> >> > > > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
> >> > > > >>
> >> > > > >> Wido
> >> > > > >>
> >> > > > >> > - basic persistence tuning (API already exist)
> >> > > > >> > - better backend monitoring, currently only a tcp connect
> >> validate
> >> > > if the
> >> > > > >> > webserver is up.
> >> > > > >> > - ssl offload
> >> > > > >> > - metric collection, more stats, maybe just export the tool
> >> status
> >> > > page
> >> > > > >> to
> >> > > > >> > the private network.
> >> > > > >> > - Container world support, right now if you have Rancher or
> >> > > kubernetes
> >> > > > >> > cluster, you need to deploy your own LB solution behing
> >> mostlikely a
> >> > > > >> static
> >> > > > >> > nat., If cloudstack would deploy a traefik instance, Kub or
> >> Rancher
> >> > > could
> >> > > > >> > reuse this instance and managed it to properly do LB between
> >> > > containers.
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > What would be your prefered LB tool:
> >> > > > >> > haproxy, traefik or nginx?
> >> > > > >> >
> >> > > > >> > CloudStack already have to code to handle SSL certs per
> >> projects and
> >> > > > >> > accounts if not mistaking because that code was added to
> support
> >> > > > >> NetScaler
> >> > > > >> > as Load-balancer in the past. so one less thing to think
> about
> >> :-)
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > [1] https://traefik.io/
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > PL,
> >> > > > >> >
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
> >> > > > >> >
> >> > > > >> > > Thanks Andrija,
> >> > > > >> > >
> >> > > > >> > > LB outside of the VR sounds like a good idea. An appliance
> >> based
> >> > > on,
> >> > > > >> say
> >> > > > >> > > cloud-init + ansible and so on could do the trick; alas
> it'd
> >> need
> >> > > to be
> >> > > > >> > > outside ACS.
> >> > > > >> > > I guess as users we could maybe come up with a spec for an
> >> > > > >> improvement, at
> >> > > > >> > > least we'd have something the devs could look at whenever
> it
> >> is
> >> > > > >> possible.
> >> > > > >> > >
> >> > > > >> > > Regards,
> >> > > > >> > > Lucian
> >> > > > >> > >
> >> > > > >> > > --
> >> > > > >> > > Sent from the Delta quadrant using Borg technology!
> >> > > > >> > >
> >> > > > >> > > Nux!
> >> > > > >> > > www.nux.ro
> >> > > > >> > >
> >> > > > >> > > ----- Original Message -----
> >> > > > >> > > > From: "Andrija Panic" <an...@gmail.com>
> >> > > > >> > > > To: "dev" <de...@cloudstack.apache.org>
> >> > > > >> > > > Cc: "users" <us...@cloudstack.apache.org>
> >> > > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
> >> > > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
> >> > > > >> > >
> >> > > > >> > > > We used to make some special stuff for one of the
> clients,
> >> > > where all
> >> > > > >> LB
> >> > > > >> > > > configuration work is done from outside of the ACS, i.e.
> >> python
> >> > > > >> script to
> >> > > > >> > > > feed/configure VR - install latest haproxy 1.5.x for
> >> transparent
> >> > > > >> proxy,
> >> > > > >> > > > since client insisted on SSL termination done on backend
> >> web SSL
> >> > > > >> > > servers....
> >> > > > >> > > > Not good idea, that is all I can say (custom
> configuration
> >> > > thing) -
> >> > > > >> but
> >> > > > >> > > the
> >> > > > >> > > > LB setup is actually good - transparent mode haproxy,
> works
> >> on
> >> > > TCP
> >> > > > >> level,
> >> > > > >> > > > so you can see "real client IP" on the backend servers
> >> (which
> >> > > must
> >> > > > >> use VR
> >> > > > >> > > > as the default gtw, as per default, so the whole setup
> works
> >> > > > >> properly).
> >> > > > >> > > >
> >> > > > >> > > > I'm still looking forward to see some special support of
> LB
> >> > > inside
> >> > > > >> VR via
> >> > > > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to
> >> enable LB
> >> > > > >> > > > provisioning SCRIPT (bash, or whatever),  where all
> needed
> >> > > > >> > > > install+configure can be done from client side  -
> otherwise
> >> > > covering
> >> > > > >> all
> >> > > > >> > > > user cases, with proper HTTP checks and similar....is
> >> > > impossible to
> >> > > > >> do
> >> > > > >> > > > IMHO.
> >> > > > >> > > >
> >> > > > >> > > > Some other clients, actually have internal FW appliance
> >> (i.e.
> >> > > > >> multihomed
> >> > > > >> > > > VM, acting as gtw for all VMs in all networks), and
> haproxy
> >> > > instaled
> >> > > > >> on
> >> > > > >> > > > this device (with NAT configured from VR to this internal
> >> > > FW/VM, so
> >> > > > >> > > remote
> >> > > > >> > > > IP can be seen properly) - this setup is fully under
> >> customer
> >> > > > >> control,
> >> > > > >> > > and
> >> > > > >> > > > can provide any kind of special haproxy config...
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> >> > > > >> > > >
> >> > > > >> > > >> Hello,
> >> > > > >> > > >>
> >> > > > >> > > >> Of the people running an LB (VR) with https backends,
> how
> >> do
> >> > > you
> >> > > > >> deal
> >> > > > >> > > with
> >> > > > >> > > >> the lack of x-forwarded-for since for port 443 there's
> just
> >> > > simple
> >> > > > >> TCP
> >> > > > >> > > >> balancing?
> >> > > > >> > > >>
> >> > > > >> > > >> Has anyone thought of terminating SSL in the VR instead?
> >> Ideas?
> >> > > > >> > > >>
> >> > > > >> > > >> Cheers
> >> > > > >> > > >>
> >> > > > >> > > >> --
> >> > > > >> > > >> Sent from the Delta quadrant using Borg technology!
> >> > > > >> > > >>
> >> > > > >> > > >> Nux!
> >> > > > >> > > >> www.nux.ro
> >> > > > >> > > >>
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > > --
> >> > > > >> > > >
> >> > > > >> > > > Andrija Panić
> >> > > > >> > >
> >> > > > >>
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > >
> >> > > > > Andrija Panić
> >> > >
>

Re: HTTPS LB and x-forwarded-for

Posted by Nux! <nu...@li.nux.ro>.
Pierre-Luc,

Haproxy docs say it should work for any kind of traffic as long as both ends are PROXY-aware and it look like a majority of software is.
So, in short, yes.

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Pierre-Luc Dion" <pd...@cloudops.com>
> To: "Wido den Hollander" <wi...@widodh.nl>
> Cc: "dev" <de...@cloudstack.apache.org>, "Khosrow Moossavi" <km...@cloudops.com>, "Will Stevens"
> <ws...@cloudops.com>, "Nux!" <nu...@li.nux.ro>, "users" <us...@cloudstack.apache.org>
> Sent: Friday, 10 November, 2017 15:32:38
> Subject: Re: HTTPS LB and x-forwarded-for

> Hi Wido, do you know if this would work for https traffic too?
> 
> Le 10 nov. 2017 09 h 35, "Wido den Hollander" <wi...@widodh.nl> a écrit :
> 
>>
>> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <pdion@cloudops.com
>> >:
>> >
>> >
>> > I kind of like the proxy backend type, ill check on our end if that would
>> > work but definitely a simple and efficient approach!
>> >
>>
>> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
>>
>> Apache HTTPd supports PROXY since 2.4.28: https://httpd.apache.org/docs/
>> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
>>
>> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
>>
>> Wido
>>
>> >
>> >
>> > Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wi...@widodh.nl> a écrit :
>> >
>> > >
>> > > > Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
>> > > >
>> > > >
>> > > > Wido,
>> > > >
>> > > > Excellent suggestion with the "transparent proxy", I was not aware of
>> > > that.
>> > > > I think that would be a great idea and wouldn't require too many
>> > > modifications, especially as Haproxy comes already with the VR.
>> > > >
>> > >
>> > > It's indeed just a matter of a HAProxy config setting. We could make it
>> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for
>> example.
>> > >
>> > > That way your problem would be solved.
>> > >
>> > > Wido
>> > >
>> > > > To Paul:
>> > > > - imho the LB solution ACS ships now is a bit handicaped since you do
>> > > not know the remote host ip. You're flying blind unless you use google
>> > > analytics (and these things have gotten more and more aggressively
>> filtered
>> > > by adblocks).
>> > > > Enhancing Haproxy as Wido suggested would go a long way, it wouldn't
>> > > break existing functionality and would also keep SSL processing off
>> the VR.
>> > > >
>> > > > --
>> > > > Sent from the Delta quadrant using Borg technology!
>> > > >
>> > > > Nux!
>> > > > www.nux.ro
>> > > >
>> > > > ----- Original Message -----
>> > > > > From: "Andrija Panic" <an...@gmail.com>
>> > > > > To: "users" <us...@cloudstack.apache.org>
>> > > > > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will Stevens" <
>> > > wstevens@cloudops.com>, "dev"
>> > > > > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <pdion@cloudops.com
>> >
>> > > > > Sent: Thursday, 9 November, 2017 13:10:58
>> > > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > > >
>> > > > > Wido,
>> > > > >
>> > > > > backend servers are not Linux only, for example we have a ton of
>> > > Windows
>> > > > > customers, some WEB solutions / IIS etc...
>> > > > >
>> > > > > @all - If we try to please/solve everyone's proxying
>> > > solution/requirement -
>> > > > > this is impossible IMHO - I'm thinking more about some "do it as
>> you
>> > > like"
>> > > > > solution, to let customer write his own haproxy config and upoad it
>> > > (for
>> > > > > example, or something better?).
>> > > > >
>> > > > > We can support newer version of haproxy (1.5+) which also implement
>> > > > > "transarent proxy" (integrate with kernel so to speak)  to allow
>> > > TCP-level
>> > > > > connections to backend (TCP mode, not HTTP mode) but to still
>> > > "preserve"
>> > > > > remote IP by faking it (fake soruce IP = transarent proxy).
>> > > > >
>> > > > > For the rest of configuration options,  I would leave it to the
>> > > customer
>> > > > > how he/she wants to configure rest of haproxy configuration,
>> inlcuding
>> > > > > custom checks, etc. Haproxy configuration is never-ending story,
>> and we
>> > > > > probably should allow custom sripts/configuration instead of
>> trying to
>> > > > > provide GUI/API way to configure everything (which is
>> impossible...)
>> > > > >
>> > > > > Just my 2 cents...
>> > > > >
>> > > > > On 9 November 2017 at 08:13, Wido den Hollander <wi...@widodh.nl>
>> > > wrote:
>> > > > >
>> > > > >>
>> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
>> > > pdion@cloudops.com
>> > > > >> >:
>> > > > >> >
>> > > > >> >
>> > > > >> > Same challenge here too!
>> > > > >> >
>> > > > >> > Let's look at improving Load-balancing offering from
>> cloudstack, I
>> > > guest
>> > > > >> we
>> > > > >> > should do a feature spec draft soon..,  from my perspective,
>> doing
>> > > SSL
>> > > > >> > offload on the VR could be problematic if the VR spec if too
>> small,
>> > > and
>> > > > >> the
>> > > > >> > default spec of the VR being 1vcpu@256MB, considering it can
>> be the
>> > > > >> router
>> > > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish...
>> What
>> > > would
>> > > > >> > be your thought about this ?
>> > > > >> >
>> > > > >> > I'd be curious to have a LB offering in ACS where it would
>> deploy a
>> > > > >> > redundant traefik[1] beside the VR for doing http and https
>> > > > >> Load-balancing.
>> > > > >> > I think it would also be useful if the API of that traefik
>> instance
>> > > would
>> > > > >> > be available from within the VPC or LBnetwork so is API would be
>> > > > >> accessible
>> > > > >> > to other apps orchestration tools such as  kubernetes or
>> rancher.
>> > > > >> >
>> > > > >> > traefik or not, here is what I think is needed by cloudstack in
>> the
>> > > LB
>> > > > >> > improvement:
>> > > > >> >
>> > > > >> > - support http, https (X-Forwarded-For)
>> > > > >>
>> > > > >> HAProxy also supports the PROXY protocol towards the backends.
>> Apache
>> > > > >> 2.4.22 supports this natively and Varnish for example can also
>> talk
>> > > PROXY.
>> > > > >>
>> > > > >> It adds a littlebit of metadata to the connection so that the
>> backend
>> > > > >> knows the original IP the connection came from for example:
>> > > > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
>> > > > >>
>> > > > >> Wido
>> > > > >>
>> > > > >> > - basic persistence tuning (API already exist)
>> > > > >> > - better backend monitoring, currently only a tcp connect
>> validate
>> > > if the
>> > > > >> > webserver is up.
>> > > > >> > - ssl offload
>> > > > >> > - metric collection, more stats, maybe just export the tool
>> status
>> > > page
>> > > > >> to
>> > > > >> > the private network.
>> > > > >> > - Container world support, right now if you have Rancher or
>> > > kubernetes
>> > > > >> > cluster, you need to deploy your own LB solution behing
>> mostlikely a
>> > > > >> static
>> > > > >> > nat., If cloudstack would deploy a traefik instance, Kub or
>> Rancher
>> > > could
>> > > > >> > reuse this instance and managed it to properly do LB between
>> > > containers.
>> > > > >> >
>> > > > >> >
>> > > > >> > What would be your prefered LB tool:
>> > > > >> > haproxy, traefik or nginx?
>> > > > >> >
>> > > > >> > CloudStack already have to code to handle SSL certs per
>> projects and
>> > > > >> > accounts if not mistaking because that code was added to support
>> > > > >> NetScaler
>> > > > >> > as Load-balancer in the past. so one less thing to think about
>> :-)
>> > > > >> >
>> > > > >> >
>> > > > >> > [1] https://traefik.io/
>> > > > >> >
>> > > > >> >
>> > > > >> > PL,
>> > > > >> >
>> > > > >> >
>> > > > >> >
>> > > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
>> > > > >> >
>> > > > >> > > Thanks Andrija,
>> > > > >> > >
>> > > > >> > > LB outside of the VR sounds like a good idea. An appliance
>> based
>> > > on,
>> > > > >> say
>> > > > >> > > cloud-init + ansible and so on could do the trick; alas it'd
>> need
>> > > to be
>> > > > >> > > outside ACS.
>> > > > >> > > I guess as users we could maybe come up with a spec for an
>> > > > >> improvement, at
>> > > > >> > > least we'd have something the devs could look at whenever it
>> is
>> > > > >> possible.
>> > > > >> > >
>> > > > >> > > Regards,
>> > > > >> > > Lucian
>> > > > >> > >
>> > > > >> > > --
>> > > > >> > > Sent from the Delta quadrant using Borg technology!
>> > > > >> > >
>> > > > >> > > Nux!
>> > > > >> > > www.nux.ro
>> > > > >> > >
>> > > > >> > > ----- Original Message -----
>> > > > >> > > > From: "Andrija Panic" <an...@gmail.com>
>> > > > >> > > > To: "dev" <de...@cloudstack.apache.org>
>> > > > >> > > > Cc: "users" <us...@cloudstack.apache.org>
>> > > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
>> > > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > > > >> > >
>> > > > >> > > > We used to make some special stuff for one of the clients,
>> > > where all
>> > > > >> LB
>> > > > >> > > > configuration work is done from outside of the ACS, i.e.
>> python
>> > > > >> script to
>> > > > >> > > > feed/configure VR - install latest haproxy 1.5.x for
>> transparent
>> > > > >> proxy,
>> > > > >> > > > since client insisted on SSL termination done on backend
>> web SSL
>> > > > >> > > servers....
>> > > > >> > > > Not good idea, that is all I can say (custom configuration
>> > > thing) -
>> > > > >> but
>> > > > >> > > the
>> > > > >> > > > LB setup is actually good - transparent mode haproxy, works
>> on
>> > > TCP
>> > > > >> level,
>> > > > >> > > > so you can see "real client IP" on the backend servers
>> (which
>> > > must
>> > > > >> use VR
>> > > > >> > > > as the default gtw, as per default, so the whole setup works
>> > > > >> properly).
>> > > > >> > > >
>> > > > >> > > > I'm still looking forward to see some special support of LB
>> > > inside
>> > > > >> VR via
>> > > > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to
>> enable LB
>> > > > >> > > > provisioning SCRIPT (bash, or whatever),  where all needed
>> > > > >> > > > install+configure can be done from client side  - otherwise
>> > > covering
>> > > > >> all
>> > > > >> > > > user cases, with proper HTTP checks and similar....is
>> > > impossible to
>> > > > >> do
>> > > > >> > > > IMHO.
>> > > > >> > > >
>> > > > >> > > > Some other clients, actually have internal FW appliance
>> (i.e.
>> > > > >> multihomed
>> > > > >> > > > VM, acting as gtw for all VMs in all networks), and haproxy
>> > > instaled
>> > > > >> on
>> > > > >> > > > this device (with NAT configured from VR to this internal
>> > > FW/VM, so
>> > > > >> > > remote
>> > > > >> > > > IP can be seen properly) - this setup is fully under
>> customer
>> > > > >> control,
>> > > > >> > > and
>> > > > >> > > > can provide any kind of special haproxy config...
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
>> > > > >> > > >
>> > > > >> > > >> Hello,
>> > > > >> > > >>
>> > > > >> > > >> Of the people running an LB (VR) with https backends, how
>> do
>> > > you
>> > > > >> deal
>> > > > >> > > with
>> > > > >> > > >> the lack of x-forwarded-for since for port 443 there's just
>> > > simple
>> > > > >> TCP
>> > > > >> > > >> balancing?
>> > > > >> > > >>
>> > > > >> > > >> Has anyone thought of terminating SSL in the VR instead?
>> Ideas?
>> > > > >> > > >>
>> > > > >> > > >> Cheers
>> > > > >> > > >>
>> > > > >> > > >> --
>> > > > >> > > >> Sent from the Delta quadrant using Borg technology!
>> > > > >> > > >>
>> > > > >> > > >> Nux!
>> > > > >> > > >> www.nux.ro
>> > > > >> > > >>
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > > --
>> > > > >> > > >
>> > > > >> > > > Andrija Panić
>> > > > >> > >
>> > > > >>
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > >
>> > > > > Andrija Panić
>> > >

Re: HTTPS LB and x-forwarded-for

Posted by Nux! <nu...@li.nux.ro>.
Pierre-Luc,

Haproxy docs say it should work for any kind of traffic as long as both ends are PROXY-aware and it look like a majority of software is.
So, in short, yes.

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Pierre-Luc Dion" <pd...@cloudops.com>
> To: "Wido den Hollander" <wi...@widodh.nl>
> Cc: "dev" <de...@cloudstack.apache.org>, "Khosrow Moossavi" <km...@cloudops.com>, "Will Stevens"
> <ws...@cloudops.com>, "Nux!" <nu...@li.nux.ro>, "users" <us...@cloudstack.apache.org>
> Sent: Friday, 10 November, 2017 15:32:38
> Subject: Re: HTTPS LB and x-forwarded-for

> Hi Wido, do you know if this would work for https traffic too?
> 
> Le 10 nov. 2017 09 h 35, "Wido den Hollander" <wi...@widodh.nl> a écrit :
> 
>>
>> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <pdion@cloudops.com
>> >:
>> >
>> >
>> > I kind of like the proxy backend type, ill check on our end if that would
>> > work but definitely a simple and efficient approach!
>> >
>>
>> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
>>
>> Apache HTTPd supports PROXY since 2.4.28: https://httpd.apache.org/docs/
>> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
>>
>> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
>>
>> Wido
>>
>> >
>> >
>> > Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wi...@widodh.nl> a écrit :
>> >
>> > >
>> > > > Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
>> > > >
>> > > >
>> > > > Wido,
>> > > >
>> > > > Excellent suggestion with the "transparent proxy", I was not aware of
>> > > that.
>> > > > I think that would be a great idea and wouldn't require too many
>> > > modifications, especially as Haproxy comes already with the VR.
>> > > >
>> > >
>> > > It's indeed just a matter of a HAProxy config setting. We could make it
>> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for
>> example.
>> > >
>> > > That way your problem would be solved.
>> > >
>> > > Wido
>> > >
>> > > > To Paul:
>> > > > - imho the LB solution ACS ships now is a bit handicaped since you do
>> > > not know the remote host ip. You're flying blind unless you use google
>> > > analytics (and these things have gotten more and more aggressively
>> filtered
>> > > by adblocks).
>> > > > Enhancing Haproxy as Wido suggested would go a long way, it wouldn't
>> > > break existing functionality and would also keep SSL processing off
>> the VR.
>> > > >
>> > > > --
>> > > > Sent from the Delta quadrant using Borg technology!
>> > > >
>> > > > Nux!
>> > > > www.nux.ro
>> > > >
>> > > > ----- Original Message -----
>> > > > > From: "Andrija Panic" <an...@gmail.com>
>> > > > > To: "users" <us...@cloudstack.apache.org>
>> > > > > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will Stevens" <
>> > > wstevens@cloudops.com>, "dev"
>> > > > > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <pdion@cloudops.com
>> >
>> > > > > Sent: Thursday, 9 November, 2017 13:10:58
>> > > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > > >
>> > > > > Wido,
>> > > > >
>> > > > > backend servers are not Linux only, for example we have a ton of
>> > > Windows
>> > > > > customers, some WEB solutions / IIS etc...
>> > > > >
>> > > > > @all - If we try to please/solve everyone's proxying
>> > > solution/requirement -
>> > > > > this is impossible IMHO - I'm thinking more about some "do it as
>> you
>> > > like"
>> > > > > solution, to let customer write his own haproxy config and upoad it
>> > > (for
>> > > > > example, or something better?).
>> > > > >
>> > > > > We can support newer version of haproxy (1.5+) which also implement
>> > > > > "transarent proxy" (integrate with kernel so to speak)  to allow
>> > > TCP-level
>> > > > > connections to backend (TCP mode, not HTTP mode) but to still
>> > > "preserve"
>> > > > > remote IP by faking it (fake soruce IP = transarent proxy).
>> > > > >
>> > > > > For the rest of configuration options,  I would leave it to the
>> > > customer
>> > > > > how he/she wants to configure rest of haproxy configuration,
>> inlcuding
>> > > > > custom checks, etc. Haproxy configuration is never-ending story,
>> and we
>> > > > > probably should allow custom sripts/configuration instead of
>> trying to
>> > > > > provide GUI/API way to configure everything (which is
>> impossible...)
>> > > > >
>> > > > > Just my 2 cents...
>> > > > >
>> > > > > On 9 November 2017 at 08:13, Wido den Hollander <wi...@widodh.nl>
>> > > wrote:
>> > > > >
>> > > > >>
>> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
>> > > pdion@cloudops.com
>> > > > >> >:
>> > > > >> >
>> > > > >> >
>> > > > >> > Same challenge here too!
>> > > > >> >
>> > > > >> > Let's look at improving Load-balancing offering from
>> cloudstack, I
>> > > guest
>> > > > >> we
>> > > > >> > should do a feature spec draft soon..,  from my perspective,
>> doing
>> > > SSL
>> > > > >> > offload on the VR could be problematic if the VR spec if too
>> small,
>> > > and
>> > > > >> the
>> > > > >> > default spec of the VR being 1vcpu@256MB, considering it can
>> be the
>> > > > >> router
>> > > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish...
>> What
>> > > would
>> > > > >> > be your thought about this ?
>> > > > >> >
>> > > > >> > I'd be curious to have a LB offering in ACS where it would
>> deploy a
>> > > > >> > redundant traefik[1] beside the VR for doing http and https
>> > > > >> Load-balancing.
>> > > > >> > I think it would also be useful if the API of that traefik
>> instance
>> > > would
>> > > > >> > be available from within the VPC or LBnetwork so is API would be
>> > > > >> accessible
>> > > > >> > to other apps orchestration tools such as  kubernetes or
>> rancher.
>> > > > >> >
>> > > > >> > traefik or not, here is what I think is needed by cloudstack in
>> the
>> > > LB
>> > > > >> > improvement:
>> > > > >> >
>> > > > >> > - support http, https (X-Forwarded-For)
>> > > > >>
>> > > > >> HAProxy also supports the PROXY protocol towards the backends.
>> Apache
>> > > > >> 2.4.22 supports this natively and Varnish for example can also
>> talk
>> > > PROXY.
>> > > > >>
>> > > > >> It adds a littlebit of metadata to the connection so that the
>> backend
>> > > > >> knows the original IP the connection came from for example:
>> > > > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
>> > > > >>
>> > > > >> Wido
>> > > > >>
>> > > > >> > - basic persistence tuning (API already exist)
>> > > > >> > - better backend monitoring, currently only a tcp connect
>> validate
>> > > if the
>> > > > >> > webserver is up.
>> > > > >> > - ssl offload
>> > > > >> > - metric collection, more stats, maybe just export the tool
>> status
>> > > page
>> > > > >> to
>> > > > >> > the private network.
>> > > > >> > - Container world support, right now if you have Rancher or
>> > > kubernetes
>> > > > >> > cluster, you need to deploy your own LB solution behing
>> mostlikely a
>> > > > >> static
>> > > > >> > nat., If cloudstack would deploy a traefik instance, Kub or
>> Rancher
>> > > could
>> > > > >> > reuse this instance and managed it to properly do LB between
>> > > containers.
>> > > > >> >
>> > > > >> >
>> > > > >> > What would be your prefered LB tool:
>> > > > >> > haproxy, traefik or nginx?
>> > > > >> >
>> > > > >> > CloudStack already have to code to handle SSL certs per
>> projects and
>> > > > >> > accounts if not mistaking because that code was added to support
>> > > > >> NetScaler
>> > > > >> > as Load-balancer in the past. so one less thing to think about
>> :-)
>> > > > >> >
>> > > > >> >
>> > > > >> > [1] https://traefik.io/
>> > > > >> >
>> > > > >> >
>> > > > >> > PL,
>> > > > >> >
>> > > > >> >
>> > > > >> >
>> > > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
>> > > > >> >
>> > > > >> > > Thanks Andrija,
>> > > > >> > >
>> > > > >> > > LB outside of the VR sounds like a good idea. An appliance
>> based
>> > > on,
>> > > > >> say
>> > > > >> > > cloud-init + ansible and so on could do the trick; alas it'd
>> need
>> > > to be
>> > > > >> > > outside ACS.
>> > > > >> > > I guess as users we could maybe come up with a spec for an
>> > > > >> improvement, at
>> > > > >> > > least we'd have something the devs could look at whenever it
>> is
>> > > > >> possible.
>> > > > >> > >
>> > > > >> > > Regards,
>> > > > >> > > Lucian
>> > > > >> > >
>> > > > >> > > --
>> > > > >> > > Sent from the Delta quadrant using Borg technology!
>> > > > >> > >
>> > > > >> > > Nux!
>> > > > >> > > www.nux.ro
>> > > > >> > >
>> > > > >> > > ----- Original Message -----
>> > > > >> > > > From: "Andrija Panic" <an...@gmail.com>
>> > > > >> > > > To: "dev" <de...@cloudstack.apache.org>
>> > > > >> > > > Cc: "users" <us...@cloudstack.apache.org>
>> > > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
>> > > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > > > >> > >
>> > > > >> > > > We used to make some special stuff for one of the clients,
>> > > where all
>> > > > >> LB
>> > > > >> > > > configuration work is done from outside of the ACS, i.e.
>> python
>> > > > >> script to
>> > > > >> > > > feed/configure VR - install latest haproxy 1.5.x for
>> transparent
>> > > > >> proxy,
>> > > > >> > > > since client insisted on SSL termination done on backend
>> web SSL
>> > > > >> > > servers....
>> > > > >> > > > Not good idea, that is all I can say (custom configuration
>> > > thing) -
>> > > > >> but
>> > > > >> > > the
>> > > > >> > > > LB setup is actually good - transparent mode haproxy, works
>> on
>> > > TCP
>> > > > >> level,
>> > > > >> > > > so you can see "real client IP" on the backend servers
>> (which
>> > > must
>> > > > >> use VR
>> > > > >> > > > as the default gtw, as per default, so the whole setup works
>> > > > >> properly).
>> > > > >> > > >
>> > > > >> > > > I'm still looking forward to see some special support of LB
>> > > inside
>> > > > >> VR via
>> > > > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to
>> enable LB
>> > > > >> > > > provisioning SCRIPT (bash, or whatever),  where all needed
>> > > > >> > > > install+configure can be done from client side  - otherwise
>> > > covering
>> > > > >> all
>> > > > >> > > > user cases, with proper HTTP checks and similar....is
>> > > impossible to
>> > > > >> do
>> > > > >> > > > IMHO.
>> > > > >> > > >
>> > > > >> > > > Some other clients, actually have internal FW appliance
>> (i.e.
>> > > > >> multihomed
>> > > > >> > > > VM, acting as gtw for all VMs in all networks), and haproxy
>> > > instaled
>> > > > >> on
>> > > > >> > > > this device (with NAT configured from VR to this internal
>> > > FW/VM, so
>> > > > >> > > remote
>> > > > >> > > > IP can be seen properly) - this setup is fully under
>> customer
>> > > > >> control,
>> > > > >> > > and
>> > > > >> > > > can provide any kind of special haproxy config...
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
>> > > > >> > > >
>> > > > >> > > >> Hello,
>> > > > >> > > >>
>> > > > >> > > >> Of the people running an LB (VR) with https backends, how
>> do
>> > > you
>> > > > >> deal
>> > > > >> > > with
>> > > > >> > > >> the lack of x-forwarded-for since for port 443 there's just
>> > > simple
>> > > > >> TCP
>> > > > >> > > >> balancing?
>> > > > >> > > >>
>> > > > >> > > >> Has anyone thought of terminating SSL in the VR instead?
>> Ideas?
>> > > > >> > > >>
>> > > > >> > > >> Cheers
>> > > > >> > > >>
>> > > > >> > > >> --
>> > > > >> > > >> Sent from the Delta quadrant using Borg technology!
>> > > > >> > > >>
>> > > > >> > > >> Nux!
>> > > > >> > > >> www.nux.ro
>> > > > >> > > >>
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > > --
>> > > > >> > > >
>> > > > >> > > > Andrija Panić
>> > > > >> > >
>> > > > >>
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > >
>> > > > > Andrija Panić
>> > >

Re: HTTPS LB and x-forwarded-for

Posted by Pierre-Luc Dion <pd...@cloudops.com>.
Hi Wido, do you know if this would work for https traffic too?

Le 10 nov. 2017 09 h 35, "Wido den Hollander" <wi...@widodh.nl> a écrit :

>
> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <pdion@cloudops.com
> >:
> >
> >
> > I kind of like the proxy backend type, ill check on our end if that would
> > work but definitely a simple and efficient approach!
> >
>
> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
>
> Apache HTTPd supports PROXY since 2.4.28: https://httpd.apache.org/docs/
> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
>
> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
>
> Wido
>
> >
> >
> > Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wi...@widodh.nl> a écrit :
> >
> > >
> > > > Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
> > > >
> > > >
> > > > Wido,
> > > >
> > > > Excellent suggestion with the "transparent proxy", I was not aware of
> > > that.
> > > > I think that would be a great idea and wouldn't require too many
> > > modifications, especially as Haproxy comes already with the VR.
> > > >
> > >
> > > It's indeed just a matter of a HAProxy config setting. We could make it
> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for
> example.
> > >
> > > That way your problem would be solved.
> > >
> > > Wido
> > >
> > > > To Paul:
> > > > - imho the LB solution ACS ships now is a bit handicaped since you do
> > > not know the remote host ip. You're flying blind unless you use google
> > > analytics (and these things have gotten more and more aggressively
> filtered
> > > by adblocks).
> > > > Enhancing Haproxy as Wido suggested would go a long way, it wouldn't
> > > break existing functionality and would also keep SSL processing off
> the VR.
> > > >
> > > > --
> > > > Sent from the Delta quadrant using Borg technology!
> > > >
> > > > Nux!
> > > > www.nux.ro
> > > >
> > > > ----- Original Message -----
> > > > > From: "Andrija Panic" <an...@gmail.com>
> > > > > To: "users" <us...@cloudstack.apache.org>
> > > > > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will Stevens" <
> > > wstevens@cloudops.com>, "dev"
> > > > > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <pdion@cloudops.com
> >
> > > > > Sent: Thursday, 9 November, 2017 13:10:58
> > > > > Subject: Re: HTTPS LB and x-forwarded-for
> > > >
> > > > > Wido,
> > > > >
> > > > > backend servers are not Linux only, for example we have a ton of
> > > Windows
> > > > > customers, some WEB solutions / IIS etc...
> > > > >
> > > > > @all - If we try to please/solve everyone's proxying
> > > solution/requirement -
> > > > > this is impossible IMHO - I'm thinking more about some "do it as
> you
> > > like"
> > > > > solution, to let customer write his own haproxy config and upoad it
> > > (for
> > > > > example, or something better?).
> > > > >
> > > > > We can support newer version of haproxy (1.5+) which also implement
> > > > > "transarent proxy" (integrate with kernel so to speak)  to allow
> > > TCP-level
> > > > > connections to backend (TCP mode, not HTTP mode) but to still
> > > "preserve"
> > > > > remote IP by faking it (fake soruce IP = transarent proxy).
> > > > >
> > > > > For the rest of configuration options,  I would leave it to the
> > > customer
> > > > > how he/she wants to configure rest of haproxy configuration,
> inlcuding
> > > > > custom checks, etc. Haproxy configuration is never-ending story,
> and we
> > > > > probably should allow custom sripts/configuration instead of
> trying to
> > > > > provide GUI/API way to configure everything (which is
> impossible...)
> > > > >
> > > > > Just my 2 cents...
> > > > >
> > > > > On 9 November 2017 at 08:13, Wido den Hollander <wi...@widodh.nl>
> > > wrote:
> > > > >
> > > > >>
> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
> > > pdion@cloudops.com
> > > > >> >:
> > > > >> >
> > > > >> >
> > > > >> > Same challenge here too!
> > > > >> >
> > > > >> > Let's look at improving Load-balancing offering from
> cloudstack, I
> > > guest
> > > > >> we
> > > > >> > should do a feature spec draft soon..,  from my perspective,
> doing
> > > SSL
> > > > >> > offload on the VR could be problematic if the VR spec if too
> small,
> > > and
> > > > >> the
> > > > >> > default spec of the VR being 1vcpu@256MB, considering it can
> be the
> > > > >> router
> > > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish...
> What
> > > would
> > > > >> > be your thought about this ?
> > > > >> >
> > > > >> > I'd be curious to have a LB offering in ACS where it would
> deploy a
> > > > >> > redundant traefik[1] beside the VR for doing http and https
> > > > >> Load-balancing.
> > > > >> > I think it would also be useful if the API of that traefik
> instance
> > > would
> > > > >> > be available from within the VPC or LBnetwork so is API would be
> > > > >> accessible
> > > > >> > to other apps orchestration tools such as  kubernetes or
> rancher.
> > > > >> >
> > > > >> > traefik or not, here is what I think is needed by cloudstack in
> the
> > > LB
> > > > >> > improvement:
> > > > >> >
> > > > >> > - support http, https (X-Forwarded-For)
> > > > >>
> > > > >> HAProxy also supports the PROXY protocol towards the backends.
> Apache
> > > > >> 2.4.22 supports this natively and Varnish for example can also
> talk
> > > PROXY.
> > > > >>
> > > > >> It adds a littlebit of metadata to the connection so that the
> backend
> > > > >> knows the original IP the connection came from for example:
> > > > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
> > > > >>
> > > > >> Wido
> > > > >>
> > > > >> > - basic persistence tuning (API already exist)
> > > > >> > - better backend monitoring, currently only a tcp connect
> validate
> > > if the
> > > > >> > webserver is up.
> > > > >> > - ssl offload
> > > > >> > - metric collection, more stats, maybe just export the tool
> status
> > > page
> > > > >> to
> > > > >> > the private network.
> > > > >> > - Container world support, right now if you have Rancher or
> > > kubernetes
> > > > >> > cluster, you need to deploy your own LB solution behing
> mostlikely a
> > > > >> static
> > > > >> > nat., If cloudstack would deploy a traefik instance, Kub or
> Rancher
> > > could
> > > > >> > reuse this instance and managed it to properly do LB between
> > > containers.
> > > > >> >
> > > > >> >
> > > > >> > What would be your prefered LB tool:
> > > > >> > haproxy, traefik or nginx?
> > > > >> >
> > > > >> > CloudStack already have to code to handle SSL certs per
> projects and
> > > > >> > accounts if not mistaking because that code was added to support
> > > > >> NetScaler
> > > > >> > as Load-balancer in the past. so one less thing to think about
> :-)
> > > > >> >
> > > > >> >
> > > > >> > [1] https://traefik.io/
> > > > >> >
> > > > >> >
> > > > >> > PL,
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
> > > > >> >
> > > > >> > > Thanks Andrija,
> > > > >> > >
> > > > >> > > LB outside of the VR sounds like a good idea. An appliance
> based
> > > on,
> > > > >> say
> > > > >> > > cloud-init + ansible and so on could do the trick; alas it'd
> need
> > > to be
> > > > >> > > outside ACS.
> > > > >> > > I guess as users we could maybe come up with a spec for an
> > > > >> improvement, at
> > > > >> > > least we'd have something the devs could look at whenever it
> is
> > > > >> possible.
> > > > >> > >
> > > > >> > > Regards,
> > > > >> > > Lucian
> > > > >> > >
> > > > >> > > --
> > > > >> > > Sent from the Delta quadrant using Borg technology!
> > > > >> > >
> > > > >> > > Nux!
> > > > >> > > www.nux.ro
> > > > >> > >
> > > > >> > > ----- Original Message -----
> > > > >> > > > From: "Andrija Panic" <an...@gmail.com>
> > > > >> > > > To: "dev" <de...@cloudstack.apache.org>
> > > > >> > > > Cc: "users" <us...@cloudstack.apache.org>
> > > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
> > > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > > > >> > >
> > > > >> > > > We used to make some special stuff for one of the clients,
> > > where all
> > > > >> LB
> > > > >> > > > configuration work is done from outside of the ACS, i.e.
> python
> > > > >> script to
> > > > >> > > > feed/configure VR - install latest haproxy 1.5.x for
> transparent
> > > > >> proxy,
> > > > >> > > > since client insisted on SSL termination done on backend
> web SSL
> > > > >> > > servers....
> > > > >> > > > Not good idea, that is all I can say (custom configuration
> > > thing) -
> > > > >> but
> > > > >> > > the
> > > > >> > > > LB setup is actually good - transparent mode haproxy, works
> on
> > > TCP
> > > > >> level,
> > > > >> > > > so you can see "real client IP" on the backend servers
> (which
> > > must
> > > > >> use VR
> > > > >> > > > as the default gtw, as per default, so the whole setup works
> > > > >> properly).
> > > > >> > > >
> > > > >> > > > I'm still looking forward to see some special support of LB
> > > inside
> > > > >> VR via
> > > > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to
> enable LB
> > > > >> > > > provisioning SCRIPT (bash, or whatever),  where all needed
> > > > >> > > > install+configure can be done from client side  - otherwise
> > > covering
> > > > >> all
> > > > >> > > > user cases, with proper HTTP checks and similar....is
> > > impossible to
> > > > >> do
> > > > >> > > > IMHO.
> > > > >> > > >
> > > > >> > > > Some other clients, actually have internal FW appliance
> (i.e.
> > > > >> multihomed
> > > > >> > > > VM, acting as gtw for all VMs in all networks), and haproxy
> > > instaled
> > > > >> on
> > > > >> > > > this device (with NAT configured from VR to this internal
> > > FW/VM, so
> > > > >> > > remote
> > > > >> > > > IP can be seen properly) - this setup is fully under
> customer
> > > > >> control,
> > > > >> > > and
> > > > >> > > > can provide any kind of special haproxy config...
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> > > > >> > > >
> > > > >> > > >> Hello,
> > > > >> > > >>
> > > > >> > > >> Of the people running an LB (VR) with https backends, how
> do
> > > you
> > > > >> deal
> > > > >> > > with
> > > > >> > > >> the lack of x-forwarded-for since for port 443 there's just
> > > simple
> > > > >> TCP
> > > > >> > > >> balancing?
> > > > >> > > >>
> > > > >> > > >> Has anyone thought of terminating SSL in the VR instead?
> Ideas?
> > > > >> > > >>
> > > > >> > > >> Cheers
> > > > >> > > >>
> > > > >> > > >> --
> > > > >> > > >> Sent from the Delta quadrant using Borg technology!
> > > > >> > > >>
> > > > >> > > >> Nux!
> > > > >> > > >> www.nux.ro
> > > > >> > > >>
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > --
> > > > >> > > >
> > > > >> > > > Andrija Panić
> > > > >> > >
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Andrija Panić
> > >
>

Re: HTTPS LB and x-forwarded-for

Posted by Pierre-Luc Dion <pd...@cloudops.com>.
Hi Wido, do you know if this would work for https traffic too?

Le 10 nov. 2017 09 h 35, "Wido den Hollander" <wi...@widodh.nl> a écrit :

>
> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <pdion@cloudops.com
> >:
> >
> >
> > I kind of like the proxy backend type, ill check on our end if that would
> > work but definitely a simple and efficient approach!
> >
>
> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
>
> Apache HTTPd supports PROXY since 2.4.28: https://httpd.apache.org/docs/
> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
>
> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
>
> Wido
>
> >
> >
> > Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wi...@widodh.nl> a écrit :
> >
> > >
> > > > Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
> > > >
> > > >
> > > > Wido,
> > > >
> > > > Excellent suggestion with the "transparent proxy", I was not aware of
> > > that.
> > > > I think that would be a great idea and wouldn't require too many
> > > modifications, especially as Haproxy comes already with the VR.
> > > >
> > >
> > > It's indeed just a matter of a HAProxy config setting. We could make it
> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for
> example.
> > >
> > > That way your problem would be solved.
> > >
> > > Wido
> > >
> > > > To Paul:
> > > > - imho the LB solution ACS ships now is a bit handicaped since you do
> > > not know the remote host ip. You're flying blind unless you use google
> > > analytics (and these things have gotten more and more aggressively
> filtered
> > > by adblocks).
> > > > Enhancing Haproxy as Wido suggested would go a long way, it wouldn't
> > > break existing functionality and would also keep SSL processing off
> the VR.
> > > >
> > > > --
> > > > Sent from the Delta quadrant using Borg technology!
> > > >
> > > > Nux!
> > > > www.nux.ro
> > > >
> > > > ----- Original Message -----
> > > > > From: "Andrija Panic" <an...@gmail.com>
> > > > > To: "users" <us...@cloudstack.apache.org>
> > > > > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will Stevens" <
> > > wstevens@cloudops.com>, "dev"
> > > > > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <pdion@cloudops.com
> >
> > > > > Sent: Thursday, 9 November, 2017 13:10:58
> > > > > Subject: Re: HTTPS LB and x-forwarded-for
> > > >
> > > > > Wido,
> > > > >
> > > > > backend servers are not Linux only, for example we have a ton of
> > > Windows
> > > > > customers, some WEB solutions / IIS etc...
> > > > >
> > > > > @all - If we try to please/solve everyone's proxying
> > > solution/requirement -
> > > > > this is impossible IMHO - I'm thinking more about some "do it as
> you
> > > like"
> > > > > solution, to let customer write his own haproxy config and upoad it
> > > (for
> > > > > example, or something better?).
> > > > >
> > > > > We can support newer version of haproxy (1.5+) which also implement
> > > > > "transarent proxy" (integrate with kernel so to speak)  to allow
> > > TCP-level
> > > > > connections to backend (TCP mode, not HTTP mode) but to still
> > > "preserve"
> > > > > remote IP by faking it (fake soruce IP = transarent proxy).
> > > > >
> > > > > For the rest of configuration options,  I would leave it to the
> > > customer
> > > > > how he/she wants to configure rest of haproxy configuration,
> inlcuding
> > > > > custom checks, etc. Haproxy configuration is never-ending story,
> and we
> > > > > probably should allow custom sripts/configuration instead of
> trying to
> > > > > provide GUI/API way to configure everything (which is
> impossible...)
> > > > >
> > > > > Just my 2 cents...
> > > > >
> > > > > On 9 November 2017 at 08:13, Wido den Hollander <wi...@widodh.nl>
> > > wrote:
> > > > >
> > > > >>
> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
> > > pdion@cloudops.com
> > > > >> >:
> > > > >> >
> > > > >> >
> > > > >> > Same challenge here too!
> > > > >> >
> > > > >> > Let's look at improving Load-balancing offering from
> cloudstack, I
> > > guest
> > > > >> we
> > > > >> > should do a feature spec draft soon..,  from my perspective,
> doing
> > > SSL
> > > > >> > offload on the VR could be problematic if the VR spec if too
> small,
> > > and
> > > > >> the
> > > > >> > default spec of the VR being 1vcpu@256MB, considering it can
> be the
> > > > >> router
> > > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish...
> What
> > > would
> > > > >> > be your thought about this ?
> > > > >> >
> > > > >> > I'd be curious to have a LB offering in ACS where it would
> deploy a
> > > > >> > redundant traefik[1] beside the VR for doing http and https
> > > > >> Load-balancing.
> > > > >> > I think it would also be useful if the API of that traefik
> instance
> > > would
> > > > >> > be available from within the VPC or LBnetwork so is API would be
> > > > >> accessible
> > > > >> > to other apps orchestration tools such as  kubernetes or
> rancher.
> > > > >> >
> > > > >> > traefik or not, here is what I think is needed by cloudstack in
> the
> > > LB
> > > > >> > improvement:
> > > > >> >
> > > > >> > - support http, https (X-Forwarded-For)
> > > > >>
> > > > >> HAProxy also supports the PROXY protocol towards the backends.
> Apache
> > > > >> 2.4.22 supports this natively and Varnish for example can also
> talk
> > > PROXY.
> > > > >>
> > > > >> It adds a littlebit of metadata to the connection so that the
> backend
> > > > >> knows the original IP the connection came from for example:
> > > > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
> > > > >>
> > > > >> Wido
> > > > >>
> > > > >> > - basic persistence tuning (API already exist)
> > > > >> > - better backend monitoring, currently only a tcp connect
> validate
> > > if the
> > > > >> > webserver is up.
> > > > >> > - ssl offload
> > > > >> > - metric collection, more stats, maybe just export the tool
> status
> > > page
> > > > >> to
> > > > >> > the private network.
> > > > >> > - Container world support, right now if you have Rancher or
> > > kubernetes
> > > > >> > cluster, you need to deploy your own LB solution behing
> mostlikely a
> > > > >> static
> > > > >> > nat., If cloudstack would deploy a traefik instance, Kub or
> Rancher
> > > could
> > > > >> > reuse this instance and managed it to properly do LB between
> > > containers.
> > > > >> >
> > > > >> >
> > > > >> > What would be your prefered LB tool:
> > > > >> > haproxy, traefik or nginx?
> > > > >> >
> > > > >> > CloudStack already have to code to handle SSL certs per
> projects and
> > > > >> > accounts if not mistaking because that code was added to support
> > > > >> NetScaler
> > > > >> > as Load-balancer in the past. so one less thing to think about
> :-)
> > > > >> >
> > > > >> >
> > > > >> > [1] https://traefik.io/
> > > > >> >
> > > > >> >
> > > > >> > PL,
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
> > > > >> >
> > > > >> > > Thanks Andrija,
> > > > >> > >
> > > > >> > > LB outside of the VR sounds like a good idea. An appliance
> based
> > > on,
> > > > >> say
> > > > >> > > cloud-init + ansible and so on could do the trick; alas it'd
> need
> > > to be
> > > > >> > > outside ACS.
> > > > >> > > I guess as users we could maybe come up with a spec for an
> > > > >> improvement, at
> > > > >> > > least we'd have something the devs could look at whenever it
> is
> > > > >> possible.
> > > > >> > >
> > > > >> > > Regards,
> > > > >> > > Lucian
> > > > >> > >
> > > > >> > > --
> > > > >> > > Sent from the Delta quadrant using Borg technology!
> > > > >> > >
> > > > >> > > Nux!
> > > > >> > > www.nux.ro
> > > > >> > >
> > > > >> > > ----- Original Message -----
> > > > >> > > > From: "Andrija Panic" <an...@gmail.com>
> > > > >> > > > To: "dev" <de...@cloudstack.apache.org>
> > > > >> > > > Cc: "users" <us...@cloudstack.apache.org>
> > > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
> > > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > > > >> > >
> > > > >> > > > We used to make some special stuff for one of the clients,
> > > where all
> > > > >> LB
> > > > >> > > > configuration work is done from outside of the ACS, i.e.
> python
> > > > >> script to
> > > > >> > > > feed/configure VR - install latest haproxy 1.5.x for
> transparent
> > > > >> proxy,
> > > > >> > > > since client insisted on SSL termination done on backend
> web SSL
> > > > >> > > servers....
> > > > >> > > > Not good idea, that is all I can say (custom configuration
> > > thing) -
> > > > >> but
> > > > >> > > the
> > > > >> > > > LB setup is actually good - transparent mode haproxy, works
> on
> > > TCP
> > > > >> level,
> > > > >> > > > so you can see "real client IP" on the backend servers
> (which
> > > must
> > > > >> use VR
> > > > >> > > > as the default gtw, as per default, so the whole setup works
> > > > >> properly).
> > > > >> > > >
> > > > >> > > > I'm still looking forward to see some special support of LB
> > > inside
> > > > >> VR via
> > > > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to
> enable LB
> > > > >> > > > provisioning SCRIPT (bash, or whatever),  where all needed
> > > > >> > > > install+configure can be done from client side  - otherwise
> > > covering
> > > > >> all
> > > > >> > > > user cases, with proper HTTP checks and similar....is
> > > impossible to
> > > > >> do
> > > > >> > > > IMHO.
> > > > >> > > >
> > > > >> > > > Some other clients, actually have internal FW appliance
> (i.e.
> > > > >> multihomed
> > > > >> > > > VM, acting as gtw for all VMs in all networks), and haproxy
> > > instaled
> > > > >> on
> > > > >> > > > this device (with NAT configured from VR to this internal
> > > FW/VM, so
> > > > >> > > remote
> > > > >> > > > IP can be seen properly) - this setup is fully under
> customer
> > > > >> control,
> > > > >> > > and
> > > > >> > > > can provide any kind of special haproxy config...
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> > > > >> > > >
> > > > >> > > >> Hello,
> > > > >> > > >>
> > > > >> > > >> Of the people running an LB (VR) with https backends, how
> do
> > > you
> > > > >> deal
> > > > >> > > with
> > > > >> > > >> the lack of x-forwarded-for since for port 443 there's just
> > > simple
> > > > >> TCP
> > > > >> > > >> balancing?
> > > > >> > > >>
> > > > >> > > >> Has anyone thought of terminating SSL in the VR instead?
> Ideas?
> > > > >> > > >>
> > > > >> > > >> Cheers
> > > > >> > > >>
> > > > >> > > >> --
> > > > >> > > >> Sent from the Delta quadrant using Borg technology!
> > > > >> > > >>
> > > > >> > > >> Nux!
> > > > >> > > >> www.nux.ro
> > > > >> > > >>
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > --
> > > > >> > > >
> > > > >> > > > Andrija Panić
> > > > >> > >
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Andrija Panić
> > >
>

Re: HTTPS LB and x-forwarded-for

Posted by Wido den Hollander <wi...@widodh.nl>.
> Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <pd...@cloudops.com>:
> 
> 
> I kind of like the proxy backend type, ill check on our end if that would
> work but definitely a simple and efficient approach!
> 

See: https://www.haproxy.com/blog/haproxy/proxy-protocol/

Apache HTTPd supports PROXY since 2.4.28: https://httpd.apache.org/docs/trunk/mod/mod_remoteip.html#remoteipproxyprotocol

"RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"

Wido

> 
> 
> Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wi...@widodh.nl> a écrit :
> 
> >
> > > Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
> > >
> > >
> > > Wido,
> > >
> > > Excellent suggestion with the "transparent proxy", I was not aware of
> > that.
> > > I think that would be a great idea and wouldn't require too many
> > modifications, especially as Haproxy comes already with the VR.
> > >
> >
> > It's indeed just a matter of a HAProxy config setting. We could make it
> > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for example.
> >
> > That way your problem would be solved.
> >
> > Wido
> >
> > > To Paul:
> > > - imho the LB solution ACS ships now is a bit handicaped since you do
> > not know the remote host ip. You're flying blind unless you use google
> > analytics (and these things have gotten more and more aggressively filtered
> > by adblocks).
> > > Enhancing Haproxy as Wido suggested would go a long way, it wouldn't
> > break existing functionality and would also keep SSL processing off the VR.
> > >
> > > --
> > > Sent from the Delta quadrant using Borg technology!
> > >
> > > Nux!
> > > www.nux.ro
> > >
> > > ----- Original Message -----
> > > > From: "Andrija Panic" <an...@gmail.com>
> > > > To: "users" <us...@cloudstack.apache.org>
> > > > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will Stevens" <
> > wstevens@cloudops.com>, "dev"
> > > > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <pd...@cloudops.com>
> > > > Sent: Thursday, 9 November, 2017 13:10:58
> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > >
> > > > Wido,
> > > >
> > > > backend servers are not Linux only, for example we have a ton of
> > Windows
> > > > customers, some WEB solutions / IIS etc...
> > > >
> > > > @all - If we try to please/solve everyone's proxying
> > solution/requirement -
> > > > this is impossible IMHO - I'm thinking more about some "do it as you
> > like"
> > > > solution, to let customer write his own haproxy config and upoad it
> > (for
> > > > example, or something better?).
> > > >
> > > > We can support newer version of haproxy (1.5+) which also implement
> > > > "transarent proxy" (integrate with kernel so to speak)  to allow
> > TCP-level
> > > > connections to backend (TCP mode, not HTTP mode) but to still
> > "preserve"
> > > > remote IP by faking it (fake soruce IP = transarent proxy).
> > > >
> > > > For the rest of configuration options,  I would leave it to the
> > customer
> > > > how he/she wants to configure rest of haproxy configuration, inlcuding
> > > > custom checks, etc. Haproxy configuration is never-ending story, and we
> > > > probably should allow custom sripts/configuration instead of trying to
> > > > provide GUI/API way to configure everything (which is impossible...)
> > > >
> > > > Just my 2 cents...
> > > >
> > > > On 9 November 2017 at 08:13, Wido den Hollander <wi...@widodh.nl>
> > wrote:
> > > >
> > > >>
> > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
> > pdion@cloudops.com
> > > >> >:
> > > >> >
> > > >> >
> > > >> > Same challenge here too!
> > > >> >
> > > >> > Let's look at improving Load-balancing offering from cloudstack, I
> > guest
> > > >> we
> > > >> > should do a feature spec draft soon..,  from my perspective, doing
> > SSL
> > > >> > offload on the VR could be problematic if the VR spec if too small,
> > and
> > > >> the
> > > >> > default spec of the VR being 1vcpu@256MB, considering it can be the
> > > >> router
> > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What
> > would
> > > >> > be your thought about this ?
> > > >> >
> > > >> > I'd be curious to have a LB offering in ACS where it would deploy a
> > > >> > redundant traefik[1] beside the VR for doing http and https
> > > >> Load-balancing.
> > > >> > I think it would also be useful if the API of that traefik instance
> > would
> > > >> > be available from within the VPC or LBnetwork so is API would be
> > > >> accessible
> > > >> > to other apps orchestration tools such as  kubernetes or rancher.
> > > >> >
> > > >> > traefik or not, here is what I think is needed by cloudstack in the
> > LB
> > > >> > improvement:
> > > >> >
> > > >> > - support http, https (X-Forwarded-For)
> > > >>
> > > >> HAProxy also supports the PROXY protocol towards the backends. Apache
> > > >> 2.4.22 supports this natively and Varnish for example can also talk
> > PROXY.
> > > >>
> > > >> It adds a littlebit of metadata to the connection so that the backend
> > > >> knows the original IP the connection came from for example:
> > > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
> > > >>
> > > >> Wido
> > > >>
> > > >> > - basic persistence tuning (API already exist)
> > > >> > - better backend monitoring, currently only a tcp connect validate
> > if the
> > > >> > webserver is up.
> > > >> > - ssl offload
> > > >> > - metric collection, more stats, maybe just export the tool status
> > page
> > > >> to
> > > >> > the private network.
> > > >> > - Container world support, right now if you have Rancher or
> > kubernetes
> > > >> > cluster, you need to deploy your own LB solution behing mostlikely a
> > > >> static
> > > >> > nat., If cloudstack would deploy a traefik instance, Kub or Rancher
> > could
> > > >> > reuse this instance and managed it to properly do LB between
> > containers.
> > > >> >
> > > >> >
> > > >> > What would be your prefered LB tool:
> > > >> > haproxy, traefik or nginx?
> > > >> >
> > > >> > CloudStack already have to code to handle SSL certs per projects and
> > > >> > accounts if not mistaking because that code was added to support
> > > >> NetScaler
> > > >> > as Load-balancer in the past. so one less thing to think about :-)
> > > >> >
> > > >> >
> > > >> > [1] https://traefik.io/
> > > >> >
> > > >> >
> > > >> > PL,
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
> > > >> >
> > > >> > > Thanks Andrija,
> > > >> > >
> > > >> > > LB outside of the VR sounds like a good idea. An appliance based
> > on,
> > > >> say
> > > >> > > cloud-init + ansible and so on could do the trick; alas it'd need
> > to be
> > > >> > > outside ACS.
> > > >> > > I guess as users we could maybe come up with a spec for an
> > > >> improvement, at
> > > >> > > least we'd have something the devs could look at whenever it is
> > > >> possible.
> > > >> > >
> > > >> > > Regards,
> > > >> > > Lucian
> > > >> > >
> > > >> > > --
> > > >> > > Sent from the Delta quadrant using Borg technology!
> > > >> > >
> > > >> > > Nux!
> > > >> > > www.nux.ro
> > > >> > >
> > > >> > > ----- Original Message -----
> > > >> > > > From: "Andrija Panic" <an...@gmail.com>
> > > >> > > > To: "dev" <de...@cloudstack.apache.org>
> > > >> > > > Cc: "users" <us...@cloudstack.apache.org>
> > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
> > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > > >> > >
> > > >> > > > We used to make some special stuff for one of the clients,
> > where all
> > > >> LB
> > > >> > > > configuration work is done from outside of the ACS, i.e. python
> > > >> script to
> > > >> > > > feed/configure VR - install latest haproxy 1.5.x for transparent
> > > >> proxy,
> > > >> > > > since client insisted on SSL termination done on backend web SSL
> > > >> > > servers....
> > > >> > > > Not good idea, that is all I can say (custom configuration
> > thing) -
> > > >> but
> > > >> > > the
> > > >> > > > LB setup is actually good - transparent mode haproxy, works on
> > TCP
> > > >> level,
> > > >> > > > so you can see "real client IP" on the backend servers (which
> > must
> > > >> use VR
> > > >> > > > as the default gtw, as per default, so the whole setup works
> > > >> properly).
> > > >> > > >
> > > >> > > > I'm still looking forward to see some special support of LB
> > inside
> > > >> VR via
> > > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
> > > >> > > > provisioning SCRIPT (bash, or whatever),  where all needed
> > > >> > > > install+configure can be done from client side  - otherwise
> > covering
> > > >> all
> > > >> > > > user cases, with proper HTTP checks and similar....is
> > impossible to
> > > >> do
> > > >> > > > IMHO.
> > > >> > > >
> > > >> > > > Some other clients, actually have internal FW appliance (i.e.
> > > >> multihomed
> > > >> > > > VM, acting as gtw for all VMs in all networks), and haproxy
> > instaled
> > > >> on
> > > >> > > > this device (with NAT configured from VR to this internal
> > FW/VM, so
> > > >> > > remote
> > > >> > > > IP can be seen properly) - this setup is fully under customer
> > > >> control,
> > > >> > > and
> > > >> > > > can provide any kind of special haproxy config...
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> > > >> > > >
> > > >> > > >> Hello,
> > > >> > > >>
> > > >> > > >> Of the people running an LB (VR) with https backends, how do
> > you
> > > >> deal
> > > >> > > with
> > > >> > > >> the lack of x-forwarded-for since for port 443 there's just
> > simple
> > > >> TCP
> > > >> > > >> balancing?
> > > >> > > >>
> > > >> > > >> Has anyone thought of terminating SSL in the VR instead? Ideas?
> > > >> > > >>
> > > >> > > >> Cheers
> > > >> > > >>
> > > >> > > >> --
> > > >> > > >> Sent from the Delta quadrant using Borg technology!
> > > >> > > >>
> > > >> > > >> Nux!
> > > >> > > >> www.nux.ro
> > > >> > > >>
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > --
> > > >> > > >
> > > >> > > > Andrija Panić
> > > >> > >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Andrija Panić
> >

Re: HTTPS LB and x-forwarded-for

Posted by Wido den Hollander <wi...@widodh.nl>.
> Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <pd...@cloudops.com>:
> 
> 
> I kind of like the proxy backend type, ill check on our end if that would
> work but definitely a simple and efficient approach!
> 

See: https://www.haproxy.com/blog/haproxy/proxy-protocol/

Apache HTTPd supports PROXY since 2.4.28: https://httpd.apache.org/docs/trunk/mod/mod_remoteip.html#remoteipproxyprotocol

"RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"

Wido

> 
> 
> Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wi...@widodh.nl> a écrit :
> 
> >
> > > Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
> > >
> > >
> > > Wido,
> > >
> > > Excellent suggestion with the "transparent proxy", I was not aware of
> > that.
> > > I think that would be a great idea and wouldn't require too many
> > modifications, especially as Haproxy comes already with the VR.
> > >
> >
> > It's indeed just a matter of a HAProxy config setting. We could make it
> > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for example.
> >
> > That way your problem would be solved.
> >
> > Wido
> >
> > > To Paul:
> > > - imho the LB solution ACS ships now is a bit handicaped since you do
> > not know the remote host ip. You're flying blind unless you use google
> > analytics (and these things have gotten more and more aggressively filtered
> > by adblocks).
> > > Enhancing Haproxy as Wido suggested would go a long way, it wouldn't
> > break existing functionality and would also keep SSL processing off the VR.
> > >
> > > --
> > > Sent from the Delta quadrant using Borg technology!
> > >
> > > Nux!
> > > www.nux.ro
> > >
> > > ----- Original Message -----
> > > > From: "Andrija Panic" <an...@gmail.com>
> > > > To: "users" <us...@cloudstack.apache.org>
> > > > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will Stevens" <
> > wstevens@cloudops.com>, "dev"
> > > > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <pd...@cloudops.com>
> > > > Sent: Thursday, 9 November, 2017 13:10:58
> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > >
> > > > Wido,
> > > >
> > > > backend servers are not Linux only, for example we have a ton of
> > Windows
> > > > customers, some WEB solutions / IIS etc...
> > > >
> > > > @all - If we try to please/solve everyone's proxying
> > solution/requirement -
> > > > this is impossible IMHO - I'm thinking more about some "do it as you
> > like"
> > > > solution, to let customer write his own haproxy config and upoad it
> > (for
> > > > example, or something better?).
> > > >
> > > > We can support newer version of haproxy (1.5+) which also implement
> > > > "transarent proxy" (integrate with kernel so to speak)  to allow
> > TCP-level
> > > > connections to backend (TCP mode, not HTTP mode) but to still
> > "preserve"
> > > > remote IP by faking it (fake soruce IP = transarent proxy).
> > > >
> > > > For the rest of configuration options,  I would leave it to the
> > customer
> > > > how he/she wants to configure rest of haproxy configuration, inlcuding
> > > > custom checks, etc. Haproxy configuration is never-ending story, and we
> > > > probably should allow custom sripts/configuration instead of trying to
> > > > provide GUI/API way to configure everything (which is impossible...)
> > > >
> > > > Just my 2 cents...
> > > >
> > > > On 9 November 2017 at 08:13, Wido den Hollander <wi...@widodh.nl>
> > wrote:
> > > >
> > > >>
> > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
> > pdion@cloudops.com
> > > >> >:
> > > >> >
> > > >> >
> > > >> > Same challenge here too!
> > > >> >
> > > >> > Let's look at improving Load-balancing offering from cloudstack, I
> > guest
> > > >> we
> > > >> > should do a feature spec draft soon..,  from my perspective, doing
> > SSL
> > > >> > offload on the VR could be problematic if the VR spec if too small,
> > and
> > > >> the
> > > >> > default spec of the VR being 1vcpu@256MB, considering it can be the
> > > >> router
> > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What
> > would
> > > >> > be your thought about this ?
> > > >> >
> > > >> > I'd be curious to have a LB offering in ACS where it would deploy a
> > > >> > redundant traefik[1] beside the VR for doing http and https
> > > >> Load-balancing.
> > > >> > I think it would also be useful if the API of that traefik instance
> > would
> > > >> > be available from within the VPC or LBnetwork so is API would be
> > > >> accessible
> > > >> > to other apps orchestration tools such as  kubernetes or rancher.
> > > >> >
> > > >> > traefik or not, here is what I think is needed by cloudstack in the
> > LB
> > > >> > improvement:
> > > >> >
> > > >> > - support http, https (X-Forwarded-For)
> > > >>
> > > >> HAProxy also supports the PROXY protocol towards the backends. Apache
> > > >> 2.4.22 supports this natively and Varnish for example can also talk
> > PROXY.
> > > >>
> > > >> It adds a littlebit of metadata to the connection so that the backend
> > > >> knows the original IP the connection came from for example:
> > > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
> > > >>
> > > >> Wido
> > > >>
> > > >> > - basic persistence tuning (API already exist)
> > > >> > - better backend monitoring, currently only a tcp connect validate
> > if the
> > > >> > webserver is up.
> > > >> > - ssl offload
> > > >> > - metric collection, more stats, maybe just export the tool status
> > page
> > > >> to
> > > >> > the private network.
> > > >> > - Container world support, right now if you have Rancher or
> > kubernetes
> > > >> > cluster, you need to deploy your own LB solution behing mostlikely a
> > > >> static
> > > >> > nat., If cloudstack would deploy a traefik instance, Kub or Rancher
> > could
> > > >> > reuse this instance and managed it to properly do LB between
> > containers.
> > > >> >
> > > >> >
> > > >> > What would be your prefered LB tool:
> > > >> > haproxy, traefik or nginx?
> > > >> >
> > > >> > CloudStack already have to code to handle SSL certs per projects and
> > > >> > accounts if not mistaking because that code was added to support
> > > >> NetScaler
> > > >> > as Load-balancer in the past. so one less thing to think about :-)
> > > >> >
> > > >> >
> > > >> > [1] https://traefik.io/
> > > >> >
> > > >> >
> > > >> > PL,
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
> > > >> >
> > > >> > > Thanks Andrija,
> > > >> > >
> > > >> > > LB outside of the VR sounds like a good idea. An appliance based
> > on,
> > > >> say
> > > >> > > cloud-init + ansible and so on could do the trick; alas it'd need
> > to be
> > > >> > > outside ACS.
> > > >> > > I guess as users we could maybe come up with a spec for an
> > > >> improvement, at
> > > >> > > least we'd have something the devs could look at whenever it is
> > > >> possible.
> > > >> > >
> > > >> > > Regards,
> > > >> > > Lucian
> > > >> > >
> > > >> > > --
> > > >> > > Sent from the Delta quadrant using Borg technology!
> > > >> > >
> > > >> > > Nux!
> > > >> > > www.nux.ro
> > > >> > >
> > > >> > > ----- Original Message -----
> > > >> > > > From: "Andrija Panic" <an...@gmail.com>
> > > >> > > > To: "dev" <de...@cloudstack.apache.org>
> > > >> > > > Cc: "users" <us...@cloudstack.apache.org>
> > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
> > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > > >> > >
> > > >> > > > We used to make some special stuff for one of the clients,
> > where all
> > > >> LB
> > > >> > > > configuration work is done from outside of the ACS, i.e. python
> > > >> script to
> > > >> > > > feed/configure VR - install latest haproxy 1.5.x for transparent
> > > >> proxy,
> > > >> > > > since client insisted on SSL termination done on backend web SSL
> > > >> > > servers....
> > > >> > > > Not good idea, that is all I can say (custom configuration
> > thing) -
> > > >> but
> > > >> > > the
> > > >> > > > LB setup is actually good - transparent mode haproxy, works on
> > TCP
> > > >> level,
> > > >> > > > so you can see "real client IP" on the backend servers (which
> > must
> > > >> use VR
> > > >> > > > as the default gtw, as per default, so the whole setup works
> > > >> properly).
> > > >> > > >
> > > >> > > > I'm still looking forward to see some special support of LB
> > inside
> > > >> VR via
> > > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
> > > >> > > > provisioning SCRIPT (bash, or whatever),  where all needed
> > > >> > > > install+configure can be done from client side  - otherwise
> > covering
> > > >> all
> > > >> > > > user cases, with proper HTTP checks and similar....is
> > impossible to
> > > >> do
> > > >> > > > IMHO.
> > > >> > > >
> > > >> > > > Some other clients, actually have internal FW appliance (i.e.
> > > >> multihomed
> > > >> > > > VM, acting as gtw for all VMs in all networks), and haproxy
> > instaled
> > > >> on
> > > >> > > > this device (with NAT configured from VR to this internal
> > FW/VM, so
> > > >> > > remote
> > > >> > > > IP can be seen properly) - this setup is fully under customer
> > > >> control,
> > > >> > > and
> > > >> > > > can provide any kind of special haproxy config...
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> > > >> > > >
> > > >> > > >> Hello,
> > > >> > > >>
> > > >> > > >> Of the people running an LB (VR) with https backends, how do
> > you
> > > >> deal
> > > >> > > with
> > > >> > > >> the lack of x-forwarded-for since for port 443 there's just
> > simple
> > > >> TCP
> > > >> > > >> balancing?
> > > >> > > >>
> > > >> > > >> Has anyone thought of terminating SSL in the VR instead? Ideas?
> > > >> > > >>
> > > >> > > >> Cheers
> > > >> > > >>
> > > >> > > >> --
> > > >> > > >> Sent from the Delta quadrant using Borg technology!
> > > >> > > >>
> > > >> > > >> Nux!
> > > >> > > >> www.nux.ro
> > > >> > > >>
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > --
> > > >> > > >
> > > >> > > > Andrija Panić
> > > >> > >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Andrija Panić
> >

Re: HTTPS LB and x-forwarded-for

Posted by Pierre-Luc Dion <pd...@cloudops.com>.
I kind of like the proxy backend type, ill check on our end if that would
work but definitely a simple and efficient approach!



Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wi...@widodh.nl> a écrit :

>
> > Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
> >
> >
> > Wido,
> >
> > Excellent suggestion with the "transparent proxy", I was not aware of
> that.
> > I think that would be a great idea and wouldn't require too many
> modifications, especially as Haproxy comes already with the VR.
> >
>
> It's indeed just a matter of a HAProxy config setting. We could make it
> configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for example.
>
> That way your problem would be solved.
>
> Wido
>
> > To Paul:
> > - imho the LB solution ACS ships now is a bit handicaped since you do
> not know the remote host ip. You're flying blind unless you use google
> analytics (and these things have gotten more and more aggressively filtered
> by adblocks).
> > Enhancing Haproxy as Wido suggested would go a long way, it wouldn't
> break existing functionality and would also keep SSL processing off the VR.
> >
> > --
> > Sent from the Delta quadrant using Borg technology!
> >
> > Nux!
> > www.nux.ro
> >
> > ----- Original Message -----
> > > From: "Andrija Panic" <an...@gmail.com>
> > > To: "users" <us...@cloudstack.apache.org>
> > > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will Stevens" <
> wstevens@cloudops.com>, "dev"
> > > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <pd...@cloudops.com>
> > > Sent: Thursday, 9 November, 2017 13:10:58
> > > Subject: Re: HTTPS LB and x-forwarded-for
> >
> > > Wido,
> > >
> > > backend servers are not Linux only, for example we have a ton of
> Windows
> > > customers, some WEB solutions / IIS etc...
> > >
> > > @all - If we try to please/solve everyone's proxying
> solution/requirement -
> > > this is impossible IMHO - I'm thinking more about some "do it as you
> like"
> > > solution, to let customer write his own haproxy config and upoad it
> (for
> > > example, or something better?).
> > >
> > > We can support newer version of haproxy (1.5+) which also implement
> > > "transarent proxy" (integrate with kernel so to speak)  to allow
> TCP-level
> > > connections to backend (TCP mode, not HTTP mode) but to still
> "preserve"
> > > remote IP by faking it (fake soruce IP = transarent proxy).
> > >
> > > For the rest of configuration options,  I would leave it to the
> customer
> > > how he/she wants to configure rest of haproxy configuration, inlcuding
> > > custom checks, etc. Haproxy configuration is never-ending story, and we
> > > probably should allow custom sripts/configuration instead of trying to
> > > provide GUI/API way to configure everything (which is impossible...)
> > >
> > > Just my 2 cents...
> > >
> > > On 9 November 2017 at 08:13, Wido den Hollander <wi...@widodh.nl>
> wrote:
> > >
> > >>
> > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
> pdion@cloudops.com
> > >> >:
> > >> >
> > >> >
> > >> > Same challenge here too!
> > >> >
> > >> > Let's look at improving Load-balancing offering from cloudstack, I
> guest
> > >> we
> > >> > should do a feature spec draft soon..,  from my perspective, doing
> SSL
> > >> > offload on the VR could be problematic if the VR spec if too small,
> and
> > >> the
> > >> > default spec of the VR being 1vcpu@256MB, considering it can be the
> > >> router
> > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What
> would
> > >> > be your thought about this ?
> > >> >
> > >> > I'd be curious to have a LB offering in ACS where it would deploy a
> > >> > redundant traefik[1] beside the VR for doing http and https
> > >> Load-balancing.
> > >> > I think it would also be useful if the API of that traefik instance
> would
> > >> > be available from within the VPC or LBnetwork so is API would be
> > >> accessible
> > >> > to other apps orchestration tools such as  kubernetes or rancher.
> > >> >
> > >> > traefik or not, here is what I think is needed by cloudstack in the
> LB
> > >> > improvement:
> > >> >
> > >> > - support http, https (X-Forwarded-For)
> > >>
> > >> HAProxy also supports the PROXY protocol towards the backends. Apache
> > >> 2.4.22 supports this natively and Varnish for example can also talk
> PROXY.
> > >>
> > >> It adds a littlebit of metadata to the connection so that the backend
> > >> knows the original IP the connection came from for example:
> > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
> > >>
> > >> Wido
> > >>
> > >> > - basic persistence tuning (API already exist)
> > >> > - better backend monitoring, currently only a tcp connect validate
> if the
> > >> > webserver is up.
> > >> > - ssl offload
> > >> > - metric collection, more stats, maybe just export the tool status
> page
> > >> to
> > >> > the private network.
> > >> > - Container world support, right now if you have Rancher or
> kubernetes
> > >> > cluster, you need to deploy your own LB solution behing mostlikely a
> > >> static
> > >> > nat., If cloudstack would deploy a traefik instance, Kub or Rancher
> could
> > >> > reuse this instance and managed it to properly do LB between
> containers.
> > >> >
> > >> >
> > >> > What would be your prefered LB tool:
> > >> > haproxy, traefik or nginx?
> > >> >
> > >> > CloudStack already have to code to handle SSL certs per projects and
> > >> > accounts if not mistaking because that code was added to support
> > >> NetScaler
> > >> > as Load-balancer in the past. so one less thing to think about :-)
> > >> >
> > >> >
> > >> > [1] https://traefik.io/
> > >> >
> > >> >
> > >> > PL,
> > >> >
> > >> >
> > >> >
> > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
> > >> >
> > >> > > Thanks Andrija,
> > >> > >
> > >> > > LB outside of the VR sounds like a good idea. An appliance based
> on,
> > >> say
> > >> > > cloud-init + ansible and so on could do the trick; alas it'd need
> to be
> > >> > > outside ACS.
> > >> > > I guess as users we could maybe come up with a spec for an
> > >> improvement, at
> > >> > > least we'd have something the devs could look at whenever it is
> > >> possible.
> > >> > >
> > >> > > Regards,
> > >> > > Lucian
> > >> > >
> > >> > > --
> > >> > > Sent from the Delta quadrant using Borg technology!
> > >> > >
> > >> > > Nux!
> > >> > > www.nux.ro
> > >> > >
> > >> > > ----- Original Message -----
> > >> > > > From: "Andrija Panic" <an...@gmail.com>
> > >> > > > To: "dev" <de...@cloudstack.apache.org>
> > >> > > > Cc: "users" <us...@cloudstack.apache.org>
> > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
> > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > >> > >
> > >> > > > We used to make some special stuff for one of the clients,
> where all
> > >> LB
> > >> > > > configuration work is done from outside of the ACS, i.e. python
> > >> script to
> > >> > > > feed/configure VR - install latest haproxy 1.5.x for transparent
> > >> proxy,
> > >> > > > since client insisted on SSL termination done on backend web SSL
> > >> > > servers....
> > >> > > > Not good idea, that is all I can say (custom configuration
> thing) -
> > >> but
> > >> > > the
> > >> > > > LB setup is actually good - transparent mode haproxy, works on
> TCP
> > >> level,
> > >> > > > so you can see "real client IP" on the backend servers (which
> must
> > >> use VR
> > >> > > > as the default gtw, as per default, so the whole setup works
> > >> properly).
> > >> > > >
> > >> > > > I'm still looking forward to see some special support of LB
> inside
> > >> VR via
> > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
> > >> > > > provisioning SCRIPT (bash, or whatever),  where all needed
> > >> > > > install+configure can be done from client side  - otherwise
> covering
> > >> all
> > >> > > > user cases, with proper HTTP checks and similar....is
> impossible to
> > >> do
> > >> > > > IMHO.
> > >> > > >
> > >> > > > Some other clients, actually have internal FW appliance (i.e.
> > >> multihomed
> > >> > > > VM, acting as gtw for all VMs in all networks), and haproxy
> instaled
> > >> on
> > >> > > > this device (with NAT configured from VR to this internal
> FW/VM, so
> > >> > > remote
> > >> > > > IP can be seen properly) - this setup is fully under customer
> > >> control,
> > >> > > and
> > >> > > > can provide any kind of special haproxy config...
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> > >> > > >
> > >> > > >> Hello,
> > >> > > >>
> > >> > > >> Of the people running an LB (VR) with https backends, how do
> you
> > >> deal
> > >> > > with
> > >> > > >> the lack of x-forwarded-for since for port 443 there's just
> simple
> > >> TCP
> > >> > > >> balancing?
> > >> > > >>
> > >> > > >> Has anyone thought of terminating SSL in the VR instead? Ideas?
> > >> > > >>
> > >> > > >> Cheers
> > >> > > >>
> > >> > > >> --
> > >> > > >> Sent from the Delta quadrant using Borg technology!
> > >> > > >>
> > >> > > >> Nux!
> > >> > > >> www.nux.ro
> > >> > > >>
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > --
> > >> > > >
> > >> > > > Andrija Panić
> > >> > >
> > >>
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
>

Re: HTTPS LB and x-forwarded-for

Posted by Pierre-Luc Dion <pd...@cloudops.com>.
I kind of like the proxy backend type, ill check on our end if that would
work but definitely a simple and efficient approach!



Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wi...@widodh.nl> a écrit :

>
> > Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
> >
> >
> > Wido,
> >
> > Excellent suggestion with the "transparent proxy", I was not aware of
> that.
> > I think that would be a great idea and wouldn't require too many
> modifications, especially as Haproxy comes already with the VR.
> >
>
> It's indeed just a matter of a HAProxy config setting. We could make it
> configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for example.
>
> That way your problem would be solved.
>
> Wido
>
> > To Paul:
> > - imho the LB solution ACS ships now is a bit handicaped since you do
> not know the remote host ip. You're flying blind unless you use google
> analytics (and these things have gotten more and more aggressively filtered
> by adblocks).
> > Enhancing Haproxy as Wido suggested would go a long way, it wouldn't
> break existing functionality and would also keep SSL processing off the VR.
> >
> > --
> > Sent from the Delta quadrant using Borg technology!
> >
> > Nux!
> > www.nux.ro
> >
> > ----- Original Message -----
> > > From: "Andrija Panic" <an...@gmail.com>
> > > To: "users" <us...@cloudstack.apache.org>
> > > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will Stevens" <
> wstevens@cloudops.com>, "dev"
> > > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <pd...@cloudops.com>
> > > Sent: Thursday, 9 November, 2017 13:10:58
> > > Subject: Re: HTTPS LB and x-forwarded-for
> >
> > > Wido,
> > >
> > > backend servers are not Linux only, for example we have a ton of
> Windows
> > > customers, some WEB solutions / IIS etc...
> > >
> > > @all - If we try to please/solve everyone's proxying
> solution/requirement -
> > > this is impossible IMHO - I'm thinking more about some "do it as you
> like"
> > > solution, to let customer write his own haproxy config and upoad it
> (for
> > > example, or something better?).
> > >
> > > We can support newer version of haproxy (1.5+) which also implement
> > > "transarent proxy" (integrate with kernel so to speak)  to allow
> TCP-level
> > > connections to backend (TCP mode, not HTTP mode) but to still
> "preserve"
> > > remote IP by faking it (fake soruce IP = transarent proxy).
> > >
> > > For the rest of configuration options,  I would leave it to the
> customer
> > > how he/she wants to configure rest of haproxy configuration, inlcuding
> > > custom checks, etc. Haproxy configuration is never-ending story, and we
> > > probably should allow custom sripts/configuration instead of trying to
> > > provide GUI/API way to configure everything (which is impossible...)
> > >
> > > Just my 2 cents...
> > >
> > > On 9 November 2017 at 08:13, Wido den Hollander <wi...@widodh.nl>
> wrote:
> > >
> > >>
> > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
> pdion@cloudops.com
> > >> >:
> > >> >
> > >> >
> > >> > Same challenge here too!
> > >> >
> > >> > Let's look at improving Load-balancing offering from cloudstack, I
> guest
> > >> we
> > >> > should do a feature spec draft soon..,  from my perspective, doing
> SSL
> > >> > offload on the VR could be problematic if the VR spec if too small,
> and
> > >> the
> > >> > default spec of the VR being 1vcpu@256MB, considering it can be the
> > >> router
> > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What
> would
> > >> > be your thought about this ?
> > >> >
> > >> > I'd be curious to have a LB offering in ACS where it would deploy a
> > >> > redundant traefik[1] beside the VR for doing http and https
> > >> Load-balancing.
> > >> > I think it would also be useful if the API of that traefik instance
> would
> > >> > be available from within the VPC or LBnetwork so is API would be
> > >> accessible
> > >> > to other apps orchestration tools such as  kubernetes or rancher.
> > >> >
> > >> > traefik or not, here is what I think is needed by cloudstack in the
> LB
> > >> > improvement:
> > >> >
> > >> > - support http, https (X-Forwarded-For)
> > >>
> > >> HAProxy also supports the PROXY protocol towards the backends. Apache
> > >> 2.4.22 supports this natively and Varnish for example can also talk
> PROXY.
> > >>
> > >> It adds a littlebit of metadata to the connection so that the backend
> > >> knows the original IP the connection came from for example:
> > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
> > >>
> > >> Wido
> > >>
> > >> > - basic persistence tuning (API already exist)
> > >> > - better backend monitoring, currently only a tcp connect validate
> if the
> > >> > webserver is up.
> > >> > - ssl offload
> > >> > - metric collection, more stats, maybe just export the tool status
> page
> > >> to
> > >> > the private network.
> > >> > - Container world support, right now if you have Rancher or
> kubernetes
> > >> > cluster, you need to deploy your own LB solution behing mostlikely a
> > >> static
> > >> > nat., If cloudstack would deploy a traefik instance, Kub or Rancher
> could
> > >> > reuse this instance and managed it to properly do LB between
> containers.
> > >> >
> > >> >
> > >> > What would be your prefered LB tool:
> > >> > haproxy, traefik or nginx?
> > >> >
> > >> > CloudStack already have to code to handle SSL certs per projects and
> > >> > accounts if not mistaking because that code was added to support
> > >> NetScaler
> > >> > as Load-balancer in the past. so one less thing to think about :-)
> > >> >
> > >> >
> > >> > [1] https://traefik.io/
> > >> >
> > >> >
> > >> > PL,
> > >> >
> > >> >
> > >> >
> > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
> > >> >
> > >> > > Thanks Andrija,
> > >> > >
> > >> > > LB outside of the VR sounds like a good idea. An appliance based
> on,
> > >> say
> > >> > > cloud-init + ansible and so on could do the trick; alas it'd need
> to be
> > >> > > outside ACS.
> > >> > > I guess as users we could maybe come up with a spec for an
> > >> improvement, at
> > >> > > least we'd have something the devs could look at whenever it is
> > >> possible.
> > >> > >
> > >> > > Regards,
> > >> > > Lucian
> > >> > >
> > >> > > --
> > >> > > Sent from the Delta quadrant using Borg technology!
> > >> > >
> > >> > > Nux!
> > >> > > www.nux.ro
> > >> > >
> > >> > > ----- Original Message -----
> > >> > > > From: "Andrija Panic" <an...@gmail.com>
> > >> > > > To: "dev" <de...@cloudstack.apache.org>
> > >> > > > Cc: "users" <us...@cloudstack.apache.org>
> > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
> > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > >> > >
> > >> > > > We used to make some special stuff for one of the clients,
> where all
> > >> LB
> > >> > > > configuration work is done from outside of the ACS, i.e. python
> > >> script to
> > >> > > > feed/configure VR - install latest haproxy 1.5.x for transparent
> > >> proxy,
> > >> > > > since client insisted on SSL termination done on backend web SSL
> > >> > > servers....
> > >> > > > Not good idea, that is all I can say (custom configuration
> thing) -
> > >> but
> > >> > > the
> > >> > > > LB setup is actually good - transparent mode haproxy, works on
> TCP
> > >> level,
> > >> > > > so you can see "real client IP" on the backend servers (which
> must
> > >> use VR
> > >> > > > as the default gtw, as per default, so the whole setup works
> > >> properly).
> > >> > > >
> > >> > > > I'm still looking forward to see some special support of LB
> inside
> > >> VR via
> > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
> > >> > > > provisioning SCRIPT (bash, or whatever),  where all needed
> > >> > > > install+configure can be done from client side  - otherwise
> covering
> > >> all
> > >> > > > user cases, with proper HTTP checks and similar....is
> impossible to
> > >> do
> > >> > > > IMHO.
> > >> > > >
> > >> > > > Some other clients, actually have internal FW appliance (i.e.
> > >> multihomed
> > >> > > > VM, acting as gtw for all VMs in all networks), and haproxy
> instaled
> > >> on
> > >> > > > this device (with NAT configured from VR to this internal
> FW/VM, so
> > >> > > remote
> > >> > > > IP can be seen properly) - this setup is fully under customer
> > >> control,
> > >> > > and
> > >> > > > can provide any kind of special haproxy config...
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> > >> > > >
> > >> > > >> Hello,
> > >> > > >>
> > >> > > >> Of the people running an LB (VR) with https backends, how do
> you
> > >> deal
> > >> > > with
> > >> > > >> the lack of x-forwarded-for since for port 443 there's just
> simple
> > >> TCP
> > >> > > >> balancing?
> > >> > > >>
> > >> > > >> Has anyone thought of terminating SSL in the VR instead? Ideas?
> > >> > > >>
> > >> > > >> Cheers
> > >> > > >>
> > >> > > >> --
> > >> > > >> Sent from the Delta quadrant using Borg technology!
> > >> > > >>
> > >> > > >> Nux!
> > >> > > >> www.nux.ro
> > >> > > >>
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > --
> > >> > > >
> > >> > > > Andrija Panić
> > >> > >
> > >>
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
>

Re: HTTPS LB and x-forwarded-for

Posted by Wido den Hollander <wi...@widodh.nl>.
> Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
> 
> 
> Wido,
> 
> Excellent suggestion with the "transparent proxy", I was not aware of that.
> I think that would be a great idea and wouldn't require too many modifications, especially as Haproxy comes already with the VR.
> 

It's indeed just a matter of a HAProxy config setting. We could make it configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for example.

That way your problem would be solved.

Wido

> To Paul:
> - imho the LB solution ACS ships now is a bit handicaped since you do not know the remote host ip. You're flying blind unless you use google analytics (and these things have gotten more and more aggressively filtered by adblocks).
> Enhancing Haproxy as Wido suggested would go a long way, it wouldn't break existing functionality and would also keep SSL processing off the VR.
> 
> --
> Sent from the Delta quadrant using Borg technology!
> 
> Nux!
> www.nux.ro
> 
> ----- Original Message -----
> > From: "Andrija Panic" <an...@gmail.com>
> > To: "users" <us...@cloudstack.apache.org>
> > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will Stevens" <ws...@cloudops.com>, "dev"
> > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <pd...@cloudops.com>
> > Sent: Thursday, 9 November, 2017 13:10:58
> > Subject: Re: HTTPS LB and x-forwarded-for
> 
> > Wido,
> > 
> > backend servers are not Linux only, for example we have a ton of Windows
> > customers, some WEB solutions / IIS etc...
> > 
> > @all - If we try to please/solve everyone's proxying solution/requirement -
> > this is impossible IMHO - I'm thinking more about some "do it as you like"
> > solution, to let customer write his own haproxy config and upoad it (for
> > example, or something better?).
> > 
> > We can support newer version of haproxy (1.5+) which also implement
> > "transarent proxy" (integrate with kernel so to speak)  to allow TCP-level
> > connections to backend (TCP mode, not HTTP mode) but to still "preserve"
> > remote IP by faking it (fake soruce IP = transarent proxy).
> > 
> > For the rest of configuration options,  I would leave it to the customer
> > how he/she wants to configure rest of haproxy configuration, inlcuding
> > custom checks, etc. Haproxy configuration is never-ending story, and we
> > probably should allow custom sripts/configuration instead of trying to
> > provide GUI/API way to configure everything (which is impossible...)
> > 
> > Just my 2 cents...
> > 
> > On 9 November 2017 at 08:13, Wido den Hollander <wi...@widodh.nl> wrote:
> > 
> >>
> >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <pdion@cloudops.com
> >> >:
> >> >
> >> >
> >> > Same challenge here too!
> >> >
> >> > Let's look at improving Load-balancing offering from cloudstack, I guest
> >> we
> >> > should do a feature spec draft soon..,  from my perspective, doing SSL
> >> > offload on the VR could be problematic if the VR spec if too small, and
> >> the
> >> > default spec of the VR being 1vcpu@256MB, considering it can be the
> >> router
> >> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What would
> >> > be your thought about this ?
> >> >
> >> > I'd be curious to have a LB offering in ACS where it would deploy a
> >> > redundant traefik[1] beside the VR for doing http and https
> >> Load-balancing.
> >> > I think it would also be useful if the API of that traefik instance would
> >> > be available from within the VPC or LBnetwork so is API would be
> >> accessible
> >> > to other apps orchestration tools such as  kubernetes or rancher.
> >> >
> >> > traefik or not, here is what I think is needed by cloudstack in the LB
> >> > improvement:
> >> >
> >> > - support http, https (X-Forwarded-For)
> >>
> >> HAProxy also supports the PROXY protocol towards the backends. Apache
> >> 2.4.22 supports this natively and Varnish for example can also talk PROXY.
> >>
> >> It adds a littlebit of metadata to the connection so that the backend
> >> knows the original IP the connection came from for example:
> >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
> >>
> >> Wido
> >>
> >> > - basic persistence tuning (API already exist)
> >> > - better backend monitoring, currently only a tcp connect validate if the
> >> > webserver is up.
> >> > - ssl offload
> >> > - metric collection, more stats, maybe just export the tool status page
> >> to
> >> > the private network.
> >> > - Container world support, right now if you have Rancher or kubernetes
> >> > cluster, you need to deploy your own LB solution behing mostlikely a
> >> static
> >> > nat., If cloudstack would deploy a traefik instance, Kub or Rancher could
> >> > reuse this instance and managed it to properly do LB between containers.
> >> >
> >> >
> >> > What would be your prefered LB tool:
> >> > haproxy, traefik or nginx?
> >> >
> >> > CloudStack already have to code to handle SSL certs per projects and
> >> > accounts if not mistaking because that code was added to support
> >> NetScaler
> >> > as Load-balancer in the past. so one less thing to think about :-)
> >> >
> >> >
> >> > [1] https://traefik.io/
> >> >
> >> >
> >> > PL,
> >> >
> >> >
> >> >
> >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
> >> >
> >> > > Thanks Andrija,
> >> > >
> >> > > LB outside of the VR sounds like a good idea. An appliance based on,
> >> say
> >> > > cloud-init + ansible and so on could do the trick; alas it'd need to be
> >> > > outside ACS.
> >> > > I guess as users we could maybe come up with a spec for an
> >> improvement, at
> >> > > least we'd have something the devs could look at whenever it is
> >> possible.
> >> > >
> >> > > Regards,
> >> > > Lucian
> >> > >
> >> > > --
> >> > > Sent from the Delta quadrant using Borg technology!
> >> > >
> >> > > Nux!
> >> > > www.nux.ro
> >> > >
> >> > > ----- Original Message -----
> >> > > > From: "Andrija Panic" <an...@gmail.com>
> >> > > > To: "dev" <de...@cloudstack.apache.org>
> >> > > > Cc: "users" <us...@cloudstack.apache.org>
> >> > > > Sent: Thursday, 2 November, 2017 23:21:37
> >> > > > Subject: Re: HTTPS LB and x-forwarded-for
> >> > >
> >> > > > We used to make some special stuff for one of the clients, where all
> >> LB
> >> > > > configuration work is done from outside of the ACS, i.e. python
> >> script to
> >> > > > feed/configure VR - install latest haproxy 1.5.x for transparent
> >> proxy,
> >> > > > since client insisted on SSL termination done on backend web SSL
> >> > > servers....
> >> > > > Not good idea, that is all I can say (custom configuration thing) -
> >> but
> >> > > the
> >> > > > LB setup is actually good - transparent mode haproxy, works on TCP
> >> level,
> >> > > > so you can see "real client IP" on the backend servers (which must
> >> use VR
> >> > > > as the default gtw, as per default, so the whole setup works
> >> properly).
> >> > > >
> >> > > > I'm still looking forward to see some special support of LB inside
> >> VR via
> >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
> >> > > > provisioning SCRIPT (bash, or whatever),  where all needed
> >> > > > install+configure can be done from client side  - otherwise covering
> >> all
> >> > > > user cases, with proper HTTP checks and similar....is impossible to
> >> do
> >> > > > IMHO.
> >> > > >
> >> > > > Some other clients, actually have internal FW appliance (i.e.
> >> multihomed
> >> > > > VM, acting as gtw for all VMs in all networks), and haproxy instaled
> >> on
> >> > > > this device (with NAT configured from VR to this internal FW/VM, so
> >> > > remote
> >> > > > IP can be seen properly) - this setup is fully under customer
> >> control,
> >> > > and
> >> > > > can provide any kind of special haproxy config...
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> >> > > >
> >> > > >> Hello,
> >> > > >>
> >> > > >> Of the people running an LB (VR) with https backends, how do you
> >> deal
> >> > > with
> >> > > >> the lack of x-forwarded-for since for port 443 there's just simple
> >> TCP
> >> > > >> balancing?
> >> > > >>
> >> > > >> Has anyone thought of terminating SSL in the VR instead? Ideas?
> >> > > >>
> >> > > >> Cheers
> >> > > >>
> >> > > >> --
> >> > > >> Sent from the Delta quadrant using Borg technology!
> >> > > >>
> >> > > >> Nux!
> >> > > >> www.nux.ro
> >> > > >>
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > >
> >> > > > Andrija Panić
> >> > >
> >>
> > 
> > 
> > 
> > --
> > 
> > Andrija Panić

Re: HTTPS LB and x-forwarded-for

Posted by Wido den Hollander <wi...@widodh.nl>.
> Op 9 november 2017 om 19:59 schreef Nux! <nu...@li.nux.ro>:
> 
> 
> Wido,
> 
> Excellent suggestion with the "transparent proxy", I was not aware of that.
> I think that would be a great idea and wouldn't require too many modifications, especially as Haproxy comes already with the VR.
> 

It's indeed just a matter of a HAProxy config setting. We could make it configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for example.

That way your problem would be solved.

Wido

> To Paul:
> - imho the LB solution ACS ships now is a bit handicaped since you do not know the remote host ip. You're flying blind unless you use google analytics (and these things have gotten more and more aggressively filtered by adblocks).
> Enhancing Haproxy as Wido suggested would go a long way, it wouldn't break existing functionality and would also keep SSL processing off the VR.
> 
> --
> Sent from the Delta quadrant using Borg technology!
> 
> Nux!
> www.nux.ro
> 
> ----- Original Message -----
> > From: "Andrija Panic" <an...@gmail.com>
> > To: "users" <us...@cloudstack.apache.org>
> > Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will Stevens" <ws...@cloudops.com>, "dev"
> > <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <pd...@cloudops.com>
> > Sent: Thursday, 9 November, 2017 13:10:58
> > Subject: Re: HTTPS LB and x-forwarded-for
> 
> > Wido,
> > 
> > backend servers are not Linux only, for example we have a ton of Windows
> > customers, some WEB solutions / IIS etc...
> > 
> > @all - If we try to please/solve everyone's proxying solution/requirement -
> > this is impossible IMHO - I'm thinking more about some "do it as you like"
> > solution, to let customer write his own haproxy config and upoad it (for
> > example, or something better?).
> > 
> > We can support newer version of haproxy (1.5+) which also implement
> > "transarent proxy" (integrate with kernel so to speak)  to allow TCP-level
> > connections to backend (TCP mode, not HTTP mode) but to still "preserve"
> > remote IP by faking it (fake soruce IP = transarent proxy).
> > 
> > For the rest of configuration options,  I would leave it to the customer
> > how he/she wants to configure rest of haproxy configuration, inlcuding
> > custom checks, etc. Haproxy configuration is never-ending story, and we
> > probably should allow custom sripts/configuration instead of trying to
> > provide GUI/API way to configure everything (which is impossible...)
> > 
> > Just my 2 cents...
> > 
> > On 9 November 2017 at 08:13, Wido den Hollander <wi...@widodh.nl> wrote:
> > 
> >>
> >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <pdion@cloudops.com
> >> >:
> >> >
> >> >
> >> > Same challenge here too!
> >> >
> >> > Let's look at improving Load-balancing offering from cloudstack, I guest
> >> we
> >> > should do a feature spec draft soon..,  from my perspective, doing SSL
> >> > offload on the VR could be problematic if the VR spec if too small, and
> >> the
> >> > default spec of the VR being 1vcpu@256MB, considering it can be the
> >> router
> >> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What would
> >> > be your thought about this ?
> >> >
> >> > I'd be curious to have a LB offering in ACS where it would deploy a
> >> > redundant traefik[1] beside the VR for doing http and https
> >> Load-balancing.
> >> > I think it would also be useful if the API of that traefik instance would
> >> > be available from within the VPC or LBnetwork so is API would be
> >> accessible
> >> > to other apps orchestration tools such as  kubernetes or rancher.
> >> >
> >> > traefik or not, here is what I think is needed by cloudstack in the LB
> >> > improvement:
> >> >
> >> > - support http, https (X-Forwarded-For)
> >>
> >> HAProxy also supports the PROXY protocol towards the backends. Apache
> >> 2.4.22 supports this natively and Varnish for example can also talk PROXY.
> >>
> >> It adds a littlebit of metadata to the connection so that the backend
> >> knows the original IP the connection came from for example:
> >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
> >>
> >> Wido
> >>
> >> > - basic persistence tuning (API already exist)
> >> > - better backend monitoring, currently only a tcp connect validate if the
> >> > webserver is up.
> >> > - ssl offload
> >> > - metric collection, more stats, maybe just export the tool status page
> >> to
> >> > the private network.
> >> > - Container world support, right now if you have Rancher or kubernetes
> >> > cluster, you need to deploy your own LB solution behing mostlikely a
> >> static
> >> > nat., If cloudstack would deploy a traefik instance, Kub or Rancher could
> >> > reuse this instance and managed it to properly do LB between containers.
> >> >
> >> >
> >> > What would be your prefered LB tool:
> >> > haproxy, traefik or nginx?
> >> >
> >> > CloudStack already have to code to handle SSL certs per projects and
> >> > accounts if not mistaking because that code was added to support
> >> NetScaler
> >> > as Load-balancer in the past. so one less thing to think about :-)
> >> >
> >> >
> >> > [1] https://traefik.io/
> >> >
> >> >
> >> > PL,
> >> >
> >> >
> >> >
> >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
> >> >
> >> > > Thanks Andrija,
> >> > >
> >> > > LB outside of the VR sounds like a good idea. An appliance based on,
> >> say
> >> > > cloud-init + ansible and so on could do the trick; alas it'd need to be
> >> > > outside ACS.
> >> > > I guess as users we could maybe come up with a spec for an
> >> improvement, at
> >> > > least we'd have something the devs could look at whenever it is
> >> possible.
> >> > >
> >> > > Regards,
> >> > > Lucian
> >> > >
> >> > > --
> >> > > Sent from the Delta quadrant using Borg technology!
> >> > >
> >> > > Nux!
> >> > > www.nux.ro
> >> > >
> >> > > ----- Original Message -----
> >> > > > From: "Andrija Panic" <an...@gmail.com>
> >> > > > To: "dev" <de...@cloudstack.apache.org>
> >> > > > Cc: "users" <us...@cloudstack.apache.org>
> >> > > > Sent: Thursday, 2 November, 2017 23:21:37
> >> > > > Subject: Re: HTTPS LB and x-forwarded-for
> >> > >
> >> > > > We used to make some special stuff for one of the clients, where all
> >> LB
> >> > > > configuration work is done from outside of the ACS, i.e. python
> >> script to
> >> > > > feed/configure VR - install latest haproxy 1.5.x for transparent
> >> proxy,
> >> > > > since client insisted on SSL termination done on backend web SSL
> >> > > servers....
> >> > > > Not good idea, that is all I can say (custom configuration thing) -
> >> but
> >> > > the
> >> > > > LB setup is actually good - transparent mode haproxy, works on TCP
> >> level,
> >> > > > so you can see "real client IP" on the backend servers (which must
> >> use VR
> >> > > > as the default gtw, as per default, so the whole setup works
> >> properly).
> >> > > >
> >> > > > I'm still looking forward to see some special support of LB inside
> >> VR via
> >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
> >> > > > provisioning SCRIPT (bash, or whatever),  where all needed
> >> > > > install+configure can be done from client side  - otherwise covering
> >> all
> >> > > > user cases, with proper HTTP checks and similar....is impossible to
> >> do
> >> > > > IMHO.
> >> > > >
> >> > > > Some other clients, actually have internal FW appliance (i.e.
> >> multihomed
> >> > > > VM, acting as gtw for all VMs in all networks), and haproxy instaled
> >> on
> >> > > > this device (with NAT configured from VR to this internal FW/VM, so
> >> > > remote
> >> > > > IP can be seen properly) - this setup is fully under customer
> >> control,
> >> > > and
> >> > > > can provide any kind of special haproxy config...
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> >> > > >
> >> > > >> Hello,
> >> > > >>
> >> > > >> Of the people running an LB (VR) with https backends, how do you
> >> deal
> >> > > with
> >> > > >> the lack of x-forwarded-for since for port 443 there's just simple
> >> TCP
> >> > > >> balancing?
> >> > > >>
> >> > > >> Has anyone thought of terminating SSL in the VR instead? Ideas?
> >> > > >>
> >> > > >> Cheers
> >> > > >>
> >> > > >> --
> >> > > >> Sent from the Delta quadrant using Borg technology!
> >> > > >>
> >> > > >> Nux!
> >> > > >> www.nux.ro
> >> > > >>
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > >
> >> > > > Andrija Panić
> >> > >
> >>
> > 
> > 
> > 
> > --
> > 
> > Andrija Panić

Re: HTTPS LB and x-forwarded-for

Posted by Nux! <nu...@li.nux.ro>.
Wido,

Excellent suggestion with the "transparent proxy", I was not aware of that.
I think that would be a great idea and wouldn't require too many modifications, especially as Haproxy comes already with the VR.

To Paul:
- imho the LB solution ACS ships now is a bit handicaped since you do not know the remote host ip. You're flying blind unless you use google analytics (and these things have gotten more and more aggressively filtered by adblocks).
Enhancing Haproxy as Wido suggested would go a long way, it wouldn't break existing functionality and would also keep SSL processing off the VR.

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Andrija Panic" <an...@gmail.com>
> To: "users" <us...@cloudstack.apache.org>
> Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will Stevens" <ws...@cloudops.com>, "dev"
> <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <pd...@cloudops.com>
> Sent: Thursday, 9 November, 2017 13:10:58
> Subject: Re: HTTPS LB and x-forwarded-for

> Wido,
> 
> backend servers are not Linux only, for example we have a ton of Windows
> customers, some WEB solutions / IIS etc...
> 
> @all - If we try to please/solve everyone's proxying solution/requirement -
> this is impossible IMHO - I'm thinking more about some "do it as you like"
> solution, to let customer write his own haproxy config and upoad it (for
> example, or something better?).
> 
> We can support newer version of haproxy (1.5+) which also implement
> "transarent proxy" (integrate with kernel so to speak)  to allow TCP-level
> connections to backend (TCP mode, not HTTP mode) but to still "preserve"
> remote IP by faking it (fake soruce IP = transarent proxy).
> 
> For the rest of configuration options,  I would leave it to the customer
> how he/she wants to configure rest of haproxy configuration, inlcuding
> custom checks, etc. Haproxy configuration is never-ending story, and we
> probably should allow custom sripts/configuration instead of trying to
> provide GUI/API way to configure everything (which is impossible...)
> 
> Just my 2 cents...
> 
> On 9 November 2017 at 08:13, Wido den Hollander <wi...@widodh.nl> wrote:
> 
>>
>> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <pdion@cloudops.com
>> >:
>> >
>> >
>> > Same challenge here too!
>> >
>> > Let's look at improving Load-balancing offering from cloudstack, I guest
>> we
>> > should do a feature spec draft soon..,  from my perspective, doing SSL
>> > offload on the VR could be problematic if the VR spec if too small, and
>> the
>> > default spec of the VR being 1vcpu@256MB, considering it can be the
>> router
>> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What would
>> > be your thought about this ?
>> >
>> > I'd be curious to have a LB offering in ACS where it would deploy a
>> > redundant traefik[1] beside the VR for doing http and https
>> Load-balancing.
>> > I think it would also be useful if the API of that traefik instance would
>> > be available from within the VPC or LBnetwork so is API would be
>> accessible
>> > to other apps orchestration tools such as  kubernetes or rancher.
>> >
>> > traefik or not, here is what I think is needed by cloudstack in the LB
>> > improvement:
>> >
>> > - support http, https (X-Forwarded-For)
>>
>> HAProxy also supports the PROXY protocol towards the backends. Apache
>> 2.4.22 supports this natively and Varnish for example can also talk PROXY.
>>
>> It adds a littlebit of metadata to the connection so that the backend
>> knows the original IP the connection came from for example:
>> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
>>
>> Wido
>>
>> > - basic persistence tuning (API already exist)
>> > - better backend monitoring, currently only a tcp connect validate if the
>> > webserver is up.
>> > - ssl offload
>> > - metric collection, more stats, maybe just export the tool status page
>> to
>> > the private network.
>> > - Container world support, right now if you have Rancher or kubernetes
>> > cluster, you need to deploy your own LB solution behing mostlikely a
>> static
>> > nat., If cloudstack would deploy a traefik instance, Kub or Rancher could
>> > reuse this instance and managed it to properly do LB between containers.
>> >
>> >
>> > What would be your prefered LB tool:
>> > haproxy, traefik or nginx?
>> >
>> > CloudStack already have to code to handle SSL certs per projects and
>> > accounts if not mistaking because that code was added to support
>> NetScaler
>> > as Load-balancer in the past. so one less thing to think about :-)
>> >
>> >
>> > [1] https://traefik.io/
>> >
>> >
>> > PL,
>> >
>> >
>> >
>> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
>> >
>> > > Thanks Andrija,
>> > >
>> > > LB outside of the VR sounds like a good idea. An appliance based on,
>> say
>> > > cloud-init + ansible and so on could do the trick; alas it'd need to be
>> > > outside ACS.
>> > > I guess as users we could maybe come up with a spec for an
>> improvement, at
>> > > least we'd have something the devs could look at whenever it is
>> possible.
>> > >
>> > > Regards,
>> > > Lucian
>> > >
>> > > --
>> > > Sent from the Delta quadrant using Borg technology!
>> > >
>> > > Nux!
>> > > www.nux.ro
>> > >
>> > > ----- Original Message -----
>> > > > From: "Andrija Panic" <an...@gmail.com>
>> > > > To: "dev" <de...@cloudstack.apache.org>
>> > > > Cc: "users" <us...@cloudstack.apache.org>
>> > > > Sent: Thursday, 2 November, 2017 23:21:37
>> > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > >
>> > > > We used to make some special stuff for one of the clients, where all
>> LB
>> > > > configuration work is done from outside of the ACS, i.e. python
>> script to
>> > > > feed/configure VR - install latest haproxy 1.5.x for transparent
>> proxy,
>> > > > since client insisted on SSL termination done on backend web SSL
>> > > servers....
>> > > > Not good idea, that is all I can say (custom configuration thing) -
>> but
>> > > the
>> > > > LB setup is actually good - transparent mode haproxy, works on TCP
>> level,
>> > > > so you can see "real client IP" on the backend servers (which must
>> use VR
>> > > > as the default gtw, as per default, so the whole setup works
>> properly).
>> > > >
>> > > > I'm still looking forward to see some special support of LB inside
>> VR via
>> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
>> > > > provisioning SCRIPT (bash, or whatever),  where all needed
>> > > > install+configure can be done from client side  - otherwise covering
>> all
>> > > > user cases, with proper HTTP checks and similar....is impossible to
>> do
>> > > > IMHO.
>> > > >
>> > > > Some other clients, actually have internal FW appliance (i.e.
>> multihomed
>> > > > VM, acting as gtw for all VMs in all networks), and haproxy instaled
>> on
>> > > > this device (with NAT configured from VR to this internal FW/VM, so
>> > > remote
>> > > > IP can be seen properly) - this setup is fully under customer
>> control,
>> > > and
>> > > > can provide any kind of special haproxy config...
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
>> > > >
>> > > >> Hello,
>> > > >>
>> > > >> Of the people running an LB (VR) with https backends, how do you
>> deal
>> > > with
>> > > >> the lack of x-forwarded-for since for port 443 there's just simple
>> TCP
>> > > >> balancing?
>> > > >>
>> > > >> Has anyone thought of terminating SSL in the VR instead? Ideas?
>> > > >>
>> > > >> Cheers
>> > > >>
>> > > >> --
>> > > >> Sent from the Delta quadrant using Borg technology!
>> > > >>
>> > > >> Nux!
>> > > >> www.nux.ro
>> > > >>
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > >
>> > > > Andrija Panić
>> > >
>>
> 
> 
> 
> --
> 
> Andrija Panić

Re: HTTPS LB and x-forwarded-for

Posted by Nux! <nu...@li.nux.ro>.
Wido,

Excellent suggestion with the "transparent proxy", I was not aware of that.
I think that would be a great idea and wouldn't require too many modifications, especially as Haproxy comes already with the VR.

To Paul:
- imho the LB solution ACS ships now is a bit handicaped since you do not know the remote host ip. You're flying blind unless you use google analytics (and these things have gotten more and more aggressively filtered by adblocks).
Enhancing Haproxy as Wido suggested would go a long way, it wouldn't break existing functionality and would also keep SSL processing off the VR.

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Andrija Panic" <an...@gmail.com>
> To: "users" <us...@cloudstack.apache.org>
> Cc: "Khosrow Moossavi" <km...@cloudops.com>, "Will Stevens" <ws...@cloudops.com>, "dev"
> <de...@cloudstack.apache.org>, "Pierre-Luc Dion" <pd...@cloudops.com>
> Sent: Thursday, 9 November, 2017 13:10:58
> Subject: Re: HTTPS LB and x-forwarded-for

> Wido,
> 
> backend servers are not Linux only, for example we have a ton of Windows
> customers, some WEB solutions / IIS etc...
> 
> @all - If we try to please/solve everyone's proxying solution/requirement -
> this is impossible IMHO - I'm thinking more about some "do it as you like"
> solution, to let customer write his own haproxy config and upoad it (for
> example, or something better?).
> 
> We can support newer version of haproxy (1.5+) which also implement
> "transarent proxy" (integrate with kernel so to speak)  to allow TCP-level
> connections to backend (TCP mode, not HTTP mode) but to still "preserve"
> remote IP by faking it (fake soruce IP = transarent proxy).
> 
> For the rest of configuration options,  I would leave it to the customer
> how he/she wants to configure rest of haproxy configuration, inlcuding
> custom checks, etc. Haproxy configuration is never-ending story, and we
> probably should allow custom sripts/configuration instead of trying to
> provide GUI/API way to configure everything (which is impossible...)
> 
> Just my 2 cents...
> 
> On 9 November 2017 at 08:13, Wido den Hollander <wi...@widodh.nl> wrote:
> 
>>
>> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <pdion@cloudops.com
>> >:
>> >
>> >
>> > Same challenge here too!
>> >
>> > Let's look at improving Load-balancing offering from cloudstack, I guest
>> we
>> > should do a feature spec draft soon..,  from my perspective, doing SSL
>> > offload on the VR could be problematic if the VR spec if too small, and
>> the
>> > default spec of the VR being 1vcpu@256MB, considering it can be the
>> router
>> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What would
>> > be your thought about this ?
>> >
>> > I'd be curious to have a LB offering in ACS where it would deploy a
>> > redundant traefik[1] beside the VR for doing http and https
>> Load-balancing.
>> > I think it would also be useful if the API of that traefik instance would
>> > be available from within the VPC or LBnetwork so is API would be
>> accessible
>> > to other apps orchestration tools such as  kubernetes or rancher.
>> >
>> > traefik or not, here is what I think is needed by cloudstack in the LB
>> > improvement:
>> >
>> > - support http, https (X-Forwarded-For)
>>
>> HAProxy also supports the PROXY protocol towards the backends. Apache
>> 2.4.22 supports this natively and Varnish for example can also talk PROXY.
>>
>> It adds a littlebit of metadata to the connection so that the backend
>> knows the original IP the connection came from for example:
>> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
>>
>> Wido
>>
>> > - basic persistence tuning (API already exist)
>> > - better backend monitoring, currently only a tcp connect validate if the
>> > webserver is up.
>> > - ssl offload
>> > - metric collection, more stats, maybe just export the tool status page
>> to
>> > the private network.
>> > - Container world support, right now if you have Rancher or kubernetes
>> > cluster, you need to deploy your own LB solution behing mostlikely a
>> static
>> > nat., If cloudstack would deploy a traefik instance, Kub or Rancher could
>> > reuse this instance and managed it to properly do LB between containers.
>> >
>> >
>> > What would be your prefered LB tool:
>> > haproxy, traefik or nginx?
>> >
>> > CloudStack already have to code to handle SSL certs per projects and
>> > accounts if not mistaking because that code was added to support
>> NetScaler
>> > as Load-balancer in the past. so one less thing to think about :-)
>> >
>> >
>> > [1] https://traefik.io/
>> >
>> >
>> > PL,
>> >
>> >
>> >
>> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
>> >
>> > > Thanks Andrija,
>> > >
>> > > LB outside of the VR sounds like a good idea. An appliance based on,
>> say
>> > > cloud-init + ansible and so on could do the trick; alas it'd need to be
>> > > outside ACS.
>> > > I guess as users we could maybe come up with a spec for an
>> improvement, at
>> > > least we'd have something the devs could look at whenever it is
>> possible.
>> > >
>> > > Regards,
>> > > Lucian
>> > >
>> > > --
>> > > Sent from the Delta quadrant using Borg technology!
>> > >
>> > > Nux!
>> > > www.nux.ro
>> > >
>> > > ----- Original Message -----
>> > > > From: "Andrija Panic" <an...@gmail.com>
>> > > > To: "dev" <de...@cloudstack.apache.org>
>> > > > Cc: "users" <us...@cloudstack.apache.org>
>> > > > Sent: Thursday, 2 November, 2017 23:21:37
>> > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > >
>> > > > We used to make some special stuff for one of the clients, where all
>> LB
>> > > > configuration work is done from outside of the ACS, i.e. python
>> script to
>> > > > feed/configure VR - install latest haproxy 1.5.x for transparent
>> proxy,
>> > > > since client insisted on SSL termination done on backend web SSL
>> > > servers....
>> > > > Not good idea, that is all I can say (custom configuration thing) -
>> but
>> > > the
>> > > > LB setup is actually good - transparent mode haproxy, works on TCP
>> level,
>> > > > so you can see "real client IP" on the backend servers (which must
>> use VR
>> > > > as the default gtw, as per default, so the whole setup works
>> properly).
>> > > >
>> > > > I'm still looking forward to see some special support of LB inside
>> VR via
>> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
>> > > > provisioning SCRIPT (bash, or whatever),  where all needed
>> > > > install+configure can be done from client side  - otherwise covering
>> all
>> > > > user cases, with proper HTTP checks and similar....is impossible to
>> do
>> > > > IMHO.
>> > > >
>> > > > Some other clients, actually have internal FW appliance (i.e.
>> multihomed
>> > > > VM, acting as gtw for all VMs in all networks), and haproxy instaled
>> on
>> > > > this device (with NAT configured from VR to this internal FW/VM, so
>> > > remote
>> > > > IP can be seen properly) - this setup is fully under customer
>> control,
>> > > and
>> > > > can provide any kind of special haproxy config...
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
>> > > >
>> > > >> Hello,
>> > > >>
>> > > >> Of the people running an LB (VR) with https backends, how do you
>> deal
>> > > with
>> > > >> the lack of x-forwarded-for since for port 443 there's just simple
>> TCP
>> > > >> balancing?
>> > > >>
>> > > >> Has anyone thought of terminating SSL in the VR instead? Ideas?
>> > > >>
>> > > >> Cheers
>> > > >>
>> > > >> --
>> > > >> Sent from the Delta quadrant using Borg technology!
>> > > >>
>> > > >> Nux!
>> > > >> www.nux.ro
>> > > >>
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > >
>> > > > Andrija Panić
>> > >
>>
> 
> 
> 
> --
> 
> Andrija Panić

Re: HTTPS LB and x-forwarded-for

Posted by Andrija Panic <an...@gmail.com>.
Wido,

backend servers are not Linux only, for example we have a ton of Windows
customers, some WEB solutions / IIS etc...

@all - If we try to please/solve everyone's proxying solution/requirement -
this is impossible IMHO - I'm thinking more about some "do it as you like"
solution, to let customer write his own haproxy config and upoad it (for
example, or something better?).

We can support newer version of haproxy (1.5+) which also implement
"transarent proxy" (integrate with kernel so to speak)  to allow TCP-level
connections to backend (TCP mode, not HTTP mode) but to still "preserve"
remote IP by faking it (fake soruce IP = transarent proxy).

For the rest of configuration options,  I would leave it to the customer
how he/she wants to configure rest of haproxy configuration, inlcuding
custom checks, etc. Haproxy configuration is never-ending story, and we
probably should allow custom sripts/configuration instead of trying to
provide GUI/API way to configure everything (which is impossible...)

Just my 2 cents...

On 9 November 2017 at 08:13, Wido den Hollander <wi...@widodh.nl> wrote:

>
> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <pdion@cloudops.com
> >:
> >
> >
> > Same challenge here too!
> >
> > Let's look at improving Load-balancing offering from cloudstack, I guest
> we
> > should do a feature spec draft soon..,  from my perspective, doing SSL
> > offload on the VR could be problematic if the VR spec if too small, and
> the
> > default spec of the VR being 1vcpu@256MB, considering it can be the
> router
> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What would
> > be your thought about this ?
> >
> > I'd be curious to have a LB offering in ACS where it would deploy a
> > redundant traefik[1] beside the VR for doing http and https
> Load-balancing.
> > I think it would also be useful if the API of that traefik instance would
> > be available from within the VPC or LBnetwork so is API would be
> accessible
> > to other apps orchestration tools such as  kubernetes or rancher.
> >
> > traefik or not, here is what I think is needed by cloudstack in the LB
> > improvement:
> >
> > - support http, https (X-Forwarded-For)
>
> HAProxy also supports the PROXY protocol towards the backends. Apache
> 2.4.22 supports this natively and Varnish for example can also talk PROXY.
>
> It adds a littlebit of metadata to the connection so that the backend
> knows the original IP the connection came from for example:
> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
>
> Wido
>
> > - basic persistence tuning (API already exist)
> > - better backend monitoring, currently only a tcp connect validate if the
> > webserver is up.
> > - ssl offload
> > - metric collection, more stats, maybe just export the tool status page
> to
> > the private network.
> > - Container world support, right now if you have Rancher or kubernetes
> > cluster, you need to deploy your own LB solution behing mostlikely a
> static
> > nat., If cloudstack would deploy a traefik instance, Kub or Rancher could
> > reuse this instance and managed it to properly do LB between containers.
> >
> >
> > What would be your prefered LB tool:
> > haproxy, traefik or nginx?
> >
> > CloudStack already have to code to handle SSL certs per projects and
> > accounts if not mistaking because that code was added to support
> NetScaler
> > as Load-balancer in the past. so one less thing to think about :-)
> >
> >
> > [1] https://traefik.io/
> >
> >
> > PL,
> >
> >
> >
> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
> >
> > > Thanks Andrija,
> > >
> > > LB outside of the VR sounds like a good idea. An appliance based on,
> say
> > > cloud-init + ansible and so on could do the trick; alas it'd need to be
> > > outside ACS.
> > > I guess as users we could maybe come up with a spec for an
> improvement, at
> > > least we'd have something the devs could look at whenever it is
> possible.
> > >
> > > Regards,
> > > Lucian
> > >
> > > --
> > > Sent from the Delta quadrant using Borg technology!
> > >
> > > Nux!
> > > www.nux.ro
> > >
> > > ----- Original Message -----
> > > > From: "Andrija Panic" <an...@gmail.com>
> > > > To: "dev" <de...@cloudstack.apache.org>
> > > > Cc: "users" <us...@cloudstack.apache.org>
> > > > Sent: Thursday, 2 November, 2017 23:21:37
> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > >
> > > > We used to make some special stuff for one of the clients, where all
> LB
> > > > configuration work is done from outside of the ACS, i.e. python
> script to
> > > > feed/configure VR - install latest haproxy 1.5.x for transparent
> proxy,
> > > > since client insisted on SSL termination done on backend web SSL
> > > servers....
> > > > Not good idea, that is all I can say (custom configuration thing) -
> but
> > > the
> > > > LB setup is actually good - transparent mode haproxy, works on TCP
> level,
> > > > so you can see "real client IP" on the backend servers (which must
> use VR
> > > > as the default gtw, as per default, so the whole setup works
> properly).
> > > >
> > > > I'm still looking forward to see some special support of LB inside
> VR via
> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
> > > > provisioning SCRIPT (bash, or whatever),  where all needed
> > > > install+configure can be done from client side  - otherwise covering
> all
> > > > user cases, with proper HTTP checks and similar....is impossible to
> do
> > > > IMHO.
> > > >
> > > > Some other clients, actually have internal FW appliance (i.e.
> multihomed
> > > > VM, acting as gtw for all VMs in all networks), and haproxy instaled
> on
> > > > this device (with NAT configured from VR to this internal FW/VM, so
> > > remote
> > > > IP can be seen properly) - this setup is fully under customer
> control,
> > > and
> > > > can provide any kind of special haproxy config...
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> > > >
> > > >> Hello,
> > > >>
> > > >> Of the people running an LB (VR) with https backends, how do you
> deal
> > > with
> > > >> the lack of x-forwarded-for since for port 443 there's just simple
> TCP
> > > >> balancing?
> > > >>
> > > >> Has anyone thought of terminating SSL in the VR instead? Ideas?
> > > >>
> > > >> Cheers
> > > >>
> > > >> --
> > > >> Sent from the Delta quadrant using Borg technology!
> > > >>
> > > >> Nux!
> > > >> www.nux.ro
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Andrija Panić
> > >
>



-- 

Andrija Panić

Re: HTTPS LB and x-forwarded-for

Posted by Andrija Panic <an...@gmail.com>.
Wido,

backend servers are not Linux only, for example we have a ton of Windows
customers, some WEB solutions / IIS etc...

@all - If we try to please/solve everyone's proxying solution/requirement -
this is impossible IMHO - I'm thinking more about some "do it as you like"
solution, to let customer write his own haproxy config and upoad it (for
example, or something better?).

We can support newer version of haproxy (1.5+) which also implement
"transarent proxy" (integrate with kernel so to speak)  to allow TCP-level
connections to backend (TCP mode, not HTTP mode) but to still "preserve"
remote IP by faking it (fake soruce IP = transarent proxy).

For the rest of configuration options,  I would leave it to the customer
how he/she wants to configure rest of haproxy configuration, inlcuding
custom checks, etc. Haproxy configuration is never-ending story, and we
probably should allow custom sripts/configuration instead of trying to
provide GUI/API way to configure everything (which is impossible...)

Just my 2 cents...

On 9 November 2017 at 08:13, Wido den Hollander <wi...@widodh.nl> wrote:

>
> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <pdion@cloudops.com
> >:
> >
> >
> > Same challenge here too!
> >
> > Let's look at improving Load-balancing offering from cloudstack, I guest
> we
> > should do a feature spec draft soon..,  from my perspective, doing SSL
> > offload on the VR could be problematic if the VR spec if too small, and
> the
> > default spec of the VR being 1vcpu@256MB, considering it can be the
> router
> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What would
> > be your thought about this ?
> >
> > I'd be curious to have a LB offering in ACS where it would deploy a
> > redundant traefik[1] beside the VR for doing http and https
> Load-balancing.
> > I think it would also be useful if the API of that traefik instance would
> > be available from within the VPC or LBnetwork so is API would be
> accessible
> > to other apps orchestration tools such as  kubernetes or rancher.
> >
> > traefik or not, here is what I think is needed by cloudstack in the LB
> > improvement:
> >
> > - support http, https (X-Forwarded-For)
>
> HAProxy also supports the PROXY protocol towards the backends. Apache
> 2.4.22 supports this natively and Varnish for example can also talk PROXY.
>
> It adds a littlebit of metadata to the connection so that the backend
> knows the original IP the connection came from for example:
> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
>
> Wido
>
> > - basic persistence tuning (API already exist)
> > - better backend monitoring, currently only a tcp connect validate if the
> > webserver is up.
> > - ssl offload
> > - metric collection, more stats, maybe just export the tool status page
> to
> > the private network.
> > - Container world support, right now if you have Rancher or kubernetes
> > cluster, you need to deploy your own LB solution behing mostlikely a
> static
> > nat., If cloudstack would deploy a traefik instance, Kub or Rancher could
> > reuse this instance and managed it to properly do LB between containers.
> >
> >
> > What would be your prefered LB tool:
> > haproxy, traefik or nginx?
> >
> > CloudStack already have to code to handle SSL certs per projects and
> > accounts if not mistaking because that code was added to support
> NetScaler
> > as Load-balancer in the past. so one less thing to think about :-)
> >
> >
> > [1] https://traefik.io/
> >
> >
> > PL,
> >
> >
> >
> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
> >
> > > Thanks Andrija,
> > >
> > > LB outside of the VR sounds like a good idea. An appliance based on,
> say
> > > cloud-init + ansible and so on could do the trick; alas it'd need to be
> > > outside ACS.
> > > I guess as users we could maybe come up with a spec for an
> improvement, at
> > > least we'd have something the devs could look at whenever it is
> possible.
> > >
> > > Regards,
> > > Lucian
> > >
> > > --
> > > Sent from the Delta quadrant using Borg technology!
> > >
> > > Nux!
> > > www.nux.ro
> > >
> > > ----- Original Message -----
> > > > From: "Andrija Panic" <an...@gmail.com>
> > > > To: "dev" <de...@cloudstack.apache.org>
> > > > Cc: "users" <us...@cloudstack.apache.org>
> > > > Sent: Thursday, 2 November, 2017 23:21:37
> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > >
> > > > We used to make some special stuff for one of the clients, where all
> LB
> > > > configuration work is done from outside of the ACS, i.e. python
> script to
> > > > feed/configure VR - install latest haproxy 1.5.x for transparent
> proxy,
> > > > since client insisted on SSL termination done on backend web SSL
> > > servers....
> > > > Not good idea, that is all I can say (custom configuration thing) -
> but
> > > the
> > > > LB setup is actually good - transparent mode haproxy, works on TCP
> level,
> > > > so you can see "real client IP" on the backend servers (which must
> use VR
> > > > as the default gtw, as per default, so the whole setup works
> properly).
> > > >
> > > > I'm still looking forward to see some special support of LB inside
> VR via
> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
> > > > provisioning SCRIPT (bash, or whatever),  where all needed
> > > > install+configure can be done from client side  - otherwise covering
> all
> > > > user cases, with proper HTTP checks and similar....is impossible to
> do
> > > > IMHO.
> > > >
> > > > Some other clients, actually have internal FW appliance (i.e.
> multihomed
> > > > VM, acting as gtw for all VMs in all networks), and haproxy instaled
> on
> > > > this device (with NAT configured from VR to this internal FW/VM, so
> > > remote
> > > > IP can be seen properly) - this setup is fully under customer
> control,
> > > and
> > > > can provide any kind of special haproxy config...
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> > > >
> > > >> Hello,
> > > >>
> > > >> Of the people running an LB (VR) with https backends, how do you
> deal
> > > with
> > > >> the lack of x-forwarded-for since for port 443 there's just simple
> TCP
> > > >> balancing?
> > > >>
> > > >> Has anyone thought of terminating SSL in the VR instead? Ideas?
> > > >>
> > > >> Cheers
> > > >>
> > > >> --
> > > >> Sent from the Delta quadrant using Borg technology!
> > > >>
> > > >> Nux!
> > > >> www.nux.ro
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Andrija Panić
> > >
>



-- 

Andrija Panić

Re: HTTPS LB and x-forwarded-for

Posted by Wido den Hollander <wi...@widodh.nl>.
> Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <pd...@cloudops.com>:
> 
> 
> Same challenge here too!
> 
> Let's look at improving Load-balancing offering from cloudstack, I guest we
> should do a feature spec draft soon..,  from my perspective, doing SSL
> offload on the VR could be problematic if the VR spec if too small, and the
> default spec of the VR being 1vcpu@256MB, considering it can be the router
> of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What would
> be your thought about this ?
> 
> I'd be curious to have a LB offering in ACS where it would deploy a
> redundant traefik[1] beside the VR for doing http and https Load-balancing.
> I think it would also be useful if the API of that traefik instance would
> be available from within the VPC or LBnetwork so is API would be accessible
> to other apps orchestration tools such as  kubernetes or rancher.
> 
> traefik or not, here is what I think is needed by cloudstack in the LB
> improvement:
> 
> - support http, https (X-Forwarded-For)

HAProxy also supports the PROXY protocol towards the backends. Apache 2.4.22 supports this natively and Varnish for example can also talk PROXY.

It adds a littlebit of metadata to the connection so that the backend knows the original IP the connection came from for example: https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

Wido

> - basic persistence tuning (API already exist)
> - better backend monitoring, currently only a tcp connect validate if the
> webserver is up.
> - ssl offload
> - metric collection, more stats, maybe just export the tool status page to
> the private network.
> - Container world support, right now if you have Rancher or kubernetes
> cluster, you need to deploy your own LB solution behing mostlikely a static
> nat., If cloudstack would deploy a traefik instance, Kub or Rancher could
> reuse this instance and managed it to properly do LB between containers.
> 
> 
> What would be your prefered LB tool:
> haproxy, traefik or nginx?
> 
> CloudStack already have to code to handle SSL certs per projects and
> accounts if not mistaking because that code was added to support NetScaler
> as Load-balancer in the past. so one less thing to think about :-)
> 
> 
> [1] https://traefik.io/
> 
> 
> PL,
> 
> 
> 
> On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
> 
> > Thanks Andrija,
> >
> > LB outside of the VR sounds like a good idea. An appliance based on, say
> > cloud-init + ansible and so on could do the trick; alas it'd need to be
> > outside ACS.
> > I guess as users we could maybe come up with a spec for an improvement, at
> > least we'd have something the devs could look at whenever it is possible.
> >
> > Regards,
> > Lucian
> >
> > --
> > Sent from the Delta quadrant using Borg technology!
> >
> > Nux!
> > www.nux.ro
> >
> > ----- Original Message -----
> > > From: "Andrija Panic" <an...@gmail.com>
> > > To: "dev" <de...@cloudstack.apache.org>
> > > Cc: "users" <us...@cloudstack.apache.org>
> > > Sent: Thursday, 2 November, 2017 23:21:37
> > > Subject: Re: HTTPS LB and x-forwarded-for
> >
> > > We used to make some special stuff for one of the clients, where all LB
> > > configuration work is done from outside of the ACS, i.e. python script to
> > > feed/configure VR - install latest haproxy 1.5.x for transparent proxy,
> > > since client insisted on SSL termination done on backend web SSL
> > servers....
> > > Not good idea, that is all I can say (custom configuration thing) - but
> > the
> > > LB setup is actually good - transparent mode haproxy, works on TCP level,
> > > so you can see "real client IP" on the backend servers (which must use VR
> > > as the default gtw, as per default, so the whole setup works properly).
> > >
> > > I'm still looking forward to see some special support of LB inside VR via
> > > ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
> > > provisioning SCRIPT (bash, or whatever),  where all needed
> > > install+configure can be done from client side  - otherwise covering all
> > > user cases, with proper HTTP checks and similar....is impossible to do
> > > IMHO.
> > >
> > > Some other clients, actually have internal FW appliance (i.e. multihomed
> > > VM, acting as gtw for all VMs in all networks), and haproxy instaled on
> > > this device (with NAT configured from VR to this internal FW/VM, so
> > remote
> > > IP can be seen properly) - this setup is fully under customer control,
> > and
> > > can provide any kind of special haproxy config...
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> > >
> > >> Hello,
> > >>
> > >> Of the people running an LB (VR) with https backends, how do you deal
> > with
> > >> the lack of x-forwarded-for since for port 443 there's just simple TCP
> > >> balancing?
> > >>
> > >> Has anyone thought of terminating SSL in the VR instead? Ideas?
> > >>
> > >> Cheers
> > >>
> > >> --
> > >> Sent from the Delta quadrant using Borg technology!
> > >>
> > >> Nux!
> > >> www.nux.ro
> > >>
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> >

Re: HTTPS LB and x-forwarded-for

Posted by Wido den Hollander <wi...@widodh.nl>.
> Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <pd...@cloudops.com>:
> 
> 
> Same challenge here too!
> 
> Let's look at improving Load-balancing offering from cloudstack, I guest we
> should do a feature spec draft soon..,  from my perspective, doing SSL
> offload on the VR could be problematic if the VR spec if too small, and the
> default spec of the VR being 1vcpu@256MB, considering it can be the router
> of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What would
> be your thought about this ?
> 
> I'd be curious to have a LB offering in ACS where it would deploy a
> redundant traefik[1] beside the VR for doing http and https Load-balancing.
> I think it would also be useful if the API of that traefik instance would
> be available from within the VPC or LBnetwork so is API would be accessible
> to other apps orchestration tools such as  kubernetes or rancher.
> 
> traefik or not, here is what I think is needed by cloudstack in the LB
> improvement:
> 
> - support http, https (X-Forwarded-For)

HAProxy also supports the PROXY protocol towards the backends. Apache 2.4.22 supports this natively and Varnish for example can also talk PROXY.

It adds a littlebit of metadata to the connection so that the backend knows the original IP the connection came from for example: https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

Wido

> - basic persistence tuning (API already exist)
> - better backend monitoring, currently only a tcp connect validate if the
> webserver is up.
> - ssl offload
> - metric collection, more stats, maybe just export the tool status page to
> the private network.
> - Container world support, right now if you have Rancher or kubernetes
> cluster, you need to deploy your own LB solution behing mostlikely a static
> nat., If cloudstack would deploy a traefik instance, Kub or Rancher could
> reuse this instance and managed it to properly do LB between containers.
> 
> 
> What would be your prefered LB tool:
> haproxy, traefik or nginx?
> 
> CloudStack already have to code to handle SSL certs per projects and
> accounts if not mistaking because that code was added to support NetScaler
> as Load-balancer in the past. so one less thing to think about :-)
> 
> 
> [1] https://traefik.io/
> 
> 
> PL,
> 
> 
> 
> On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:
> 
> > Thanks Andrija,
> >
> > LB outside of the VR sounds like a good idea. An appliance based on, say
> > cloud-init + ansible and so on could do the trick; alas it'd need to be
> > outside ACS.
> > I guess as users we could maybe come up with a spec for an improvement, at
> > least we'd have something the devs could look at whenever it is possible.
> >
> > Regards,
> > Lucian
> >
> > --
> > Sent from the Delta quadrant using Borg technology!
> >
> > Nux!
> > www.nux.ro
> >
> > ----- Original Message -----
> > > From: "Andrija Panic" <an...@gmail.com>
> > > To: "dev" <de...@cloudstack.apache.org>
> > > Cc: "users" <us...@cloudstack.apache.org>
> > > Sent: Thursday, 2 November, 2017 23:21:37
> > > Subject: Re: HTTPS LB and x-forwarded-for
> >
> > > We used to make some special stuff for one of the clients, where all LB
> > > configuration work is done from outside of the ACS, i.e. python script to
> > > feed/configure VR - install latest haproxy 1.5.x for transparent proxy,
> > > since client insisted on SSL termination done on backend web SSL
> > servers....
> > > Not good idea, that is all I can say (custom configuration thing) - but
> > the
> > > LB setup is actually good - transparent mode haproxy, works on TCP level,
> > > so you can see "real client IP" on the backend servers (which must use VR
> > > as the default gtw, as per default, so the whole setup works properly).
> > >
> > > I'm still looking forward to see some special support of LB inside VR via
> > > ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
> > > provisioning SCRIPT (bash, or whatever),  where all needed
> > > install+configure can be done from client side  - otherwise covering all
> > > user cases, with proper HTTP checks and similar....is impossible to do
> > > IMHO.
> > >
> > > Some other clients, actually have internal FW appliance (i.e. multihomed
> > > VM, acting as gtw for all VMs in all networks), and haproxy instaled on
> > > this device (with NAT configured from VR to this internal FW/VM, so
> > remote
> > > IP can be seen properly) - this setup is fully under customer control,
> > and
> > > can provide any kind of special haproxy config...
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> > >
> > >> Hello,
> > >>
> > >> Of the people running an LB (VR) with https backends, how do you deal
> > with
> > >> the lack of x-forwarded-for since for port 443 there's just simple TCP
> > >> balancing?
> > >>
> > >> Has anyone thought of terminating SSL in the VR instead? Ideas?
> > >>
> > >> Cheers
> > >>
> > >> --
> > >> Sent from the Delta quadrant using Borg technology!
> > >>
> > >> Nux!
> > >> www.nux.ro
> > >>
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> >

Re: HTTPS LB and x-forwarded-for

Posted by Pierre-Luc Dion <pd...@cloudops.com>.
Same challenge here too!

Let's look at improving Load-balancing offering from cloudstack, I guest we
should do a feature spec draft soon..,  from my perspective, doing SSL
offload on the VR could be problematic if the VR spec if too small, and the
default spec of the VR being 1vcpu@256MB, considering it can be the router
of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What would
be your thought about this ?

I'd be curious to have a LB offering in ACS where it would deploy a
redundant traefik[1] beside the VR for doing http and https Load-balancing.
I think it would also be useful if the API of that traefik instance would
be available from within the VPC or LBnetwork so is API would be accessible
to other apps orchestration tools such as  kubernetes or rancher.

traefik or not, here is what I think is needed by cloudstack in the LB
improvement:

- support http, https (X-Forwarded-For)
- basic persistence tuning (API already exist)
- better backend monitoring, currently only a tcp connect validate if the
webserver is up.
- ssl offload
- metric collection, more stats, maybe just export the tool status page to
the private network.
- Container world support, right now if you have Rancher or kubernetes
cluster, you need to deploy your own LB solution behing mostlikely a static
nat., If cloudstack would deploy a traefik instance, Kub or Rancher could
reuse this instance and managed it to properly do LB between containers.


What would be your prefered LB tool:
haproxy, traefik or nginx?

CloudStack already have to code to handle SSL certs per projects and
accounts if not mistaking because that code was added to support NetScaler
as Load-balancer in the past. so one less thing to think about :-)


[1] https://traefik.io/


PL,



On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:

> Thanks Andrija,
>
> LB outside of the VR sounds like a good idea. An appliance based on, say
> cloud-init + ansible and so on could do the trick; alas it'd need to be
> outside ACS.
> I guess as users we could maybe come up with a spec for an improvement, at
> least we'd have something the devs could look at whenever it is possible.
>
> Regards,
> Lucian
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Andrija Panic" <an...@gmail.com>
> > To: "dev" <de...@cloudstack.apache.org>
> > Cc: "users" <us...@cloudstack.apache.org>
> > Sent: Thursday, 2 November, 2017 23:21:37
> > Subject: Re: HTTPS LB and x-forwarded-for
>
> > We used to make some special stuff for one of the clients, where all LB
> > configuration work is done from outside of the ACS, i.e. python script to
> > feed/configure VR - install latest haproxy 1.5.x for transparent proxy,
> > since client insisted on SSL termination done on backend web SSL
> servers....
> > Not good idea, that is all I can say (custom configuration thing) - but
> the
> > LB setup is actually good - transparent mode haproxy, works on TCP level,
> > so you can see "real client IP" on the backend servers (which must use VR
> > as the default gtw, as per default, so the whole setup works properly).
> >
> > I'm still looking forward to see some special support of LB inside VR via
> > ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
> > provisioning SCRIPT (bash, or whatever),  where all needed
> > install+configure can be done from client side  - otherwise covering all
> > user cases, with proper HTTP checks and similar....is impossible to do
> > IMHO.
> >
> > Some other clients, actually have internal FW appliance (i.e. multihomed
> > VM, acting as gtw for all VMs in all networks), and haproxy instaled on
> > this device (with NAT configured from VR to this internal FW/VM, so
> remote
> > IP can be seen properly) - this setup is fully under customer control,
> and
> > can provide any kind of special haproxy config...
> >
> >
> >
> >
> >
> >
> > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> >
> >> Hello,
> >>
> >> Of the people running an LB (VR) with https backends, how do you deal
> with
> >> the lack of x-forwarded-for since for port 443 there's just simple TCP
> >> balancing?
> >>
> >> Has anyone thought of terminating SSL in the VR instead? Ideas?
> >>
> >> Cheers
> >>
> >> --
> >> Sent from the Delta quadrant using Borg technology!
> >>
> >> Nux!
> >> www.nux.ro
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
>

Re: HTTPS LB and x-forwarded-for

Posted by Pierre-Luc Dion <pd...@cloudops.com>.
Same challenge here too!

Let's look at improving Load-balancing offering from cloudstack, I guest we
should do a feature spec draft soon..,  from my perspective, doing SSL
offload on the VR could be problematic if the VR spec if too small, and the
default spec of the VR being 1vcpu@256MB, considering it can be the router
of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What would
be your thought about this ?

I'd be curious to have a LB offering in ACS where it would deploy a
redundant traefik[1] beside the VR for doing http and https Load-balancing.
I think it would also be useful if the API of that traefik instance would
be available from within the VPC or LBnetwork so is API would be accessible
to other apps orchestration tools such as  kubernetes or rancher.

traefik or not, here is what I think is needed by cloudstack in the LB
improvement:

- support http, https (X-Forwarded-For)
- basic persistence tuning (API already exist)
- better backend monitoring, currently only a tcp connect validate if the
webserver is up.
- ssl offload
- metric collection, more stats, maybe just export the tool status page to
the private network.
- Container world support, right now if you have Rancher or kubernetes
cluster, you need to deploy your own LB solution behing mostlikely a static
nat., If cloudstack would deploy a traefik instance, Kub or Rancher could
reuse this instance and managed it to properly do LB between containers.


What would be your prefered LB tool:
haproxy, traefik or nginx?

CloudStack already have to code to handle SSL certs per projects and
accounts if not mistaking because that code was added to support NetScaler
as Load-balancer in the past. so one less thing to think about :-)


[1] https://traefik.io/


PL,



On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nu...@li.nux.ro> wrote:

> Thanks Andrija,
>
> LB outside of the VR sounds like a good idea. An appliance based on, say
> cloud-init + ansible and so on could do the trick; alas it'd need to be
> outside ACS.
> I guess as users we could maybe come up with a spec for an improvement, at
> least we'd have something the devs could look at whenever it is possible.
>
> Regards,
> Lucian
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Andrija Panic" <an...@gmail.com>
> > To: "dev" <de...@cloudstack.apache.org>
> > Cc: "users" <us...@cloudstack.apache.org>
> > Sent: Thursday, 2 November, 2017 23:21:37
> > Subject: Re: HTTPS LB and x-forwarded-for
>
> > We used to make some special stuff for one of the clients, where all LB
> > configuration work is done from outside of the ACS, i.e. python script to
> > feed/configure VR - install latest haproxy 1.5.x for transparent proxy,
> > since client insisted on SSL termination done on backend web SSL
> servers....
> > Not good idea, that is all I can say (custom configuration thing) - but
> the
> > LB setup is actually good - transparent mode haproxy, works on TCP level,
> > so you can see "real client IP" on the backend servers (which must use VR
> > as the default gtw, as per default, so the whole setup works properly).
> >
> > I'm still looking forward to see some special support of LB inside VR via
> > ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
> > provisioning SCRIPT (bash, or whatever),  where all needed
> > install+configure can be done from client side  - otherwise covering all
> > user cases, with proper HTTP checks and similar....is impossible to do
> > IMHO.
> >
> > Some other clients, actually have internal FW appliance (i.e. multihomed
> > VM, acting as gtw for all VMs in all networks), and haproxy instaled on
> > this device (with NAT configured from VR to this internal FW/VM, so
> remote
> > IP can be seen properly) - this setup is fully under customer control,
> and
> > can provide any kind of special haproxy config...
> >
> >
> >
> >
> >
> >
> > On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> >
> >> Hello,
> >>
> >> Of the people running an LB (VR) with https backends, how do you deal
> with
> >> the lack of x-forwarded-for since for port 443 there's just simple TCP
> >> balancing?
> >>
> >> Has anyone thought of terminating SSL in the VR instead? Ideas?
> >>
> >> Cheers
> >>
> >> --
> >> Sent from the Delta quadrant using Borg technology!
> >>
> >> Nux!
> >> www.nux.ro
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
>

Re: HTTPS LB and x-forwarded-for

Posted by Nux! <nu...@li.nux.ro>.
Thanks Andrija,

LB outside of the VR sounds like a good idea. An appliance based on, say cloud-init + ansible and so on could do the trick; alas it'd need to be outside ACS.
I guess as users we could maybe come up with a spec for an improvement, at least we'd have something the devs could look at whenever it is possible.

Regards,
Lucian

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Andrija Panic" <an...@gmail.com>
> To: "dev" <de...@cloudstack.apache.org>
> Cc: "users" <us...@cloudstack.apache.org>
> Sent: Thursday, 2 November, 2017 23:21:37
> Subject: Re: HTTPS LB and x-forwarded-for

> We used to make some special stuff for one of the clients, where all LB
> configuration work is done from outside of the ACS, i.e. python script to
> feed/configure VR - install latest haproxy 1.5.x for transparent proxy,
> since client insisted on SSL termination done on backend web SSL servers....
> Not good idea, that is all I can say (custom configuration thing) - but the
> LB setup is actually good - transparent mode haproxy, works on TCP level,
> so you can see "real client IP" on the backend servers (which must use VR
> as the default gtw, as per default, so the whole setup works properly).
> 
> I'm still looking forward to see some special support of LB inside VR via
> ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
> provisioning SCRIPT (bash, or whatever),  where all needed
> install+configure can be done from client side  - otherwise covering all
> user cases, with proper HTTP checks and similar....is impossible to do
> IMHO.
> 
> Some other clients, actually have internal FW appliance (i.e. multihomed
> VM, acting as gtw for all VMs in all networks), and haproxy instaled on
> this device (with NAT configured from VR to this internal FW/VM, so remote
> IP can be seen properly) - this setup is fully under customer control, and
> can provide any kind of special haproxy config...
> 
> 
> 
> 
> 
> 
> On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> 
>> Hello,
>>
>> Of the people running an LB (VR) with https backends, how do you deal with
>> the lack of x-forwarded-for since for port 443 there's just simple TCP
>> balancing?
>>
>> Has anyone thought of terminating SSL in the VR instead? Ideas?
>>
>> Cheers
>>
>> --
>> Sent from the Delta quadrant using Borg technology!
>>
>> Nux!
>> www.nux.ro
>>
> 
> 
> 
> --
> 
> Andrija Panić

Re: HTTPS LB and x-forwarded-for

Posted by Nux! <nu...@li.nux.ro>.
Thanks Andrija,

LB outside of the VR sounds like a good idea. An appliance based on, say cloud-init + ansible and so on could do the trick; alas it'd need to be outside ACS.
I guess as users we could maybe come up with a spec for an improvement, at least we'd have something the devs could look at whenever it is possible.

Regards,
Lucian

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Andrija Panic" <an...@gmail.com>
> To: "dev" <de...@cloudstack.apache.org>
> Cc: "users" <us...@cloudstack.apache.org>
> Sent: Thursday, 2 November, 2017 23:21:37
> Subject: Re: HTTPS LB and x-forwarded-for

> We used to make some special stuff for one of the clients, where all LB
> configuration work is done from outside of the ACS, i.e. python script to
> feed/configure VR - install latest haproxy 1.5.x for transparent proxy,
> since client insisted on SSL termination done on backend web SSL servers....
> Not good idea, that is all I can say (custom configuration thing) - but the
> LB setup is actually good - transparent mode haproxy, works on TCP level,
> so you can see "real client IP" on the backend servers (which must use VR
> as the default gtw, as per default, so the whole setup works properly).
> 
> I'm still looking forward to see some special support of LB inside VR via
> ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
> provisioning SCRIPT (bash, or whatever),  where all needed
> install+configure can be done from client side  - otherwise covering all
> user cases, with proper HTTP checks and similar....is impossible to do
> IMHO.
> 
> Some other clients, actually have internal FW appliance (i.e. multihomed
> VM, acting as gtw for all VMs in all networks), and haproxy instaled on
> this device (with NAT configured from VR to this internal FW/VM, so remote
> IP can be seen properly) - this setup is fully under customer control, and
> can provide any kind of special haproxy config...
> 
> 
> 
> 
> 
> 
> On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:
> 
>> Hello,
>>
>> Of the people running an LB (VR) with https backends, how do you deal with
>> the lack of x-forwarded-for since for port 443 there's just simple TCP
>> balancing?
>>
>> Has anyone thought of terminating SSL in the VR instead? Ideas?
>>
>> Cheers
>>
>> --
>> Sent from the Delta quadrant using Borg technology!
>>
>> Nux!
>> www.nux.ro
>>
> 
> 
> 
> --
> 
> Andrija Panić

Re: HTTPS LB and x-forwarded-for

Posted by Andrija Panic <an...@gmail.com>.
We used to make some special stuff for one of the clients, where all LB
configuration work is done from outside of the ACS, i.e. python script to
feed/configure VR - install latest haproxy 1.5.x for transparent proxy,
since client insisted on SSL termination done on backend web SSL servers....
Not good idea, that is all I can say (custom configuration thing) - but the
LB setup is actually good - transparent mode haproxy, works on TCP level,
so you can see "real client IP" on the backend servers (which must use VR
as the default gtw, as per default, so the whole setup works properly).

I'm still looking forward to see some special support of LB inside VR via
ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
provisioning SCRIPT (bash, or whatever),  where all needed
install+configure can be done from client side  - otherwise covering all
user cases, with proper HTTP checks and similar....is impossible to do
IMHO.

Some other clients, actually have internal FW appliance (i.e. multihomed
VM, acting as gtw for all VMs in all networks), and haproxy instaled on
this device (with NAT configured from VR to this internal FW/VM, so remote
IP can be seen properly) - this setup is fully under customer control, and
can provide any kind of special haproxy config...






On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:

> Hello,
>
> Of the people running an LB (VR) with https backends, how do you deal with
> the lack of x-forwarded-for since for port 443 there's just simple TCP
> balancing?
>
> Has anyone thought of terminating SSL in the VR instead? Ideas?
>
> Cheers
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>



-- 

Andrija Panić

Re: HTTPS LB and x-forwarded-for

Posted by Andrija Panic <an...@gmail.com>.
We used to make some special stuff for one of the clients, where all LB
configuration work is done from outside of the ACS, i.e. python script to
feed/configure VR - install latest haproxy 1.5.x for transparent proxy,
since client insisted on SSL termination done on backend web SSL servers....
Not good idea, that is all I can say (custom configuration thing) - but the
LB setup is actually good - transparent mode haproxy, works on TCP level,
so you can see "real client IP" on the backend servers (which must use VR
as the default gtw, as per default, so the whole setup works properly).

I'm still looking forward to see some special support of LB inside VR via
ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
provisioning SCRIPT (bash, or whatever),  where all needed
install+configure can be done from client side  - otherwise covering all
user cases, with proper HTTP checks and similar....is impossible to do
IMHO.

Some other clients, actually have internal FW appliance (i.e. multihomed
VM, acting as gtw for all VMs in all networks), and haproxy instaled on
this device (with NAT configured from VR to this internal FW/VM, so remote
IP can be seen properly) - this setup is fully under customer control, and
can provide any kind of special haproxy config...






On 31 October 2017 at 19:54, Nux! <nu...@li.nux.ro> wrote:

> Hello,
>
> Of the people running an LB (VR) with https backends, how do you deal with
> the lack of x-forwarded-for since for port 443 there's just simple TCP
> balancing?
>
> Has anyone thought of terminating SSL in the VR instead? Ideas?
>
> Cheers
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>



-- 

Andrija Panić