You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Jim Jagielski <ji...@jaguNET.com> on 2015/12/03 15:59:00 UTC
reverse proxy wishlist
I put out a call on Twitter regarding this, but wanted to
close the loop here as well.
What would *you* like to see as new features or enhancements
w/ mod_proxy, esp reverse proxy. I was thinking about some
sort of active backend monitoring, utilizing watchdog, which
could also maybe, eventually, pull in performance and load
data for the backend for a more accurate LB provider. But
what about new LB methods? Any ideas there?
tia.
Re: reverse proxy wishlist
Posted by Jim Jagielski <ji...@jaguNET.com>.
> On Dec 9, 2015, at 2:02 AM, jean-frederic clere <jf...@gmail.com> wrote:
>
> On 12/03/2015 03:59 PM, Jim Jagielski wrote:
>> I put out a call on Twitter regarding this, but wanted to
>> close the loop here as well.
>>
>> What would *you* like to see as new features or enhancements
>> w/ mod_proxy, esp reverse proxy. I was thinking about some
>> sort of active backend monitoring, utilizing watchdog, which
>> could also maybe, eventually, pull in performance and load
>> data for the backend for a more accurate LB provider. But
>> what about new LB methods? Any ideas there?
>
> I need a more dynamic list of workers and balancers basically in
> mod_cluster we are able to add and remove those.
Yeah, adding balancers has been on my TODO for awhile and I have
some krufty implementations scattered around. I think most of
the main hooks and scars are already in trunk and httpd-2.4,
it's just the actual impl.
Does mod_cluster actually allow you to remove, or just
disable?
> Our health check is
> done by the nodes which also requires some changes in the proxy logic too.
For healthcheck I was thinking of just using our watchdog
and allowing it to update the balancer and worker's shared
memory as needed.
Re: reverse proxy wishlist
Posted by jean-frederic clere <jf...@gmail.com>.
On 12/03/2015 03:59 PM, Jim Jagielski wrote:
> I put out a call on Twitter regarding this, but wanted to
> close the loop here as well.
>
> What would *you* like to see as new features or enhancements
> w/ mod_proxy, esp reverse proxy. I was thinking about some
> sort of active backend monitoring, utilizing watchdog, which
> could also maybe, eventually, pull in performance and load
> data for the backend for a more accurate LB provider. But
> what about new LB methods? Any ideas there?
I need a more dynamic list of workers and balancers basically in
mod_cluster we are able to add and remove those. Our health check is
done by the nodes which also requires some changes in the proxy logic too.
Cheers
Jean-Frederic
Re: reverse proxy wishlist
Posted by Mark Thomas <ma...@apache.org>.
On 2015-12-03 14:59, Jim Jagielski <ji...@jaguNET.com> wrote:
> I put out a call on Twitter regarding this, but wanted to
> close the loop here as well.
>
> What would *you* like to see as new features or enhancements
> w/ mod_proxy, esp reverse proxy. I was thinking about some
> sort of active backend monitoring, utilizing watchdog, which
> could also maybe, eventually, pull in performance and load
> data for the backend for a more accurate LB provider. But
> what about new LB methods? Any ideas there?
>
> tia.
With my Tomcat hat on:
HTTP/2 support for mod_proxy_http
HTTP upgrade support for mod_proxy_ajp (we'll need to do work on the Tomcat side as well)
Improved WebSocket support in mod_proxy_wstunnel [1].
I'm happy to help out on the Tomcat side of things where required.
Mark
[1] The mod_wstunnel assumes (at least it did the last time I looked at it) that all requests under a given URL space will be WebSocket requests. That doesn't seem to be the way many apps are being implemented. It would be great if mod_proxy would allow both mod_proxy_[ajp|http] and mod_proxy_wstunnel to be mapped to the same URL space with the 'right' one being selected based on the request.
------
Sent via Pony Mail for dev@httpd.apache.org.
View this email online at:
https://pony-poc.apache.org/list.html?dev@httpd.apache.org
Re: reverse proxy wishlist
Posted by jean-frederic clere <jf...@gmail.com>.
On 12/03/2015 07:39 PM, Houser, Rick wrote:
> I would definitely expect to see a substantial improvement if the thread reservation could be delayed until the response headers are fully received.
That is something tricky you need to mix blocking and non-blocking of
have a bunch (configurable) threads that receive the data and are able
to know they have a complete message they can forward to the worker
thread that does the send (which also needs a bunch of threads dedicated
to send in a non-blocking way to the client).
Cheers
Jean-Frederic
RE: reverse proxy wishlist
Posted by "Houser, Rick" <ri...@jackson.com>.
> Out of curiosity, is it usually time to first byte where the delays show up?
> I am wondering if there is a big bang-for-the-buck enhancement possible there
> that doesn't prereq changes to some of the areas where hopping on and off the
> thread is more complicated.
Ultimately, most of this stuff is getting bottlenecked on database accesses (or similar resources) somewhere in the backend. The problem is, I deal with hundreds of different applications (with both JBoss and Tomcat using the mod_proxy backend) and each one performs slightly different. It would take more than a trivial amount of effort to get specifics on whether these are blocking before the first byte of the header or before the first byte of the content itself ( as in https://docs.oracle.com/javaee/6/api/javax/servlet/ServletResponse.html#isCommitted%28%29 ). In at least the case of WebSphere Application Server (granted, a non-mod_proxy backend), I think there was even an option to control the data chunking to have the server either wait for a complete response or to start sending it out in pieces.
How the JEE tiers typically work is that one or more non-presentation layers first collect all the data needed and perform the business logic -- this would typically get called from the Servlet itself (ex. doGet method -- or an equivalent spot in a framework). Then, that display object gets sent to the presentation layer (JSP or the like) in the form of an object, where the final form is rendered (XHTML, HTML, etc.) strictly from the contents of that object. As the presentation logic is a very small part of the overall execution timeline (and should not be making backend calls itself), users typically see nothing for a while, then the whole page just pops up. The headers are a similar state, and I suspect (but have not confirmed) they only leave the server as a complete set (hence the isCommitted method above). This is in direct contrast to typical LAMP stack setups without a dedicated presentation tier (like a simple PHP site), where end users often see a page partially render output, pausing at specific database queries.
I would definitely expect to see a substantial improvement if the thread reservation could be delayed until the response headers are fully received. However, it would take some additional research to figure out how much if any of that savings could also be gained by allocating the thread when the response headers first start coming in. That additional research may also be highly dependent on the specific backend in question and any options in use. At least under this type of JEE workload, however, I would not expect to see a noticable benefit to being able to detach and re-attach the worker threads after we start processing the response body.
Rick Houser
Web Administration
> -----Original Message-----
> From: Eric Covener [mailto:covener@gmail.com]
> Sent: Thursday, December 03, 2015 12:40 PM
> To: Apache HTTP Server Development List <de...@httpd.apache.org>
> Subject: Re: reverse proxy wishlist
>
> On Thu, Dec 3, 2015 at 12:36 PM, Houser, Rick <ri...@jackson.com>
> wrote:
> > An async mod_proxy backend would be huge for my workloads. In the JEE
> space I deal with, much more time is spent waiting on the application backends
> then with the clients, especially now that we have the event mpm. Something
> like this would allow me to drastically reduce thread counts and free up a lot of
> associated memory.
>
> Out of curiosity, is it usually time to first byte where the delays show up?
> I am wondering if there is a big bang-for-the-buck enhancement possible there
> that doesn't prereq changes to some of the areas where hopping on and off the
> thread is more complicated.
Re: reverse proxy wishlist
Posted by Eric Covener <co...@gmail.com>.
On Thu, Dec 3, 2015 at 12:36 PM, Houser, Rick <ri...@jackson.com> wrote:
> An async mod_proxy backend would be huge for my workloads. In the JEE space I deal with, much more time is spent waiting on the application backends then with the clients, especially now that we have the event mpm. Something like this would allow me to drastically reduce thread counts and free up a lot of associated memory.
Out of curiosity, is it usually time to first byte where the delays show up?
I am wondering if there is a big bang-for-the-buck enhancement possible there
that doesn't prereq changes to some of the areas where hopping on and off the
thread is more complicated.
RE: reverse proxy wishlist
Posted by "Houser, Rick" <ri...@jackson.com>.
> assuming slow backends
My workload isn't so much backends slowly and steadily producing data on each connection, but significantly long pauses before delivering any content at all. The "slow" vs "paused" backend seems more applicable to something like a PHP backend without a View/Presentation tier (as in MVC).
> how would you like httpd to handle it: should it just slurp in the data from the backend and buffer it and send it to the client all in one go? Should it instead forward data as soon as it gets it?
I would think any such backend should forward the response body (or complete request without a body) as soon as it comes in, streaming if the content is large enough (ex. a PDF).
If I desired to buffer all the data before sending to clients (not applicable to me, but I can imagine some use cases), that would be the job of a separate module sitting on the output filter chain, wouldn't it?
Rick Houser
Web Administration
(517)367-3516
> -----Original Message-----
> From: Jim Jagielski [mailto:jim@jaguNET.com]
> Sent: Thursday, December 03, 2015 4:42 PM
> To: dev@httpd.apache.org
> Subject: Re: reverse proxy wishlist
>
> Thx! assuming slow backends, how would you like httpd to
> handle it: should it just slurp in the data from the backend
> and buffer it and send it to the client all in one go? Should
> it instead forward data as soon as it gets it?
> > On Dec 3, 2015, at 12:36 PM, Houser, Rick <ri...@jackson.com>
> wrote:
> >
> > An async mod_proxy backend would be huge for my workloads. In the JEE
> space I deal with, much more time is spent waiting on the application backends
> then with the clients, especially now that we have the event mpm. Something
> like this would allow me to drastically reduce thread counts and free up a lot of
> associated memory.
> >
> >
> > Rick Houser
> > Web Administration
> > (517)367-3516
> >
> >
> >> -----Original Message-----
> >> From: Plüm, Rüdiger, Vodafone Group
> [mailto:ruediger.pluem@vodafone.com]
> >> Sent: Thursday, December 03, 2015 10:50 AM
> >> To: dev@httpd.apache.org
> >> Subject: AW: reverse proxy wishlist
> >>
> >> How about an async proxy that frees up the thread while waiting on a slow
> >> backend?
> >>
> >> Regards
> >>
> >> Rüdiger
> >>
> >>> -----Ursprüngliche Nachricht-----
> >>> Von: Jim Jagielski [mailto:jim@jaguNET.com]
> >>> Gesendet: Donnerstag, 3. Dezember 2015 15:59
> >>> An: httpd <de...@httpd.apache.org>
> >>> Betreff: reverse proxy wishlist
> >>>
> >>> I put out a call on Twitter regarding this, but wanted to
> >>> close the loop here as well.
> >>>
> >>> What would *you* like to see as new features or enhancements
> >>> w/ mod_proxy, esp reverse proxy. I was thinking about some
> >>> sort of active backend monitoring, utilizing watchdog, which
> >>> could also maybe, eventually, pull in performance and load
> >>> data for the backend for a more accurate LB provider. But
> >>> what about new LB methods? Any ideas there?
> >>>
> >>> tia.
Re: reverse proxy wishlist
Posted by Jim Jagielski <ji...@jaguNET.com>.
Thx! assuming slow backends, how would you like httpd to
handle it: should it just slurp in the data from the backend
and buffer it and send it to the client all in one go? Should
it instead forward data as soon as it gets it?
> On Dec 3, 2015, at 12:36 PM, Houser, Rick <ri...@jackson.com> wrote:
>
> An async mod_proxy backend would be huge for my workloads. In the JEE space I deal with, much more time is spent waiting on the application backends then with the clients, especially now that we have the event mpm. Something like this would allow me to drastically reduce thread counts and free up a lot of associated memory.
>
>
> Rick Houser
> Web Administration
> (517)367-3516
>
>
>> -----Original Message-----
>> From: Plüm, Rüdiger, Vodafone Group [mailto:ruediger.pluem@vodafone.com]
>> Sent: Thursday, December 03, 2015 10:50 AM
>> To: dev@httpd.apache.org
>> Subject: AW: reverse proxy wishlist
>>
>> How about an async proxy that frees up the thread while waiting on a slow
>> backend?
>>
>> Regards
>>
>> Rüdiger
>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Jim Jagielski [mailto:jim@jaguNET.com]
>>> Gesendet: Donnerstag, 3. Dezember 2015 15:59
>>> An: httpd <de...@httpd.apache.org>
>>> Betreff: reverse proxy wishlist
>>>
>>> I put out a call on Twitter regarding this, but wanted to
>>> close the loop here as well.
>>>
>>> What would *you* like to see as new features or enhancements
>>> w/ mod_proxy, esp reverse proxy. I was thinking about some
>>> sort of active backend monitoring, utilizing watchdog, which
>>> could also maybe, eventually, pull in performance and load
>>> data for the backend for a more accurate LB provider. But
>>> what about new LB methods? Any ideas there?
>>>
>>> tia.
RE: reverse proxy wishlist
Posted by "Houser, Rick" <ri...@jackson.com>.
An async mod_proxy backend would be huge for my workloads. In the JEE space I deal with, much more time is spent waiting on the application backends then with the clients, especially now that we have the event mpm. Something like this would allow me to drastically reduce thread counts and free up a lot of associated memory.
Rick Houser
Web Administration
(517)367-3516
> -----Original Message-----
> From: Plüm, Rüdiger, Vodafone Group [mailto:ruediger.pluem@vodafone.com]
> Sent: Thursday, December 03, 2015 10:50 AM
> To: dev@httpd.apache.org
> Subject: AW: reverse proxy wishlist
>
> How about an async proxy that frees up the thread while waiting on a slow
> backend?
>
> Regards
>
> Rüdiger
>
> > -----Ursprüngliche Nachricht-----
> > Von: Jim Jagielski [mailto:jim@jaguNET.com]
> > Gesendet: Donnerstag, 3. Dezember 2015 15:59
> > An: httpd <de...@httpd.apache.org>
> > Betreff: reverse proxy wishlist
> >
> > I put out a call on Twitter regarding this, but wanted to
> > close the loop here as well.
> >
> > What would *you* like to see as new features or enhancements
> > w/ mod_proxy, esp reverse proxy. I was thinking about some
> > sort of active backend monitoring, utilizing watchdog, which
> > could also maybe, eventually, pull in performance and load
> > data for the backend for a more accurate LB provider. But
> > what about new LB methods? Any ideas there?
> >
> > tia.
Re: AW: reverse proxy wishlist
Posted by Graham Leggett <mi...@sharp.fm>.
On 3 Dec 2015, at 15:50, Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com> wrote:
> How about an async proxy that frees up the thread while waiting on a slow backend?
This is generic httpd thing, not restricted to mod_proxy.
From what I have seen it shouldn't be difficult, we just need a way for EAGAIN to be properly handled with the ability to skip forward back to a hook when we come back.
I looked at doing this in the filter stack, which is also possible, but doing this generically across the server is better.
Regards,
Graham
--
AW: reverse proxy wishlist
Posted by Plüm,
Rüdiger,
Vodafone Group <ru...@vodafone.com>.
How about an async proxy that frees up the thread while waiting on a slow backend?
Regards
Rüdiger
> -----Ursprüngliche Nachricht-----
> Von: Jim Jagielski [mailto:jim@jaguNET.com]
> Gesendet: Donnerstag, 3. Dezember 2015 15:59
> An: httpd <de...@httpd.apache.org>
> Betreff: reverse proxy wishlist
>
> I put out a call on Twitter regarding this, but wanted to
> close the loop here as well.
>
> What would *you* like to see as new features or enhancements
> w/ mod_proxy, esp reverse proxy. I was thinking about some
> sort of active backend monitoring, utilizing watchdog, which
> could also maybe, eventually, pull in performance and load
> data for the backend for a more accurate LB provider. But
> what about new LB methods? Any ideas there?
>
> tia.
Re: reverse proxy wishlist
Posted by Jim Jagielski <ji...@jaguNET.com>.
> On Dec 7, 2015, at 8:32 AM, Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com> wrote:
> So I would go for a pragmatic approach with no sharp line. I guess we can bundle all balancers we deliver into one module. IMHO we wouldn't lose anything here
In this particular case, I would be +1. They are extremely small.
AW: reverse proxy wishlist
Posted by Plüm,
Rüdiger,
Vodafone Group <ru...@vodafone.com>.
> -----Ursprüngliche Nachricht-----
> Von: Jim Jagielski [mailto:jim@jaguNET.com]
> Gesendet: Montag, 7. Dezember 2015 14:09
> An: dev@httpd.apache.org
> Betreff: Re: reverse proxy wishlist
>
>
> > On Dec 5, 2015, at 9:30 PM, William A Rowe Jr <wr...@rowe-clan.net>
> wrote:
> >
> > On Sat, Dec 5, 2015 at 3:48 PM, Jim Jagielski <ji...@jagunet.com> wrote:
> >
> > > On Dec 4, 2015, at 10:25 AM, William A Rowe Jr <wr...@rowe-clan.net>
> wrote:
> > >
> > > My observation was that that the mapped pages for 2-6 fundamental
> socache, lbmethod or slotmem providers are the same as for a single
> module due to page alignment - load any two and you are already wasting
> kernel resources and memory.
> > >
> >
> > Since you refer to these 3 modules, what to you recommend? That
> > we bundle all providers into 1 module? That's fine with me but
> > my point was that the whole idea of submodules and providers,
> > whether from us or not, was that the admin only needs to load
> > the ones they use. This recommendation seems to ignore this
> > basic advice.
> >
> > That advise always applies if the module interacts with the user-
> agent.
> >
>
> I have no idea why it also doesn't apply no matter what.
> Isn't the whole idea of a loadable modules is so one
> can add, or remove, functionality, no matter what, when
> it's not needed? The idea of it only applying when it (directly)
> "interacts with the user-agent" seems extremely artificial.
I guess the point is a tradeoff, between "wasting" Kernel resources and pages by loading lots of very small modules and being able to really only load the code you need.
As with all such stuff there is no real sharp line. For some one is more important than the other one. For others it is vice versa.
So I would go for a pragmatic approach with no sharp line. I guess we can bundle all balancers we deliver into one module. IMHO we wouldn't lose anything here.
Of course we keep it possible to have balancers in separate modules for other people. With other modules I am not so sure if this really makes sense and we should keep it
as is.
Regards
Rüdiger
Re: reverse proxy wishlist
Posted by Jim Jagielski <ji...@jaguNET.com>.
> On Dec 5, 2015, at 9:30 PM, William A Rowe Jr <wr...@rowe-clan.net> wrote:
>
> On Sat, Dec 5, 2015 at 3:48 PM, Jim Jagielski <ji...@jagunet.com> wrote:
>
> > On Dec 4, 2015, at 10:25 AM, William A Rowe Jr <wr...@rowe-clan.net> wrote:
> >
> > My observation was that that the mapped pages for 2-6 fundamental socache, lbmethod or slotmem providers are the same as for a single module due to page alignment - load any two and you are already wasting kernel resources and memory.
> >
>
> Since you refer to these 3 modules, what to you recommend? That
> we bundle all providers into 1 module? That's fine with me but
> my point was that the whole idea of submodules and providers,
> whether from us or not, was that the admin only needs to load
> the ones they use. This recommendation seems to ignore this
> basic advice.
>
> That advise always applies if the module interacts with the user-agent.
>
I have no idea why it also doesn't apply no matter what.
Isn't the whole idea of a loadable modules is so one
can add, or remove, functionality, no matter what, when
it's not needed? The idea of it only applying when it (directly)
"interacts with the user-agent" seems extremely artificial.
Re: reverse proxy wishlist
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Sat, Dec 5, 2015 at 3:48 PM, Jim Jagielski <ji...@jagunet.com> wrote:
>
> > On Dec 4, 2015, at 10:25 AM, William A Rowe Jr <wr...@rowe-clan.net>
> wrote:
> >
> > My observation was that that the mapped pages for 2-6 fundamental
> socache, lbmethod or slotmem providers are the same as for a single module
> due to page alignment - load any two and you are already wasting kernel
> resources and memory.
> >
>
> Since you refer to these 3 modules, what to you recommend? That
> we bundle all providers into 1 module? That's fine with me but
> my point was that the whole idea of submodules and providers,
> whether from us or not, was that the admin only needs to load
> the ones they use. This recommendation seems to ignore this
> basic advice.
>
That advise always applies if the module interacts with the user-agent.
I just finished pointing out that only the admin causes httpd to interact
with different socache, lbmethod or slotmem providers. The modules
of each group combined, while appearing larger are going to occupy
the same space all together in the paging schema as any one module
alone. So merging the 'core' providers into a single module is a win
for each admin who is using two or more mechanisms, and net-net
null for any admin who consumes only one. Those who don't use
the facility at all need not load even these core groups.
Re: reverse proxy wishlist
Posted by Jim Jagielski <ji...@jaguNET.com>.
> On Dec 4, 2015, at 10:25 AM, William A Rowe Jr <wr...@rowe-clan.net> wrote:
>
> My observation was that that the mapped pages for 2-6 fundamental socache, lbmethod or slotmem providers are the same as for a single module due to page alignment - load any two and you are already wasting kernel resources and memory.
>
Since you refer to these 3 modules, what to you recommend? That
we bundle all providers into 1 module? That's fine with me but
my point was that the whole idea of submodules and providers,
whether from us or not, was that the admin only needs to load
the ones they use. This recommendation seems to ignore this
basic advice.
Re: reverse proxy wishlist
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
My observation was that that the mapped pages for 2-6 fundamental socache,
lbmethod or slotmem providers are the same as for a single module due to
page alignment - load any two and you are already wasting kernel resources
and memory.
I agree that any user agent or content facing modules should remain
distinct - only load those you use and trust. These core feature sets,
when used, only face the conf file or in the case of lbmethod, the admin
console. So they should be safe to load in parallel, when leveraged for a
given module. They should even be safe to compile static, if the user
likes.
On Dec 4, 2015 07:33, "Jim Jagielski" <ji...@jagunet.com> wrote:
> I thought the idea of providers and submodules were so
> that, for example, if someone only used byrequests, for
> example, they only needed to build and load that specific
> submodule and nothing else... Not seeing what issue exactly
> you're trying to address.
>
> > On Dec 3, 2015, at 6:25 PM, William A Rowe Jr <wr...@rowe-clan.net>
> wrote:
> >
> > On Thu, Dec 3, 2015 at 3:40 PM, Jim Jagielski <ji...@jagunet.com> wrote:
> >
> > > On Dec 3, 2015, at 11:09 AM, William A Rowe Jr <wr...@rowe-clan.net>
> wrote:
> > >
> > > My personal wish list is that we eliminate module bloat by coalescing
> > > alternative "standard" implementations into a single module again in
> > > 2.next, and not just limited to lbmethod, but also the core socache
> > > and slotmem providers. Most stock/core implementations shouldn't
> > > change if a user wants to plug in 'yet another' option, but there is
> really
> > > no excuse for us to map so many ldobjects and text pages into the
> > > memory map of a given process, when the actual program text page
> > > of each is a few hundred opcodes, max.
> > >
> >
> > Not following you there... what do you mean? You mention the
> > lbmethods; do you mean instead of having each method be its
> > own module, making them all one?
> >
> > Not specific to lbmethods but yes, one mod_lbmethod_core.so provider
> > of the basic set of 3-4 methods that right now eat up a lot of 64kb
> process
> > page mappings for no particular benefit. The builder could even decide
> these
> > are compiled-in to the base httpd. We don't dismiss the loadable
> approach,
> > we simply decide this subset is effectively baked, and then use the
> loadable
> > lbmethod to extend yet another experimental/dynamic/evolving method, from
> > the ASF or third parties.
> >
> >
>
>
Re: reverse proxy wishlist
Posted by Jim Jagielski <ji...@jaguNET.com>.
I thought the idea of providers and submodules were so
that, for example, if someone only used byrequests, for
example, they only needed to build and load that specific
submodule and nothing else... Not seeing what issue exactly
you're trying to address.
> On Dec 3, 2015, at 6:25 PM, William A Rowe Jr <wr...@rowe-clan.net> wrote:
>
> On Thu, Dec 3, 2015 at 3:40 PM, Jim Jagielski <ji...@jagunet.com> wrote:
>
> > On Dec 3, 2015, at 11:09 AM, William A Rowe Jr <wr...@rowe-clan.net> wrote:
> >
> > My personal wish list is that we eliminate module bloat by coalescing
> > alternative "standard" implementations into a single module again in
> > 2.next, and not just limited to lbmethod, but also the core socache
> > and slotmem providers. Most stock/core implementations shouldn't
> > change if a user wants to plug in 'yet another' option, but there is really
> > no excuse for us to map so many ldobjects and text pages into the
> > memory map of a given process, when the actual program text page
> > of each is a few hundred opcodes, max.
> >
>
> Not following you there... what do you mean? You mention the
> lbmethods; do you mean instead of having each method be its
> own module, making them all one?
>
> Not specific to lbmethods but yes, one mod_lbmethod_core.so provider
> of the basic set of 3-4 methods that right now eat up a lot of 64kb process
> page mappings for no particular benefit. The builder could even decide these
> are compiled-in to the base httpd. We don't dismiss the loadable approach,
> we simply decide this subset is effectively baked, and then use the loadable
> lbmethod to extend yet another experimental/dynamic/evolving method, from
> the ASF or third parties.
>
>
Re: reverse proxy wishlist
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Thu, Dec 3, 2015 at 3:40 PM, Jim Jagielski <ji...@jagunet.com> wrote:
>
> > On Dec 3, 2015, at 11:09 AM, William A Rowe Jr <wr...@rowe-clan.net>
> wrote:
> >
> > My personal wish list is that we eliminate module bloat by coalescing
> > alternative "standard" implementations into a single module again in
> > 2.next, and not just limited to lbmethod, but also the core socache
> > and slotmem providers. Most stock/core implementations shouldn't
> > change if a user wants to plug in 'yet another' option, but there is
> really
> > no excuse for us to map so many ldobjects and text pages into the
> > memory map of a given process, when the actual program text page
> > of each is a few hundred opcodes, max.
> >
>
> Not following you there... what do you mean? You mention the
> lbmethods; do you mean instead of having each method be its
> own module, making them all one?
>
Not specific to lbmethods but yes, one mod_lbmethod_core.so provider
of the basic set of 3-4 methods that right now eat up a lot of 64kb process
page mappings for no particular benefit. The builder could even decide
these
are compiled-in to the base httpd. We don't dismiss the loadable approach,
we simply decide this subset is effectively baked, and then use the loadable
lbmethod to extend yet another experimental/dynamic/evolving method, from
the ASF or third parties.
Re: reverse proxy wishlist
Posted by Jim Jagielski <ji...@jaguNET.com>.
> On Dec 3, 2015, at 11:09 AM, William A Rowe Jr <wr...@rowe-clan.net> wrote:
>
> My personal wish list is that we eliminate module bloat by coalescing
> alternative "standard" implementations into a single module again in
> 2.next, and not just limited to lbmethod, but also the core socache
> and slotmem providers. Most stock/core implementations shouldn't
> change if a user wants to plug in 'yet another' option, but there is really
> no excuse for us to map so many ldobjects and text pages into the
> memory map of a given process, when the actual program text page
> of each is a few hundred opcodes, max.
>
Not following you there... what do you mean? You mention the
lbmethods; do you mean instead of having each method be its
own module, making them all one?
Re: reverse proxy wishlist
Posted by Jim Jagielski <ji...@jaguNET.com>.
afaik, we aren't tied to nghttp2, in that if there
is a more viable solution and it's easier to implement/use
that than, without a whole lot of work, it's something to
consider. Back then, and for now, nghttp2 made the whole
thing possible. If serf gets to the point where it
could be used, then cool.
The issue has always been that except for a handful of
people, *how* serf works has always been a mystery. It's
documentation is awful and even though its used in svn,
there is still limited knowledge externally on how it
can, and should, be used.
> On Dec 3, 2015, at 7:05 PM, Bert Huijben <be...@qqmail.nl> wrote:
>
>
>
>> -----Original Message-----
>> From: Jim Jagielski [mailto:jim@jaguNET.com]
>> Sent: donderdag 3 december 2015 22:20
>> To: dev@httpd.apache.org
>> Subject: Re: reverse proxy wishlist
>>
>>
>>> On Dec 3, 2015, at 11:09 AM, William A Rowe Jr <wr...@rowe-clan.net>
>> wrote:
>>>
>>> On Thu, Dec 3, 2015 at 8:59 AM, Jim Jagielski <ji...@jagunet.com> wrote:
>>>
>>> What would *you* like to see as new features or enhancements
>>> w/ mod_proxy, esp reverse proxy.
>>>
>>> HTTP/2 support, of course :) It will be interesting to be able to
> leverage
>>> and compare a mod_proxy_serf vs a mod_proxy_http2 (nghttp2-based)
>>> engine, as mentioned in another thread - multiple implementations
>>> are always good for ferreting out protocol anomalies.
>>>
>>
>> It's kind of funny... the "need" for http/2 between proxy and
>> origin seems pretty non-existant. There is a blog post by Cloudflare
>> somewhere about how they don't see servers talking http/2 to the
>> backend as anywhere near a driver, since all the things that make
>> it "important" (koff koff!) between browser and server don't really
>> apply.
>
> After having implemented fcgi (server support) and http/2 (server and
> client) in Apache Serf I was thinking that it would be nice if H2 would
> replace the existing server side protocols.
>
> http/1.1 requires chunking or explicit content-length, while http/2 and fcgi
> don't have that requirement.
>
>
>
> The reason I implemented the server side of those protocols in Apache Serf
> was exactly to allow writing such origins with Serf...
>
> Adding such a backend server process is one of the (many) possible
> directions Subversion might take in the future.
>
> Bert
RE: reverse proxy wishlist
Posted by Bert Huijben <be...@qqmail.nl>.
> -----Original Message-----
> From: Jim Jagielski [mailto:jim@jaguNET.com]
> Sent: donderdag 3 december 2015 22:20
> To: dev@httpd.apache.org
> Subject: Re: reverse proxy wishlist
>
>
> > On Dec 3, 2015, at 11:09 AM, William A Rowe Jr <wr...@rowe-clan.net>
> wrote:
> >
> > On Thu, Dec 3, 2015 at 8:59 AM, Jim Jagielski <ji...@jagunet.com> wrote:
> >
> > What would *you* like to see as new features or enhancements
> > w/ mod_proxy, esp reverse proxy.
> >
> > HTTP/2 support, of course :) It will be interesting to be able to
leverage
> > and compare a mod_proxy_serf vs a mod_proxy_http2 (nghttp2-based)
> > engine, as mentioned in another thread - multiple implementations
> > are always good for ferreting out protocol anomalies.
> >
>
> It's kind of funny... the "need" for http/2 between proxy and
> origin seems pretty non-existant. There is a blog post by Cloudflare
> somewhere about how they don't see servers talking http/2 to the
> backend as anywhere near a driver, since all the things that make
> it "important" (koff koff!) between browser and server don't really
> apply.
After having implemented fcgi (server support) and http/2 (server and
client) in Apache Serf I was thinking that it would be nice if H2 would
replace the existing server side protocols.
http/1.1 requires chunking or explicit content-length, while http/2 and fcgi
don't have that requirement.
The reason I implemented the server side of those protocols in Apache Serf
was exactly to allow writing such origins with Serf...
Adding such a backend server process is one of the (many) possible
directions Subversion might take in the future.
Bert
Re: reverse proxy wishlist
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Thu, Dec 3, 2015 at 3:20 PM, Jim Jagielski <ji...@jagunet.com> wrote:
>
> > On Dec 3, 2015, at 11:09 AM, William A Rowe Jr <wr...@rowe-clan.net>
> wrote:
> >
> > On Thu, Dec 3, 2015 at 8:59 AM, Jim Jagielski <ji...@jagunet.com> wrote:
> >
> > What would *you* like to see as new features or enhancements
> > w/ mod_proxy, esp reverse proxy.
> >
> > HTTP/2 support, of course :) It will be interesting to be able to
> leverage
> > and compare a mod_proxy_serf vs a mod_proxy_http2 (nghttp2-based)
> > engine, as mentioned in another thread - multiple implementations
> > are always good for ferreting out protocol anomalies.
> >
>
> It's kind of funny... the "need" for http/2 between proxy and
> origin seems pretty non-existant. There is a blog post by Cloudflare
> somewhere about how they don't see servers talking http/2 to the
> backend as anywhere near a driver, since all the things that make
> it "important" (koff koff!) between browser and server don't really
> apply.
>
AIUI the reason for http/2 to ignore the OSI Network definition was that
they knew better, and there is a demand for concurrent requests and
responses. It seems that fits the fat-pipe of 400 concurrent requests
between a gateway and backend app server more than the few pipes
needed between a web browser and the gateway. /shrug :)
> Even so, I do think that re-looking into leveraging serf might
> be useful, even if for other reasons.
>
I'd agree, either as the h2, h2c or http/1.1 provider.
Re: reverse proxy wishlist
Posted by Jim Jagielski <ji...@jaguNET.com>.
> On Dec 3, 2015, at 11:09 AM, William A Rowe Jr <wr...@rowe-clan.net> wrote:
>
> On Thu, Dec 3, 2015 at 8:59 AM, Jim Jagielski <ji...@jagunet.com> wrote:
>
> What would *you* like to see as new features or enhancements
> w/ mod_proxy, esp reverse proxy.
>
> HTTP/2 support, of course :) It will be interesting to be able to leverage
> and compare a mod_proxy_serf vs a mod_proxy_http2 (nghttp2-based)
> engine, as mentioned in another thread - multiple implementations
> are always good for ferreting out protocol anomalies.
>
It's kind of funny... the "need" for http/2 between proxy and
origin seems pretty non-existant. There is a blog post by Cloudflare
somewhere about how they don't see servers talking http/2 to the
backend as anywhere near a driver, since all the things that make
it "important" (koff koff!) between browser and server don't really
apply.
Even so, I do think that re-looking into leveraging serf might
be useful, even if for other reasons.
Re: reverse proxy wishlist
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Fri, Dec 4, 2015 at 10:17 AM, Nick Kew <ni...@webthing.com> wrote:
>
> > I'm looking, none of these seem like huge hacks, wondering
> > which of them trigger your concern?
>
> Well, your talk of refactoring config led me to wonder
> whether you were proposing another tilt at the whole directory
> walk stuff. While that may have merit, past experience
> tells us it's not going to be easy!
>
If I never have to refactor that code again, it will still be too soon :-/
More likely that someone enthusiastic could take the base/short
unique <Directory> blocks and pre-merge those to global/vhost
scope, as the shortcut to a more specific scope as the first-match
hit. Deeper pre-merge optimizations would be rough and all of it
presupposes there are no request-specific eval blocks and no
directory permitting an .htaccess could be optimized that way.
So... no... much like the non-iterating fnmatch implementation, I'd
prefer to let others wrap their heads around that puzzle and come
up with something new and more efficient. All I was thinking of
doing was extending the core concept of 'compiled in modules'
to allow a module-of-modules, and build unique create/merge
call tables at the end of register_hooks, instead of the slightly
odd way we we presume that each module does or does not have
one of each such creater/merger, skipping along the NULL refs.
When we last tried to optimize this, hooks were attempted and
proved to have much too much overhead. A startup-time straight
array of function pointer/conf index pointer elements will be
the only way to keep the logic as efficient as it is today, and
then build upon that model. I don't expect we will allow the
create/merge table to change after the register_hooks phase.
Re: reverse proxy wishlist
Posted by Nick Kew <ni...@webthing.com>.
On Thu, 3 Dec 2015 12:23:24 -0600
William A Rowe Jr <wr...@rowe-clan.net> wrote:
> On Thu, Dec 3, 2015 at 10:32 AM, Nick Kew <ni...@apache.org> wrote:
> Yup, and I'm not proposing to eliminate the mechanism, I'm proposing
> that the existing 'core' subset be codified in fewer, still
> lightweight modules.
Fairy nuff. At this level of detail I can't really tell
if you're proposing anything that would concern me.
> I'm looking, none of these seem like huge hacks, wondering
> which of them trigger your concern?
Well, your talk of refactoring config led me to wonder
whether you were proposing another tilt at the whole directory
walk stuff. While that may have merit, past experience
tells us it's not going to be easy!
--
Nick Kew
Re: reverse proxy wishlist
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Thu, Dec 3, 2015 at 10:32 AM, Nick Kew <ni...@apache.org> wrote:
> On Thu, 3 Dec 2015 10:09:08 -0600
> William A Rowe Jr <wr...@rowe-clan.net> wrote:
>
> > Most stock/core implementations shouldn't
> > change if a user wants to plug in 'yet another' option, but there is
> > really no excuse for us to map so many ldobjects and text pages into
> > the memory map of a given process, when the actual program text page
> > of each is a few hundred opcodes, max.
>
> Extensibility. Flexibility.
>
> They're our biggest single strength when compared to
> the likes of nginx/tengine.
>
Yup, and I'm not proposing to eliminate the mechanism, I'm proposing
that the existing 'core' subset be codified in fewer, still lightweight
modules.
> > (You can submit that the user could want to replace the bytraffic
> > implementation, for example, but I'd counter that they can certainly
> > rebuild any mod_lbmethod_core module with the others still using
> > stock sources, and deploy their alternative, or they can give their
> > custom fork of a provide yet another provider name.)
>
> Someone wanting different functionality shouldn't have to
> take on the maintenance headache of hacking into one of
> our modules. We have our API/ABI promise to make life easy
> for them introducing their own modules.
>
If they are *replacing* our tokens, that's on them. They can add their
own lbname, e.g. bytraffic-hacked, without rebuilding a core provider.
And I'm not going to propose any of this for backport, in fact I don't plan
to condense the source files at all, only simplify their hook registration
and compile the several providers under a core module umbrella.
> I'm not asking anyone to coalesce these, though. It was already
> > on my rather long list of 'nice to have' along with supporting
> > multiple conf mergers per module (effectively allowing 'multiple
> > modules' to be a single module and splitting our monstrous core
> > server and dir configs into related digestible pieces that rarely
> > need to be merged, and optimizing away modules with no conf merge at
> > all). And along with cleaning up the httpd 2.next API, and i18n of
> > error output which I am continuing on first once the mod_ssl issues
> > for mod_ftp are resolved (my current project).
>
> Hmmm. There's some nice-to-have in there, but it also sounds like
> a big hack.
>
Actually they are the ideas and justifications;
1. multiple conf sections) right now, a module such as core or mod_ssl
has a small subset of "key" config directives that change often,
perhaps in nearly every <VirtualHost > block or <Directory > level.
Then there is a majority of directives that never change from either
the global or first-triggered block (e.g. toggling auth from denied to
a site-wide global permitted state). These are merges that just
should not have to happen on each and every merge. Directory is
obviously the worst offender, while Server blocks are merged at
startup and don't enjoy nearly as much optimization. Dropping
any module from the merge list that has a NULL merge should
speed up directory merges, too. Unsuitable for backport.
2. API cleanup) I'm looking at a util_compat.h that would map all of
our long-deprecated ap_ functions into their apr_ or newer _ex
equivalents. And busting up httpd.h, starting with util_strings.h
Unsuitable for backport.
3. i18n) You ask about our "strengths", we have little globally until
we actually internationalize error output. Most servers beat us
a long time ago in this respect. Errors are more confusing than
learning the few tokens you need to write a config (however, some
internationalized .conf.in files for descriptive text would be a help.)
The specific error lookup has can be overridden per-server, giving
shared hosting providers the ability to let vhost admins grok the
errors their site is receiving. It's nothing more than a hash table
now that we have decorated APnnnnn throughout the 2.x and 2.4
error output, so this one can be considered for backport.
4. mod_ssl) yes, I know my last Upgrade: TLS/1.0 patch doesn't
interact with the Protocols directive and h2c, since Protocols
directive patch failed to pick up the existing mod_ssl Upgrade/
Connection logic. Working on that fix right now so that both
options present just one unified Upgrade/Connection message.
That fix aught to be backported for completeness.
5. mod_ftp) several recent mod_ssl changes busted mod_ftp's
AUTH TLS support - and possibly due to my implicit HOST
command hack (the result of a HOST command is the same
221 welcome as an initial connection, so this was a fakeout
to cause that initial greeting - based on the SNI hostname in
the case of explicit SSL.) Both the implicit port 995 and
plaintext port 21 flavors were and are still working, so it is
just a matter of better coupling mod_ftp AUTH TLS to the
upgrade mechanisms of mod_ssl. Hopefully minimal or no
core mod_ssl changes are needed.
I'm looking, none of these seem like huge hacks, wondering
which of them trigger your concern?
Cheers,
Bill
Re: reverse proxy wishlist
Posted by Nick Kew <ni...@apache.org>.
On Thu, 3 Dec 2015 10:09:08 -0600
William A Rowe Jr <wr...@rowe-clan.net> wrote:
> Most stock/core implementations shouldn't
> change if a user wants to plug in 'yet another' option, but there is
> really no excuse for us to map so many ldobjects and text pages into
> the memory map of a given process, when the actual program text page
> of each is a few hundred opcodes, max.
Extensibility. Flexibility.
They're our biggest single strength when compared to
the likes of nginx/tengine.
> (You can submit that the user could want to replace the bytraffic
> implementation, for example, but I'd counter that they can certainly
> rebuild any mod_lbmethod_core module with the others still using
> stock sources, and deploy their alternative, or they can give their
> custom fork of a provide yet another provider name.)
Someone wanting different functionality shouldn't have to
take on the maintenance headache of hacking into one of
our modules. We have our API/ABI promise to make life easy
for them introducing their own modules.
> I'm not asking anyone to coalesce these, though. It was already
> on my rather long list of 'nice to have' along with supporting
> multiple conf mergers per module (effectively allowing 'multiple
> modules' to be a single module and splitting our monstrous core
> server and dir configs into related digestible pieces that rarely
> need to be merged, and optimizing away modules with no conf merge at
> all). And along with cleaning up the httpd 2.next API, and i18n of
> error output which I am continuing on first once the mod_ssl issues
> for mod_ftp are resolved (my current project).
Hmmm. There's some nice-to-have in there, but it also sounds like
a big hack.
> Last thought, you might want to share your question with users@?
+1
--
Nick Kew
Re: reverse proxy wishlist
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Thu, Dec 3, 2015 at 8:59 AM, Jim Jagielski <ji...@jagunet.com> wrote:
>
> What would *you* like to see as new features or enhancements
> w/ mod_proxy, esp reverse proxy.
HTTP/2 support, of course :) It will be interesting to be able to leverage
and compare a mod_proxy_serf vs a mod_proxy_http2 (nghttp2-based)
engine, as mentioned in another thread - multiple implementations
are always good for ferreting out protocol anomalies.
AIUI in the TLS age of HTTP, there is no forward proxy use case for
HTTP/2 that I can surmise, except as a CONNECT tunnel which 1.1
handled just fine already (AIUI that tunnel can then be a multiplexed
HTTP/2 conversation to the tunneled server.) So this whole discussion
seems to revolve around reverse proxy backend connection. Windowing
might prove effective at reducing huge connection pools to app servers,
if they can be multiplexed over more heavily loaded fast TCP
connections.
My personal wish list is that we eliminate module bloat by coalescing
alternative "standard" implementations into a single module again in
2.next, and not just limited to lbmethod, but also the core socache
and slotmem providers. Most stock/core implementations shouldn't
change if a user wants to plug in 'yet another' option, but there is really
no excuse for us to map so many ldobjects and text pages into the
memory map of a given process, when the actual program text page
of each is a few hundred opcodes, max.
(You can submit that the user could want to replace the bytraffic
implementation, for example, but I'd counter that they can certainly
rebuild any mod_lbmethod_core module with the others still using
stock sources, and deploy their alternative, or they can give their
custom fork of a provide yet another provider name.)
I'm not asking anyone to coalesce these, though. It was already
on my rather long list of 'nice to have' along with supporting multiple
conf mergers per module (effectively allowing 'multiple modules' to
be a single module and splitting our monstrous core server and dir
configs into related digestible pieces that rarely need to be merged,
and optimizing away modules with no conf merge at all). And along
with cleaning up the httpd 2.next API, and i18n of error output
which I am continuing on first once the mod_ssl issues for mod_ftp
are resolved (my current project).
I was thinking about some
> sort of active backend monitoring, utilizing watchdog, which
> could also maybe, eventually, pull in performance and load
> data for the backend for a more accurate LB provider.
>
I had the impression there was some effort out there to create a
schema for backend servers to report load in a standardized way,
beyond heartbeat? But I've lost my references.
Last thought, you might want to share your question with users@?
Cheers,
Bill
Re: reverse proxy wishlist
Posted by Christian Folini <ch...@netnea.com>.
On Sat, Dec 05, 2015 at 11:01:54AM +0000, Tim Bannister wrote:
> ProxyErrorOverride is a good starting point. Often I want to let through only some error pages: the ones explicitly coded to be shown to this website's visitors. If the backend fails and produces an unstyled page of jargon and diagnostics, I want httpd to intervene.
I'd like to follow up on that. The last time I checked,
ProxyErrorOverride was silent in the logs. A notice- or warning-level
message when it intervenes would be helpful in many situations. The
typical conversation around ProxyErrorOverride starts with "It's the
error page of the proxy, so the proxy must have caused the error." That
discussion could be cut short with a log message stating
~"ProxyErrorOverride applied after receiving status XXX from the
backend."
> The application could signal to httpd that its response has a user-friendly body via a special header.
I thought about this before. I think a more general, more flexible
approach would be very helpful. In the end it boils down to something
like rewrite rules on the response.
You can to this with ModSecurity, but that is too late for
ProxyErrorOverride AFAICT.
Ahoj,
Christian Folini
--
Christian Folini - <ch...@netnea.com>
Re: reverse proxy wishlist
Posted by Tim Bannister <is...@c8h10n4o2.org.uk>.
On 3 December 2015 14:59:00 GMT, Jim Jagielski wrote:
>What would *you* like to see as new features or enhancements
>w/ mod_proxy, esp reverse proxy.
I'd like to have more options about error responses. Where httpd is reverse proxy for an application that may fail, and I want to have httpd send nicer 5xx responses.
ProxyErrorOverride is a good starting point. Often I want to let through only some error pages: the ones explicitly coded to be shown to this website's visitors. If the backend fails and produces an unstyled page of jargon and diagnostics, I want httpd to intervene.
The application could signal to httpd that its response has a user-friendly body via a special header.
I don't think httpd can do what I have in mind yet (maybe with mod_lua, but that's too much for many webmasters).
Tim
--
Tim Bannister – isoma@c8h10n4o2.org.uk