You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Jim Jagielski <ji...@jaguNET.com> on 2008/02/13 17:07:28 UTC
ping for http in mod_proxy
I've started looking at adding "ping" support for
mod_proxy_http to complement whats in mod_proxy_ajp...
The idea is to send a simple OPTIONS * to the backend
and hope for a reply.
The rub is that I've been working on 2 separate
implementations: one talks direct to the socket and the
other basically tries to do what ap_proxy_http_request()
and ap_proxy_http_process_response() does, as far
as using brigades, etc... The former is much further
along, btw.
In any case, both implementations can have what I can
envision as some "weird effects" as far as how they
should actually work in the real world. For example,
should it honor force-proxy-request-1.0 and/or
proxy-nokeepalive, and, if so, how does that affect
the subsequent "real" request... In other words,
it's not as clean as AJP which has a "real" ping.
And all this is wondering whether or not it's
really useful in the 1st place...
So before I go any further, any comments or ideas
on whether is is even a Good Idea or not... I'm starting
to have 2nd thoughts :)
Re: ping for http in mod_proxy
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 13, 2008, at 1:04 PM, Plüm, Rüdiger, VF-Group wrote:
>
> Sorry for my I-want-it-all-at-once approach :-). But this leaves
> the problems (most notably PR 37770) open for SSL backend connection
> (which would be a pity). So IMHO the socket approach would be only a
> first step.
No, I want it all too... The problem is that you then
start going beyond a simply yes/no implementation to something
more robust and information-filled, but also with more overhead...
Re: ping for http in mod_proxy
Posted by Plüm,
Rüdiger,
VF-Group <ru...@vodafone.com>.
> -----Ursprüngliche Nachricht-----
> Von: Jim Jagielski [mailto:jim@jaguNET.com]
> Gesendet: Mittwoch, 13. Februar 2008 18:55
> An: dev@httpd.apache.org
> Betreff: Re: ping for http in mod_proxy
>
>
> On Feb 13, 2008, at 12:23 PM, Plüm, Rüdiger, VF-Group wrote:
>
> >
> >
> >> -----Ursprüngliche Nachricht-----
> >> Von: Jim Jagielski
> >> Gesendet: Mittwoch, 13. Februar 2008 17:07
> >> An: dev@httpd.apache.org
> >> Betreff: ping for http in mod_proxy
> >>
> >> I've started looking at adding "ping" support for
> >> mod_proxy_http to complement whats in mod_proxy_ajp...
> >> The idea is to send a simple OPTIONS * to the backend
> >> and hope for a reply.
> >>
> >> The rub is that I've been working on 2 separate
> >> implementations: one talks direct to the socket and the
> >> other basically tries to do what ap_proxy_http_request()
> >> and ap_proxy_http_process_response() does, as far
> >> as using brigades, etc... The former is much further
> >> along, btw.
> >
> > We cannot use the socket approach since we need to be able to
> > handle SSL backend connections and this requires the brigades -
> > filters
> > approach.
>
> The initial approach was to not honor SSL connections and
> just ignore that... The idea that we really want a fast and
> dirty "Are you there" flag, so to avoid as much overhead
> as possible...
Sorry for my I-want-it-all-at-once approach :-). But this leaves
the problems (most notably PR 37770) open for SSL backend connection
(which would be a pity). So IMHO the socket approach would be only a first step.
Regards
Rüdiger
Re: ping for http in mod_proxy
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 13, 2008, at 12:23 PM, Plüm, Rüdiger, VF-Group wrote:
>
>
>> -----Ursprüngliche Nachricht-----
>> Von: Jim Jagielski
>> Gesendet: Mittwoch, 13. Februar 2008 17:07
>> An: dev@httpd.apache.org
>> Betreff: ping for http in mod_proxy
>>
>> I've started looking at adding "ping" support for
>> mod_proxy_http to complement whats in mod_proxy_ajp...
>> The idea is to send a simple OPTIONS * to the backend
>> and hope for a reply.
>>
>> The rub is that I've been working on 2 separate
>> implementations: one talks direct to the socket and the
>> other basically tries to do what ap_proxy_http_request()
>> and ap_proxy_http_process_response() does, as far
>> as using brigades, etc... The former is much further
>> along, btw.
>
> We cannot use the socket approach since we need to be able to
> handle SSL backend connections and this requires the brigades -
> filters
> approach.
The initial approach was to not honor SSL connections and
just ignore that... The idea that we really want a fast and
dirty "Are you there" flag, so to avoid as much overhead
as possible...
re: ping for http in mod_proxy
Posted by Plüm,
Rüdiger,
VF-Group <ru...@vodafone.com>.
> -----Ursprüngliche Nachricht-----
> Von: Jim Jagielski
> Gesendet: Mittwoch, 13. Februar 2008 17:07
> An: dev@httpd.apache.org
> Betreff: ping for http in mod_proxy
>
> I've started looking at adding "ping" support for
> mod_proxy_http to complement whats in mod_proxy_ajp...
> The idea is to send a simple OPTIONS * to the backend
> and hope for a reply.
>
> The rub is that I've been working on 2 separate
> implementations: one talks direct to the socket and the
> other basically tries to do what ap_proxy_http_request()
> and ap_proxy_http_process_response() does, as far
> as using brigades, etc... The former is much further
> along, btw.
We cannot use the socket approach since we need to be able to
handle SSL backend connections and this requires the brigades - filters
approach.
>
> In any case, both implementations can have what I can
> envision as some "weird effects" as far as how they
> should actually work in the real world. For example,
> should it honor force-proxy-request-1.0 and/or
> proxy-nokeepalive, and, if so, how does that affect
I think the ping is of limited use for HTTP/1.0 / non keepalive requests.
I see it mainly as a measure to avoid race conditions with keepalive
connections that get closed just in the moment as the request is being sent.
The only benefit I would see for HTTP/1.0 / non keepalive requests would be
that a successful connect call (we always create a new connections in this case)
does not guarantee us that you get *any* response (in a timely manner). This could
be because the connect is queued up on the backend in the connection backlog
or in an application specific request queue and will be never processed.
> the subsequent "real" request... In other words,
> it's not as clean as AJP which has a "real" ping.
> And all this is wondering whether or not it's
> really useful in the 1st place...
>
> So before I go any further, any comments or ideas
> on whether is is even a Good Idea or not... I'm starting
> to have 2nd thoughts :)
>
It is a good idea for certain purposes and I really appreciate that you
work on this. This issue is still on my when-I-got-time-TODO list.
If not done until then, maybe something for the hackathon to work on.
Regards
Rüdiger
Re: ping for http in mod_proxy
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 13, 2008, at 1:09 PM, Plüm, Rüdiger, VF-Group wrote:
>
> Right. Furthermore I guess we could create a generic module that
> needs a
> URL configured for a HEAD request that only replies 200 if the
> backend can
> still handle more requests. If it does not respond or with a
> different code
> this backend would be dead. I guess this module needs to fork a
> separate
> process / thread to do this checking periodically and modify the
> worker status
> accordingly.
> So much interesting work and so few time :-(.
Or send a custom request header that has Apache respond with
a custom response header that contains specific health
info...
Re: ping for http in mod_proxy
Posted by Plüm,
Rüdiger,
VF-Group <ru...@vodafone.com>.
> -----Ursprüngliche Nachricht-----
> Von: Jim Jagielski
> Gesendet: Mittwoch, 13. Februar 2008 19:00
> An: dev@httpd.apache.org
> Betreff: Re: ping for http in mod_proxy
>
>
> On Feb 13, 2008, at 12:41 PM, Plüm, Rüdiger, VF-Group wrote:
> >
> > Agreed, but I doubt that it is possible with a reasonable amout of
> > health
> > check frequency to find out before the first real request falls
> > through,
> > provided that your health checks are designed to only fail if the
> > backend
> > is down. If your health checks are smarter and notice that the
> > backend will
> > fail soon (e.g. because it reached 98% or 99% percent of its
> > capacity) then
> > this is a different story and can be very useful.
> > Regarding the reactivation of a failed backend I fully
> agree: Active
> > health
> > checks would be an asset here instead of just retrying failed
> > backends periodically
> > with real requests.
> >
>
> Ideally, it would be nice if we had better insight on the
> actual health of the backends than a simple "do they respond
> to OPTIONS * and how long does it take", but that's pretty
> much all we can do unless go full-on multicasting of info
> ala mod_backhand... At least the balancer is setup to allow
> for load balancing based on that, if the submodule/provider
> actually existed :)
Right. Furthermore I guess we could create a generic module that needs a
URL configured for a HEAD request that only replies 200 if the backend can
still handle more requests. If it does not respond or with a different code
this backend would be dead. I guess this module needs to fork a separate
process / thread to do this checking periodically and modify the worker status
accordingly.
So much interesting work and so few time :-(.
Regards
Rüdiger
Re: ping for http in mod_proxy
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
Graham Leggett wrote:
> Jim Jagielski wrote:
>
>> Ideally, it would be nice if we had better insight on the
>> actual health of the backends than a simple "do they respond
>> to OPTIONS * and how long does it take", but that's pretty
>> much all we can do unless go full-on multicasting of info
>> ala mod_backhand... At least the balancer is setup to allow
>> for load balancing based on that, if the submodule/provider
>> actually existed :)
>
> Is there anything stopping us going the multicasting route, say by
> adding a hook or hooks of some kind to proxy that keeps track of known
> server states?
>
> A dedicated module could then have it's job as handling the
> multicasting and letting the rest of the server know if a server is on
> or off the air, in realtime.
>
> The proxy can then just say "which of these servers should I connect
> to?", and the hook will return the suggested ones in the given order.
>
> As a dedicated module, people who don't need it can turn it off or not
> build it.
this is an idea I've been sitting on for a couple of years now.
Tomcat's cluster can also multicast more than just that the node is
alive, it can broadcast, what applications (URL contexts) are available,
how many threads are in use, what the server load is etc. It can even
broadcast automount points, or when applications are being taken
offline, while the server is still up.
Filip
Re: ping for http in mod_proxy
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 13, 2008, at 6:01 PM, Graham Leggett wrote:
> Jim Jagielski wrote:
>
>> Ideally, it would be nice if we had better insight on the
>> actual health of the backends than a simple "do they respond
>> to OPTIONS * and how long does it take", but that's pretty
>> much all we can do unless go full-on multicasting of info
>> ala mod_backhand... At least the balancer is setup to allow
>> for load balancing based on that, if the submodule/provider
>> actually existed :)
>
> Is there anything stopping us going the multicasting route, say by
> adding a hook or hooks of some kind to proxy that keeps track of
> known server states?
>
> A dedicated module could then have it's job as handling the
> multicasting and letting the rest of the server know if a server is
> on or off the air, in realtime.
>
> The proxy can then just say "which of these servers should I connect
> to?", and the hook will return the suggested ones in the given order.
>
Well, we almost do that now... :)
> As a dedicated module, people who don't need it can turn it off or
> not build it.
>
Re: ping for http in mod_proxy
Posted by "Akins, Brian" <Br...@turner.com>.
On 2/15/08 11:03 AM, "Plüm, Rüdiger, VF-Group" <ru...@vodafone.com>
wrote:
> My main point is that I want to avoid
> using both hook and provider if not really needed, as it
Agreed. Was just stating my preference. As long as it's "easy" to use, I
have no strong feelings either way.
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: ping for http in mod_proxy
Posted by Plüm,
Rüdiger,
VF-Group <ru...@vodafone.com>.
> -----Ursprüngliche Nachricht-----
> Von: Akins, Brian
> Gesendet: Freitag, 15. Februar 2008 16:44
> An: dev@httpd.apache.org
> Betreff: Re: ping for http in mod_proxy
>
> On 2/15/08 8:13 AM, "Plüm, Rüdiger, VF-Group"
> <ru...@vodafone.com>
> wrote:
>
>
> > Any specific reason why we need to add an hook here and why
> this cannot be
> > done by the existing provider (interface). I am scared of
> adding another
> > level of indirection here if it is not really needed and
> things can be already
> > done with the existing infrastructure.
>
>
> I like hooks bcs providers are "one-shot." I use both, but
> find my self
> using hooks more and more.
>
> A good example is the discussion around having stacked providers in
> mod_cache. If it were a hook, you'd already have that...
>
> Providers are good when you will have one, and only one,
> "thing" that needs
> to munge/manipulate/compute like database stuff. With the
> proxy stuff, it
> looks like we want "n" things to be able to manipulate the
> data. Once you
> make the leap from 1 to 2, might as well make it a hook.
But in the Auth framework we also work with providers and may call
more then one provider per request to do authn / authz. So I guess
this will be also doable. My main point is that I want to avoid
using both hook and provider if not really needed, as it
- creates unneeded overhead which lowers performance
- makes it harder to extend things with own modules as things are
more complex
- make debugging harder
So as far as possible for solving the problem in a flexible manner
I would like to stick to KISS.
Regards
Rüdiger
Re: ping for http in mod_proxy
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 15, 2008, at 8:13 AM, Plüm, Rüdiger, VF-Group wrote:
>
>
>> -----Ursprüngliche Nachricht-----
>> Von: Jim Jagielski
>> Gesendet: Freitag, 15. Februar 2008 14:05
>> An: dev@httpd.apache.org
>> Betreff: Re: ping for http in mod_proxy
>>
>>
>>
>> Well, right now all it does is say "give me the best worker" and
>> leaves it up to the selected provider on "what that means", so it
>> queries a provider, not a hook. Of course, we could create an actual
>> hook which then queries providers, to allow people to extend
>> the selection even more...
>
> Any specific reason why we need to add an hook here and why this
> cannot be
> done by the existing provider (interface). I am scared of adding
> another
> level of indirection here if it is not really needed and things can
> be already
> done with the existing infrastructure.
>
No, no reason. In fact, I think the provider interface works
relatively well. I do mention 2 existing hooks which should
be changed (RUN_FIRST to RUN_ALL) which would allow even more
coolness...
I was just thinking that allowing a hook would allow people
to add things without even using providers, which some
people may like. Plus, it seemed an easy accommodation for
people who "wanted" it (I wasn't one).
Re: ping for http in mod_proxy
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Fri, Feb 15, 2008 at 8:26 AM, Jim Jagielski <ji...@jagunet.com> wrote:
> Not quite... there's nothing in providers, per se, that limit
> how they are *used* to be oneshot... Just ask Brad and aaa :)
+1.
I personally find the vtable interface easier to deal with - you have
a 'set' of functions that all go together in one structure - whereas
for a hook, everything must be individual and one-off. *shrug* --
justin
Re: ping for http in mod_proxy
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 15, 2008, at 10:43 AM, Akins, Brian wrote:
> On 2/15/08 8:13 AM, "Plüm, Rüdiger, VF-Group" <ruediger.pluem@vodafone.com
> >
> wrote:
>
>
>> Any specific reason why we need to add an hook here and why this
>> cannot be
>> done by the existing provider (interface). I am scared of adding
>> another
>> level of indirection here if it is not really needed and things can
>> be already
>> done with the existing infrastructure.
>
>
> I like hooks bcs providers are "one-shot." I use both, but find my
> self
> using hooks more and more.
>
> A good example is the discussion around having stacked providers in
> mod_cache. If it were a hook, you'd already have that...
>
> Providers are good when you will have one, and only one, "thing"
> that needs
> to munge/manipulate/compute like database stuff. With the proxy
> stuff, it
> looks like we want "n" things to be able to manipulate the data.
> Once you
> make the leap from 1 to 2, might as well make it a hook.
Not quite... there's nothing in providers, per se, that limit
how they are *used* to be oneshot... Just ask Brad and aaa :)
Re: ping for http in mod_proxy
Posted by "Akins, Brian" <Br...@turner.com>.
On 2/15/08 8:13 AM, "Plüm, Rüdiger, VF-Group" <ru...@vodafone.com>
wrote:
> Any specific reason why we need to add an hook here and why this cannot be
> done by the existing provider (interface). I am scared of adding another
> level of indirection here if it is not really needed and things can be already
> done with the existing infrastructure.
I like hooks bcs providers are "one-shot." I use both, but find my self
using hooks more and more.
A good example is the discussion around having stacked providers in
mod_cache. If it were a hook, you'd already have that...
Providers are good when you will have one, and only one, "thing" that needs
to munge/manipulate/compute like database stuff. With the proxy stuff, it
looks like we want "n" things to be able to manipulate the data. Once you
make the leap from 1 to 2, might as well make it a hook.
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: ping for http in mod_proxy
Posted by Plüm,
Rüdiger,
VF-Group <ru...@vodafone.com>.
> -----Ursprüngliche Nachricht-----
> Von: Jim Jagielski
> Gesendet: Freitag, 15. Februar 2008 14:05
> An: dev@httpd.apache.org
> Betreff: Re: ping for http in mod_proxy
>
>
> On Feb 14, 2008, at 6:41 PM, Graham Leggett wrote:
>
> > Akins, Brian wrote:
> >
> >> Yeah that was my thought. I guess you pass around the array of
> >> servers.
> >> Just remove (or mark as N/A) from the list and/or reorder it. At
> >> the end,
> >> core proxy picks whatever is in index 0 (possibly walking
> the list
> >> in case
> >> of connection error or something).
> >
> > Essentially that is what proxy does now (or did before the
> balancer
> > stuff arrived) - it loops through everything that came back
> from the
> > DNS round robin query, trying to connect in turn until it gets
> > something other than bad gateway.
> >
> > Rather than just the simplistic round robin DNS query, it should
> > query a hook (in which is embedded as the sensible default, a
> > simplistic round robin DNS query).
> >
>
> Well, right now all it does is say "give me the best worker" and
> leaves it up to the selected provider on "what that means", so it
> queries a provider, not a hook. Of course, we could create an actual
> hook which then queries providers, to allow people to extend
> the selection even more...
Any specific reason why we need to add an hook here and why this cannot be
done by the existing provider (interface). I am scared of adding another
level of indirection here if it is not really needed and things can be already
done with the existing infrastructure.
Regards
Rüdiger
Re: ping for http in mod_proxy
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 14, 2008, at 6:41 PM, Graham Leggett wrote:
> Akins, Brian wrote:
>
>> Yeah that was my thought. I guess you pass around the array of
>> servers.
>> Just remove (or mark as N/A) from the list and/or reorder it. At
>> the end,
>> core proxy picks whatever is in index 0 (possibly walking the list
>> in case
>> of connection error or something).
>
> Essentially that is what proxy does now (or did before the balancer
> stuff arrived) - it loops through everything that came back from the
> DNS round robin query, trying to connect in turn until it gets
> something other than bad gateway.
>
> Rather than just the simplistic round robin DNS query, it should
> query a hook (in which is embedded as the sensible default, a
> simplistic round robin DNS query).
>
Well, right now all it does is say "give me the best worker" and
leaves it up to the selected provider on "what that means", so it
queries a provider, not a hook. Of course, we could create an actual
hook which then queries providers, to allow people to extend
the selection even more...
Re: ping for http in mod_proxy
Posted by Graham Leggett <mi...@sharp.fm>.
Akins, Brian wrote:
> Yeah that was my thought. I guess you pass around the array of servers.
> Just remove (or mark as N/A) from the list and/or reorder it. At the end,
> core proxy picks whatever is in index 0 (possibly walking the list in case
> of connection error or something).
Essentially that is what proxy does now (or did before the balancer
stuff arrived) - it loops through everything that came back from the DNS
round robin query, trying to connect in turn until it gets something
other than bad gateway.
Rather than just the simplistic round robin DNS query, it should query a
hook (in which is embedded as the sensible default, a simplistic round
robin DNS query).
Regards,
Graham
--
Re: ping for http in mod_proxy
Posted by "Akins, Brian" <Br...@turner.com>.
On 2/14/08 9:54 AM, "Graham Leggett" <mi...@sharp.fm> wrote:
> In theory, you should be able to stack the providers, so that a balancer
> module could return the list of servers to try in the right order, and
> then another module could further reduce that list down to servers that
> are actually up.
Yeah that was my thought. I guess you pass around the array of servers.
Just remove (or mark as N/A) from the list and/or reorder it. At the end,
core proxy picks whatever is in index 0 (possibly walking the list in case
of connection error or something).
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: ping for http in mod_proxy
Posted by Graham Leggett <mi...@sharp.fm>.
Akins, Brian wrote:
> Someone could write a spread based module for origin status (or mysql, or
> memcache, or...) if the interface was well defined and "clean." The way
> balancer is so hooked into proxy makes it hard to write a replacement
> without hacking "core" proxy. In proxy it could be as simple a call as:
>
> apr_status_t ap_get_origin_server(request_rec *r, proxy_origin_t **o)
>
> And that figures out all the balancer and sticky stuff.
That was the idea.
In theory, you should be able to stack the providers, so that a balancer
module could return the list of servers to try in the right order, and
then another module could further reduce that list down to servers that
are actually up.
Smells like a filter actually.
Regards,
Graham
--
Re: ping for http in mod_proxy
Posted by "Akins, Brian" <Br...@turner.com>.
On 2/13/08 6:01 PM, "Graham Leggett" <mi...@sharp.fm> wrote:
> Is there anything stopping us going the multicasting route, say by
> adding a hook or hooks of some kind to proxy that keeps track of known
> server states?
Multicasting doesn't work well for us, for example, because servers are
spread across several different vlan's that explicitly don't allow multicast
between them.
If it were "hookable" or used providers, that would be fine. I think sticky
sessions should be hookable/provider as well.
Someone could write a spread based module for origin status (or mysql, or
memcache, or...) if the interface was well defined and "clean." The way
balancer is so hooked into proxy makes it hard to write a replacement
without hacking "core" proxy. In proxy it could be as simple a call as:
apr_status_t ap_get_origin_server(request_rec *r, proxy_origin_t **o)
And that figures out all the balancer and sticky stuff.
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: ping for http in mod_proxy
Posted by Graham Leggett <mi...@sharp.fm>.
Jim Jagielski wrote:
> Ideally, it would be nice if we had better insight on the
> actual health of the backends than a simple "do they respond
> to OPTIONS * and how long does it take", but that's pretty
> much all we can do unless go full-on multicasting of info
> ala mod_backhand... At least the balancer is setup to allow
> for load balancing based on that, if the submodule/provider
> actually existed :)
Is there anything stopping us going the multicasting route, say by
adding a hook or hooks of some kind to proxy that keeps track of known
server states?
A dedicated module could then have it's job as handling the multicasting
and letting the rest of the server know if a server is on or off the
air, in realtime.
The proxy can then just say "which of these servers should I connect
to?", and the hook will return the suggested ones in the given order.
As a dedicated module, people who don't need it can turn it off or not
build it.
Regards,
Graham
--
Re: ping for http in mod_proxy
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 13, 2008, at 12:41 PM, Plüm, Rüdiger, VF-Group wrote:
>
> Agreed, but I doubt that it is possible with a reasonable amout of
> health
> check frequency to find out before the first real request falls
> through,
> provided that your health checks are designed to only fail if the
> backend
> is down. If your health checks are smarter and notice that the
> backend will
> fail soon (e.g. because it reached 98% or 99% percent of its
> capacity) then
> this is a different story and can be very useful.
> Regarding the reactivation of a failed backend I fully agree: Active
> health
> checks would be an asset here instead of just retrying failed
> backends periodically
> with real requests.
>
Ideally, it would be nice if we had better insight on the
actual health of the backends than a simple "do they respond
to OPTIONS * and how long does it take", but that's pretty
much all we can do unless go full-on multicasting of info
ala mod_backhand... At least the balancer is setup to allow
for load balancing based on that, if the submodule/provider
actually existed :)
Re: ping for http in mod_proxy
Posted by "Akins, Brian" <Br...@turner.com>.
On 2/13/08 12:41 PM, "Plüm, Rüdiger, VF-Group" <ru...@vodafone.com>
wrote:
> If your health checks are smarter and notice that the backend will
> fail soon (e.g. because it reached 98% or 99% percent of its capacity) then
> this is a different story and can be very useful.
Correct. Perhaps a weighted round-robin that is based on response time
would be fairly easy to code... (Says the guy with no time.)
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: ping for http in mod_proxy
Posted by Plüm,
Rüdiger,
VF-Group <ru...@vodafone.com>.
> -----Ursprüngliche Nachricht-----
> Von: Akins, Brian
> Gesendet: Mittwoch, 13. Februar 2008 18:34
> An: dev@httpd.apache.org
> Betreff: Re: ping for http in mod_proxy
>
> On 2/13/08 12:27 PM, "Plüm, Rüdiger, VF-Group"
> <ru...@vodafone.com>
> wrote:
>
> > This does not help with race conditions on HTTP keepalive
> connections.
> > Nevertheless active healthchecking could be useful. But on
> a busy site
> > I guess a real request will notice before the healthcheck
> that one backend
> > is down or the frequence of health checks needs to be insane.
>
>
> On a busy site you want to know if a server is down before
> you send a few
> thousand requests to it and want to know that it's up as soon as it's
> available so you can send a few thousand requests to it.
Agreed, but I doubt that it is possible with a reasonable amout of health
check frequency to find out before the first real request falls through,
provided that your health checks are designed to only fail if the backend
is down. If your health checks are smarter and notice that the backend will
fail soon (e.g. because it reached 98% or 99% percent of its capacity) then
this is a different story and can be very useful.
Regarding the reactivation of a failed backend I fully agree: Active health
checks would be an asset here instead of just retrying failed backends periodically
with real requests.
Regards
Rüdiger
Re: ping for http in mod_proxy
Posted by "Akins, Brian" <Br...@turner.com>.
On 2/14/08 6:44 AM, "Plüm, Rüdiger, VF-Group" <ru...@vodafone.com>
wrote:
> 1. We currently have no mechanism in place that "simulates" these kind of
> failures we experience ourselves with the backend for the client. Returning
> a 500 or 503 does not cause the client to repeat the request. IMHO we would
> need to cut off the network connection without sending anything to trigger
> this behaviour in a well designed client.
Hrm.. Seems like the HTTP client should "just handle" this case.
> 2. Clients are only allowed to resend the request automatically, if the
> request
> was idempotent. Clients are not allowed to do so with non-idempotent
> requests
> like POST's without user intervention. So by probing the keepalive
> connection
> before sending the request we want to reduce these cases.
>From real world experience, I can say that we have rarely every had an issue
with POSTS. The active health-checking we do is based on how our load
balancers do it. They (the load balancers) can occasionally send requests
to a down server for a few seconds if it goes down in between healthchecks.
Sound bad in theory, but in reality, it has never been a real issue.
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: ping for http in mod_proxy
Posted by Plüm,
Rüdiger,
VF-Group <ru...@vodafone.com>.
> -----Ursprüngliche Nachricht-----
> Von: Akins, Brian [mailto:Brian.Akins@turner.com]
> Gesendet: Donnerstag, 14. Februar 2008 12:37
> An: dev@httpd.apache.org
> Betreff: Re: ping for http in mod_proxy
>
> On 2/13/08 1:09 PM, "Jim Jagielski" <ji...@jaguNET.com> wrote:
>
> > And again, we're basically doubling traffic and adding
> > overheard (more overhead than AJP's cping/cpong) at which
> > point I go back into wondering whether this sort of
> > implementation makes sense at all...
>
>
> So is the main issue we are trying to solve is that we have a
> keepalive
> connection to an origin but it does away (closes) before we
> make or next
> request and this causes an "error." (Sorry if I'm behind on
> the thread.)
Basicly yes.
> Shouldn't the HTTP client handle all of this? I know that
> libcurl handles
> this situations very well - it just tries to reconnect. How does serf
> handle it?
There are two problems here:
1. We currently have no mechanism in place that "simulates" these kind of
failures we experience ourselves with the backend for the client. Returning
a 500 or 503 does not cause the client to repeat the request. IMHO we would
need to cut off the network connection without sending anything to trigger
this behaviour in a well designed client.
2. Clients are only allowed to resend the request automatically, if the request
was idempotent. Clients are not allowed to do so with non-idempotent requests
like POST's without user intervention. So by probing the keepalive connection
before sending the request we want to reduce these cases.
Regards
Rüdiger
Re: ping for http in mod_proxy
Posted by "Akins, Brian" <Br...@turner.com>.
On 2/13/08 1:09 PM, "Jim Jagielski" <ji...@jaguNET.com> wrote:
> And again, we're basically doubling traffic and adding
> overheard (more overhead than AJP's cping/cpong) at which
> point I go back into wondering whether this sort of
> implementation makes sense at all...
So is the main issue we are trying to solve is that we have a keepalive
connection to an origin but it does away (closes) before we make or next
request and this causes an "error." (Sorry if I'm behind on the thread.)
Shouldn't the HTTP client handle all of this? I know that libcurl handles
this situations very well - it just tries to reconnect. How does serf
handle it?
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: ping for http in mod_proxy
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 13, 2008, at 12:59 PM, Plüm, Rüdiger, VF-Group wrote:
>>
>> We will never be able to completely avoid race conditions...
>> whether keepalives are in place or not.
>
> But at least the one that comes from the keepalive timer expiring on
> the backend at the same time I sent the request to it. If the
> backend answered to OPTIONS * on a keepalive connection it is
> reasonable to assume that the keepalive timeout will not expire
> until the real request hit the backend.
> Regarding other races I agree, but I think the above one is the
> one that is hit most frequently.
>
Ahh... but that goes back to one of my original questions:
do we honor force-proxy-request-1.0 and/or proxy-nokeepalive
for example or not... I can see reasons for both :)
And again, we're basically doubling traffic and adding
overheard (more overhead than AJP's cping/cpong) at which
point I go back into wondering whether this sort of
implementation makes sense at all...
Re: ping for http in mod_proxy
Posted by Plüm,
Rüdiger,
VF-Group <ru...@vodafone.com>.
> -----Ursprüngliche Nachricht-----
> Von: Jim Jagielski
> Gesendet: Mittwoch, 13. Februar 2008 18:52
> An: dev@httpd.apache.org
> Betreff: Re: ping for http in mod_proxy
>
>
> On Feb 13, 2008, at 12:27 PM, Plüm, Rüdiger, VF-Group wrote:
>
> >
> >
> >> -----Ursprüngliche Nachricht-----
> >> Von: Akins, Brian
> >> Gesendet: Mittwoch, 13. Februar 2008 18:23
> >> An: dev@httpd.apache.org
> >> Betreff: Re: ping for http in mod_proxy
> >>
> >> On 2/13/08 11:07 AM, "Jim Jagielski" <ji...@jaguNET.com> wrote:
> >>
> >>> I've started looking at adding "ping" support for
> >>> mod_proxy_http to complement whats in mod_proxy_ajp...
> >>> The idea is to send a simple OPTIONS * to the backend
> >>> and hope for a reply.
> >>
> >>
> >> Would it be more useful to have active healthchecking to
> >> backend servers?
> >
> > This does not help with race conditions on HTTP keepalive
> connections.
>
> We will never be able to completely avoid race conditions...
> whether keepalives are in place or not.
But at least the one that comes from the keepalive timer expiring on
the backend at the same time I sent the request to it. If the
backend answered to OPTIONS * on a keepalive connection it is
reasonable to assume that the keepalive timeout will not expire
until the real request hit the backend.
Regarding other races I agree, but I think the above one is the
one that is hit most frequently.
Regards
Rüdiger
Re: ping for http in mod_proxy
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 13, 2008, at 12:27 PM, Plüm, Rüdiger, VF-Group wrote:
>
>
>> -----Ursprüngliche Nachricht-----
>> Von: Akins, Brian
>> Gesendet: Mittwoch, 13. Februar 2008 18:23
>> An: dev@httpd.apache.org
>> Betreff: Re: ping for http in mod_proxy
>>
>> On 2/13/08 11:07 AM, "Jim Jagielski" <ji...@jaguNET.com> wrote:
>>
>>> I've started looking at adding "ping" support for
>>> mod_proxy_http to complement whats in mod_proxy_ajp...
>>> The idea is to send a simple OPTIONS * to the backend
>>> and hope for a reply.
>>
>>
>> Would it be more useful to have active healthchecking to
>> backend servers?
>
> This does not help with race conditions on HTTP keepalive connections.
We will never be able to completely avoid race conditions...
whether keepalives are in place or not.
Re: ping for http in mod_proxy
Posted by "Akins, Brian" <Br...@turner.com>.
On 2/13/08 12:27 PM, "Plüm, Rüdiger, VF-Group" <ru...@vodafone.com>
wrote:
> This does not help with race conditions on HTTP keepalive connections.
> Nevertheless active healthchecking could be useful. But on a busy site
> I guess a real request will notice before the healthcheck that one backend
> is down or the frequence of health checks needs to be insane.
On a busy site you want to know if a server is down before you send a few
thousand requests to it and want to know that it's up as soon as it's
available so you can send a few thousand requests to it.
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: ping for http in mod_proxy
Posted by Plüm,
Rüdiger,
VF-Group <ru...@vodafone.com>.
> -----Ursprüngliche Nachricht-----
> Von: Akins, Brian
> Gesendet: Mittwoch, 13. Februar 2008 18:23
> An: dev@httpd.apache.org
> Betreff: Re: ping for http in mod_proxy
>
> On 2/13/08 11:07 AM, "Jim Jagielski" <ji...@jaguNET.com> wrote:
>
> > I've started looking at adding "ping" support for
> > mod_proxy_http to complement whats in mod_proxy_ajp...
> > The idea is to send a simple OPTIONS * to the backend
> > and hope for a reply.
>
>
> Would it be more useful to have active healthchecking to
> backend servers?
This does not help with race conditions on HTTP keepalive connections.
Nevertheless active healthchecking could be useful. But on a busy site
I guess a real request will notice before the healthcheck that one backend
is down or the frequence of health checks needs to be insane.
Regards
Rüdiger
Re: ping for http in mod_proxy
Posted by "Akins, Brian" <Br...@turner.com>.
On 2/13/08 2:10 PM, "Jim Jagielski" <ji...@jaguNET.com> wrote:
> The latter is relatively easy to do with the current
> impl... Maybe I'll drop the ping idea and work on this ;)
+1 ;)
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: ping for http in mod_proxy
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 13, 2008, at 1:49 PM, Mads Toftum wrote:
> On Wed, Feb 13, 2008 at 12:22:43PM -0500, Akins, Brian wrote:
>> Would it be more useful to have active healthchecking to backend
>> servers?
>> Ie, periodically hit a url on each origin and mark them up/down
>> based on
>> response. Only send traffic to up servers. I think mod_backhand
>> does
>> something similar but much more complex (?).
>>
> Yes. Being able to check a URL seems more useful than plain ping.
> Even more useful would be recording the time a backend takes to
> answer a
> request and using that to distribute requests.
>
The latter is relatively easy to do with the current
impl... Maybe I'll drop the ping idea and work on this ;)
Re: ping for http in mod_proxy
Posted by Mads Toftum <ma...@toftum.dk>.
On Wed, Feb 13, 2008 at 12:22:43PM -0500, Akins, Brian wrote:
> Would it be more useful to have active healthchecking to backend servers?
> Ie, periodically hit a url on each origin and mark them up/down based on
> response. Only send traffic to up servers. I think mod_backhand does
> something similar but much more complex (?).
>
Yes. Being able to check a URL seems more useful than plain ping.
Even more useful would be recording the time a backend takes to answer a
request and using that to distribute requests.
vh
Mads Toftum
--
http://soulfood.dk
Re: ping for http in mod_proxy
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 14, 2008, at 6:33 AM, Akins, Brian wrote:
> On 2/13/08 12:50 PM, "Jim Jagielski" <ji...@jaguNET.com> wrote:
>> That was the other option as well... some sort of hearbeat
>> loop which updates worker status. Still, we get into the issue
>> with how much of "how proxy connects to and communicates with
>> the backend" to honor or work around.
>
> An external process (using serf maybe) would be easy. Just have
> some of the
> worker stats in shared memory. The healthchecker writes status to
> it, and
> httpd reads from it.
>
Yep... the process would fork off at Apache start time
and update it's own current health as well as update its
own knowledge of the health of the external backends...
Not convinced that multicasting is right for this due to
the types of architectures web servers usually find
themselves in :)
Re: ping for http in mod_proxy
Posted by "Akins, Brian" <Br...@turner.com>.
On 2/13/08 12:50 PM, "Jim Jagielski" <ji...@jaguNET.com> wrote:
> That was the other option as well... some sort of hearbeat
> loop which updates worker status. Still, we get into the issue
> with how much of "how proxy connects to and communicates with
> the backend" to honor or work around.
An external process (using serf maybe) would be easy. Just have some of the
worker stats in shared memory. The healthchecker writes status to it, and
httpd reads from it.
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: ping for http in mod_proxy
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 13, 2008, at 12:22 PM, Akins, Brian wrote:
> On 2/13/08 11:07 AM, "Jim Jagielski" <ji...@jaguNET.com> wrote:
>
>> I've started looking at adding "ping" support for
>> mod_proxy_http to complement whats in mod_proxy_ajp...
>> The idea is to send a simple OPTIONS * to the backend
>> and hope for a reply.
>
>
> Would it be more useful to have active healthchecking to backend
> servers?
> Ie, periodically hit a url on each origin and mark them up/down
> based on
> response. Only send traffic to up servers. I think mod_backhand does
> something similar but much more complex (?).
>
> I had started looking to add this to balancer, but I have no time.
>
That was the other option as well... some sort of hearbeat
loop which updates worker status. Still, we get into the issue
with how much of "how proxy connects to and communicates with
the backend" to honor or work around.
Re: ping for http in mod_proxy
Posted by "Akins, Brian" <Br...@turner.com>.
On 2/13/08 11:07 AM, "Jim Jagielski" <ji...@jaguNET.com> wrote:
> I've started looking at adding "ping" support for
> mod_proxy_http to complement whats in mod_proxy_ajp...
> The idea is to send a simple OPTIONS * to the backend
> and hope for a reply.
Would it be more useful to have active healthchecking to backend servers?
Ie, periodically hit a url on each origin and mark them up/down based on
response. Only send traffic to up servers. I think mod_backhand does
something similar but much more complex (?).
I had started looking to add this to balancer, but I have no time.
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies