You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by "William A. Rowe Jr." <wr...@rowe-clan.net> on 2010/06/30 04:17:44 UTC
Missing proxy_balancer feature
Yet again, in class another student pointed out that the Enabled/Disabled
choice in mod_proxy_balancer totally ignores the concept of quiescing,
where we are taking a server offline, but continuing to serve those
requests targeted by session to that server. Once the number of sessions
settles on something quite low, the user then takes that server offline
entirely and the remaining users are subjected to 'expired session' results.
A boolean Enabled/Disabled flag doesn't address this need.
Does anyone feel like working on this feature, since those I had
previously approached didn't have all that much interest, or got
busy with other things?
Re: Missing proxy_balancer feature
Posted by Tom Evans <te...@googlemail.com>.
On Wed, Jun 30, 2010 at 1:05 PM, Mark Watts <m....@eris.qinetiq.com> wrote:
> A restart of httpd (graceful or otherwise) has no understanding that the
> running config can be any different from that on disk.
>
> Indeed, how would httpd differentiate between a restart to reconfigure a
> given balancer (which may itself have a different in-memory loadfactor)
> or some other part of the server?
I believe that a graceful restart attempts to do this - it certainly
applies BalancerMember state from before the restart to
BalancerMembers after the restart. The problem is that it doesn't
alway apply the state to the right member, eg adding additional
members can result in the state of member b being applied to b'.
Having said all that, it may not be trying to do that, and it is just
coincidence that state is incorrectly applied post graceful restart, I
have not looked at the code.
>
> One 'solution' may be to change the way you disable a balancer member
> such that you change it in httpd.conf and do a graceful restart,
> combined with the quiescence changes mentioned earlier so that balancer
> members are withdrawn nicely.
This then brings you into the problems with graceful restart when
combined with balancer members. It isn't fun to be woken up in the
middle of the night because a sysadmin has restarted the proxy, and
all your balancer members have been set to disabled, because of
incorrect state being applied to the member.
>
> I'd guess that the only true way to handle this problem is to somehow
> compare the balancer configuration on disk with that in memory, and
> restore each BalancerMember's loadfactor on restart if the
> configurations match.
If it is a graceful restart,
>
> I suspect, but have no proof, that other load balancer software has this
> same issue.
>
Yes, this is the issue. In an ideal world, we want a balancer that
allows these kinds of features:
Add/configure vhosts without interrupting current sessions
Disable/enable balancers dynamically
Have state persisted across restarts
Internal redirection: reproxying requests, controlled from the
backend, eg send header 'X-Location:
http://otherappserver/some_resource' - just a nice to have tbh.
Apache falls short of those criteria (but only just!), and is a
maintenance nightmare to do simple things like adding an additional
vhost, since we must do a full restart, disrupting user sessions.
Cheers
Tom
Re: Missing proxy_balancer feature
Posted by Mark Watts <m....@eris.qinetiq.com>.
> As a reasonably heavy user of mod_proxy - all our web sites are
> proxied through a pair of reverse proxies - would you be interested in
> the problems we have encountered using it?
>
> In particular, bug 45950 is symptomatic of our issues. We can't
> adjust/tune our reverse proxy servers and gracefully restart, because
> the proxy worker state is utterly mangled following a graceful
> restart.
> This means we must do a full restart to avoid mangling state, which in
> turn means any balancer that has had its configuration dynamically
> changed via balancer_manager will have its configuration reset to the
> config on disk.
>
> The usual result of this is that sysadmin A will restart the proxies,
> unaware that sysadmin B has configured site S to be reverse proxied to
> application server app2 rather than app1, as app1 is undergoing
> maintenance and is down => we just broke one of our sites.
>
> The other net result is that we are now loath to adjust/restart apache
> on the proxy servers, since we must do a hard restart, disrupting any
> sessions on that server. This is so onerous that we are now discussing
> moving away from apache as a reverse proxy, and instead looking at
> things like pound, varnish, perlbal.
>
> This is a shame, as mod_proxy has many plus points - particularly that
> it's apache underneath, with all the flexibility that allows, and the
> mindshare that apache has already.
>
> Cheers
>
> Tom
A restart of httpd (graceful or otherwise) has no understanding that the
running config can be any different from that on disk.
Indeed, how would httpd differentiate between a restart to reconfigure a
given balancer (which may itself have a different in-memory loadfactor)
or some other part of the server?
One 'solution' may be to change the way you disable a balancer member
such that you change it in httpd.conf and do a graceful restart,
combined with the quiescence changes mentioned earlier so that balancer
members are withdrawn nicely.
I'd guess that the only true way to handle this problem is to somehow
compare the balancer configuration on disk with that in memory, and
restore each BalancerMember's loadfactor on restart if the
configurations match.
I suspect, but have no proof, that other load balancer software has this
same issue.
Mark.
--
Mark Watts BSc RHCE MBCS
Senior Systems Engineer, Managed Services Manpower
www.QinetiQ.com
QinetiQ - Delivering customer-focused solutions
GPG Key: http://www.linux-corner.info/mwatts.gpg
Re: Missing proxy_balancer feature
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Jun 30, 2010, at 4:32 AM, Tom Evans wrote:
>
> As a reasonably heavy user of mod_proxy - all our web sites are
> proxied through a pair of reverse proxies - would you be interested in
> the problems we have encountered using it?
Of course! Any bugs and/or improvements would be welcomed.
FWIW: I have a number of improvements that I am working on
that will be donated back to the ASF and submitted for
approval (or simply committed, assuming trunk).
Re: Missing proxy_balancer feature
Posted by Tom Evans <te...@googlemail.com>.
On Wed, Jun 30, 2010 at 3:47 AM, Daniel Ruggeri <DR...@primary.net> wrote:
> On 6/29/2010 9:17 PM, William A. Rowe Jr. wrote:
>>
>> Yet again, in class another student pointed out that the Enabled/Disabled
>> choice in mod_proxy_balancer totally ignores the concept of quiescing,
>> where we are taking a server offline, but continuing to serve those
>> requests targeted by session to that server. Once the number of sessions
>> settles on something quite low, the user then takes that server offline
>> entirely and the remaining users are subjected to 'expired session'
>> results.
>>
>> A boolean Enabled/Disabled flag doesn't address this need.
>>
>> Does anyone feel like working on this feature, since those I had
>> previously approached didn't have all that much interest, or got
>> busy with other things?
>>
>>
>>
>
> William;
> I actually wrote a patch in response to a submitted bug (48841) in April
> that will set the load factor to zero for a member. With a zero lbfactor,
> only items coming through with a sticky session are sent to that back end. I
> was hoping for comments from other devs about the method used, though,
> because I had to resort to a const (aka magic number) because of compiler
> oddness. In one specific case I can recall with the SUN C compiler, when
> lbfactor was not set, a check for lbfactor being equal to zero would return
> true... gcc did not replicate the same bug (null/undef is NOT 0 after all).
> This was only used during the configuration stage and the initialization of
> the balancer.
>
> I hope this helps suit your needs and that I understood the question. If
> I'm on the right track, but there are qualms with the implementation, I'll
> happily adjust the work.
>
> Thanks
> --
> Daniel Ruggeri
>
>
As a reasonably heavy user of mod_proxy - all our web sites are
proxied through a pair of reverse proxies - would you be interested in
the problems we have encountered using it?
In particular, bug 45950 is symptomatic of our issues. We can't
adjust/tune our reverse proxy servers and gracefully restart, because
the proxy worker state is utterly mangled following a graceful
restart.
This means we must do a full restart to avoid mangling state, which in
turn means any balancer that has had its configuration dynamically
changed via balancer_manager will have its configuration reset to the
config on disk.
The usual result of this is that sysadmin A will restart the proxies,
unaware that sysadmin B has configured site S to be reverse proxied to
application server app2 rather than app1, as app1 is undergoing
maintenance and is down => we just broke one of our sites.
The other net result is that we are now loath to adjust/restart apache
on the proxy servers, since we must do a hard restart, disrupting any
sessions on that server. This is so onerous that we are now discussing
moving away from apache as a reverse proxy, and instead looking at
things like pound, varnish, perlbal.
This is a shame, as mod_proxy has many plus points - particularly that
it's apache underneath, with all the flexibility that allows, and the
mindshare that apache has already.
Cheers
Tom
Re: Missing proxy_balancer feature
Posted by Daniel Ruggeri <DR...@primary.net>.
On 6/29/2010 9:17 PM, William A. Rowe Jr. wrote:
> Yet again, in class another student pointed out that the Enabled/Disabled
> choice in mod_proxy_balancer totally ignores the concept of quiescing,
> where we are taking a server offline, but continuing to serve those
> requests targeted by session to that server. Once the number of sessions
> settles on something quite low, the user then takes that server offline
> entirely and the remaining users are subjected to 'expired session' results.
>
> A boolean Enabled/Disabled flag doesn't address this need.
>
> Does anyone feel like working on this feature, since those I had
> previously approached didn't have all that much interest, or got
> busy with other things?
>
>
>
William;
I actually wrote a patch in response to a submitted bug (48841) in
April that will set the load factor to zero for a member. With a zero
lbfactor, only items coming through with a sticky session are sent to
that back end. I was hoping for comments from other devs about the
method used, though, because I had to resort to a const (aka magic
number) because of compiler oddness. In one specific case I can recall
with the SUN C compiler, when lbfactor was not set, a check for lbfactor
being equal to zero would return true... gcc did not replicate the same
bug (null/undef is NOT 0 after all). This was only used during the
configuration stage and the initialization of the balancer.
I hope this helps suit your needs and that I understood the
question. If I'm on the right track, but there are qualms with the
implementation, I'll happily adjust the work.
Thanks
--
Daniel Ruggeri
Re: Missing proxy_balancer feature
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Jun 30, 2010, at 8:34 AM, jean-frederic clere wrote:
> On 06/30/2010 04:17 AM, William A. Rowe Jr. wrote:
>> Yet again, in class another student pointed out that the Enabled/Disabled
>> choice in mod_proxy_balancer totally ignores the concept of quiescing,
>> where we are taking a server offline, but continuing to serve those
>> requests targeted by session to that server. Once the number of sessions
>> settles on something quite low, the user then takes that server offline
>> entirely and the remaining users are subjected to 'expired session' results.
>>
>> A boolean Enabled/Disabled flag doesn't address this need.
>>
>> Does anyone feel like working on this feature, since those I had
>> previously approached didn't have all that much interest, or got
>> busy with other things?
>>
>>
>
> There are several things that requires improvements:
> 1 - The default load balancing logic doesn't work well when you restart
> a back-end node.
>
> 2 - Nothing from the shared information is persisted so a restart may
> have weird effect.
>
> 3 - Additionally the forward and reserve proxy logic are mixed togother
> so it makes hard to change anything.
>
Yep... all stuff that should be fixed in 2.4... or *will* be ;)
Re: Missing proxy_balancer feature
Posted by jean-frederic clere <jf...@gmail.com>.
On 06/30/2010 04:17 AM, William A. Rowe Jr. wrote:
> Yet again, in class another student pointed out that the Enabled/Disabled
> choice in mod_proxy_balancer totally ignores the concept of quiescing,
> where we are taking a server offline, but continuing to serve those
> requests targeted by session to that server. Once the number of sessions
> settles on something quite low, the user then takes that server offline
> entirely and the remaining users are subjected to 'expired session' results.
>
> A boolean Enabled/Disabled flag doesn't address this need.
>
> Does anyone feel like working on this feature, since those I had
> previously approached didn't have all that much interest, or got
> busy with other things?
>
>
There are several things that requires improvements:
1 - The default load balancing logic doesn't work well when you restart
a back-end node.
2 - Nothing from the shared information is persisted so a restart may
have weird effect.
3 - Additionally the forward and reserve proxy logic are mixed togother
so it makes hard to change anything.
With mod_cluster (http://www.jboss.org/mod_cluster) I have a balancer
which accepts dynamic configuration (and persist it if needed).
If there is enough interest I could extent code already in ASF (slotmem,
mod_serf and heartbeat as well as httpd-proxy-scoreboard) to reach a
similar features: basically a bunch of slotmems that will be filled
"dynamically" use to create the worker and the routing decisions (a kind
of dynamic ProxyPass table).
Cheers
Jean-Frederic
Re: Missing proxy_balancer feature
Posted by jean-frederic clere <jf...@gmail.com>.
On 07/04/2010 04:38 PM, Daniel Ruggeri wrote:
> On 7/1/2010 1:46 AM, jean-frederic clere wrote:
>> On 07/01/2010 03:14 AM, Daniel Ruggeri wrote:
>>
>>> On 6/30/2010 7:34 AM, jean-frederic clere wrote:
>>>
>>>> On 06/30/2010 04:17 AM, William A. Rowe Jr. wrote:
>>>>
>>>>
>>>>> Yet again, in class another student pointed out that the
>>>>> Enabled/Disabled
>>>>> choice in mod_proxy_balancer totally ignores the concept of quiescing,
>>>>> where we are taking a server offline, but continuing to serve those
>>>>> requests targeted by session to that server. Once the number of
>>>>> sessions
>>>>> settles on something quite low, the user then takes that server
>>>>> offline
>>>>> entirely and the remaining users are subjected to 'expired session'
>>>>> results.
>>>>>
>>>>> A boolean Enabled/Disabled flag doesn't address this need.
>>>>>
>>>>> Does anyone feel like working on this feature, since those I had
>>>>> previously approached didn't have all that much interest, or got
>>>>> busy with other things?
>>>>>
>>>>>
>>>>>
>>>>>
>>>> There are several things that requires improvements:
>>>> 1 - The default load balancing logic doesn't work well when you restart
>>>> a back-end node.
>>>>
>>>> ...
>>>>
>>>> Cheers
>>>>
>>>> Jean-Frederic
>>>>
>>>>
>>>>
>>> Jean-Frederic;
>>> Can you elaborate more on point 1? I have had issues where a back
>>> end
>>> application was not ready during a restart and throws a 503 to Apache.
>>> Since this is a perfectly valid response to the proxied request, Apache
>>> happily sends it along to the user. I wrote a bug (48939) and submitted
>>> a patch that creates a configuration directive to help mitigate this
>>> issue which is the most significant issue I have come across during a
>>> simple back end restart. The patch is currently proposed in STATUS, but
>>> has not received votes to go forward (or stay in STATUS).
>>>
>> That is something different the algorithm used doesn't work correctly
>> when you have long lasting sessions and the load for the past is not
>> forgotten... Basically to much load is going to the returning back-end.
>>
>> Your patch is just a work-around to the problem that httpd doesn't know
>> that the back-end shouldn't be used at that time.
>>
>> Cheers
>>
>> Jean-Frederic
>>
>>
> Jean-Frederic;
> That's a very good point and a qualm I have had with httpd for a
> while myself. As it is, the current balancer setup relies on live
> traffic to figure out if the backend is OK. Some hardware load balancers
> have the ability to "ping" backends with anything from simple TCP, HTTP
> HEAD or even SSL handshakes. Perhaps adding something like that to HTTPD
> would be the trick? The logistics of configuring such a thing might be a
> little complicated in httpd.conf... perhaps another argument that can be
> passed to ProxyPass?
> Configured like so?
> ProxyPass (or ProxySet or BalancerMember) /context
> http://someBackend:1234/context httpProbe="HEAD /context/helloWorld.do"
> probeInterval=15
>
>
> Of course, this is all conceptual - what do you think of the idea?
That is something like the worker.maintain directive of mod_jk but with
the request to do.
Something like:
maintain=time|method|[URI]
maintain=60|CPING|
maintain=60|HEAD|/context/helloWorld.do
In the case we make a request that requires the application to be able
to process it as a probe request.
> While
> this is a feature that would be *incredibly* useful, I don't think I
> have the required expertise with httpd's code to execute it.
Well that is not so easy: it requires a thread (a thread per worker) and
to update the status of the worker out of a real request which is
requires some care.
Cheers
Jean-Frederic
Re: Missing proxy_balancer feature
Posted by Daniel Ruggeri <DR...@primary.net>.
On 7/1/2010 1:46 AM, jean-frederic clere wrote:
> On 07/01/2010 03:14 AM, Daniel Ruggeri wrote:
>
>> On 6/30/2010 7:34 AM, jean-frederic clere wrote:
>>
>>> On 06/30/2010 04:17 AM, William A. Rowe Jr. wrote:
>>>
>>>
>>>> Yet again, in class another student pointed out that the
>>>> Enabled/Disabled
>>>> choice in mod_proxy_balancer totally ignores the concept of quiescing,
>>>> where we are taking a server offline, but continuing to serve those
>>>> requests targeted by session to that server. Once the number of
>>>> sessions
>>>> settles on something quite low, the user then takes that server offline
>>>> entirely and the remaining users are subjected to 'expired session'
>>>> results.
>>>>
>>>> A boolean Enabled/Disabled flag doesn't address this need.
>>>>
>>>> Does anyone feel like working on this feature, since those I had
>>>> previously approached didn't have all that much interest, or got
>>>> busy with other things?
>>>>
>>>>
>>>>
>>>>
>>> There are several things that requires improvements:
>>> 1 - The default load balancing logic doesn't work well when you restart
>>> a back-end node.
>>>
>>> ...
>>>
>>> Cheers
>>>
>>> Jean-Frederic
>>>
>>>
>>>
>> Jean-Frederic;
>> Can you elaborate more on point 1? I have had issues where a back end
>> application was not ready during a restart and throws a 503 to Apache.
>> Since this is a perfectly valid response to the proxied request, Apache
>> happily sends it along to the user. I wrote a bug (48939) and submitted
>> a patch that creates a configuration directive to help mitigate this
>> issue which is the most significant issue I have come across during a
>> simple back end restart. The patch is currently proposed in STATUS, but
>> has not received votes to go forward (or stay in STATUS).
>>
> That is something different the algorithm used doesn't work correctly
> when you have long lasting sessions and the load for the past is not
> forgotten... Basically to much load is going to the returning back-end.
>
> Your patch is just a work-around to the problem that httpd doesn't know
> that the back-end shouldn't be used at that time.
>
> Cheers
>
> Jean-Frederic
>
>
Jean-Frederic;
That's a very good point and a qualm I have had with httpd for a
while myself. As it is, the current balancer setup relies on live
traffic to figure out if the backend is OK. Some hardware load balancers
have the ability to "ping" backends with anything from simple TCP, HTTP
HEAD or even SSL handshakes. Perhaps adding something like that to HTTPD
would be the trick? The logistics of configuring such a thing might be a
little complicated in httpd.conf... perhaps another argument that can be
passed to ProxyPass?
Configured like so?
ProxyPass (or ProxySet or BalancerMember) /context
http://someBackend:1234/context httpProbe="HEAD /context/helloWorld.do"
probeInterval=15
Of course, this is all conceptual - what do you think of the idea? While
this is a feature that would be *incredibly* useful, I don't think I
have the required expertise with httpd's code to execute it.
--
Daniel Ruggeri
Re: Missing proxy_balancer feature
Posted by jean-frederic clere <jf...@gmail.com>.
On 07/01/2010 03:14 AM, Daniel Ruggeri wrote:
>
> On 6/30/2010 7:34 AM, jean-frederic clere wrote:
>> On 06/30/2010 04:17 AM, William A. Rowe Jr. wrote:
>>
>>> Yet again, in class another student pointed out that the
>>> Enabled/Disabled
>>> choice in mod_proxy_balancer totally ignores the concept of quiescing,
>>> where we are taking a server offline, but continuing to serve those
>>> requests targeted by session to that server. Once the number of
>>> sessions
>>> settles on something quite low, the user then takes that server offline
>>> entirely and the remaining users are subjected to 'expired session'
>>> results.
>>>
>>> A boolean Enabled/Disabled flag doesn't address this need.
>>>
>>> Does anyone feel like working on this feature, since those I had
>>> previously approached didn't have all that much interest, or got
>>> busy with other things?
>>>
>>>
>>>
>> There are several things that requires improvements:
>> 1 - The default load balancing logic doesn't work well when you restart
>> a back-end node.
>>
>> ...
>>
>> Cheers
>>
>> Jean-Frederic
>>
>>
> Jean-Frederic;
> Can you elaborate more on point 1? I have had issues where a back end
> application was not ready during a restart and throws a 503 to Apache.
> Since this is a perfectly valid response to the proxied request, Apache
> happily sends it along to the user. I wrote a bug (48939) and submitted
> a patch that creates a configuration directive to help mitigate this
> issue which is the most significant issue I have come across during a
> simple back end restart. The patch is currently proposed in STATUS, but
> has not received votes to go forward (or stay in STATUS).
That is something different the algorithm used doesn't work correctly
when you have long lasting sessions and the load for the past is not
forgotten... Basically to much load is going to the returning back-end.
Your patch is just a work-around to the problem that httpd doesn't know
that the back-end shouldn't be used at that time.
Cheers
Jean-Frederic
Re: Missing proxy_balancer feature
Posted by Daniel Ruggeri <DR...@primary.net>.
On 6/30/2010 7:34 AM, jean-frederic clere wrote:
> On 06/30/2010 04:17 AM, William A. Rowe Jr. wrote:
>
>> Yet again, in class another student pointed out that the Enabled/Disabled
>> choice in mod_proxy_balancer totally ignores the concept of quiescing,
>> where we are taking a server offline, but continuing to serve those
>> requests targeted by session to that server. Once the number of sessions
>> settles on something quite low, the user then takes that server offline
>> entirely and the remaining users are subjected to 'expired session' results.
>>
>> A boolean Enabled/Disabled flag doesn't address this need.
>>
>> Does anyone feel like working on this feature, since those I had
>> previously approached didn't have all that much interest, or got
>> busy with other things?
>>
>>
>>
> There are several things that requires improvements:
> 1 - The default load balancing logic doesn't work well when you restart
> a back-end node.
>
> ...
>
> Cheers
>
> Jean-Frederic
>
>
Jean-Frederic;
Can you elaborate more on point 1? I have had issues where a back
end application was not ready during a restart and throws a 503 to
Apache. Since this is a perfectly valid response to the proxied request,
Apache happily sends it along to the user. I wrote a bug (48939) and
submitted a patch that creates a configuration directive to help
mitigate this issue which is the most significant issue I have come
across during a simple back end restart. The patch is currently proposed
in STATUS, but has not received votes to go forward (or stay in STATUS).
--
Daniel Ruggeri
Re: Missing proxy_balancer feature
Posted by jean-frederic clere <jf...@gmail.com>.
On 06/30/2010 04:17 AM, William A. Rowe Jr. wrote:
> Yet again, in class another student pointed out that the Enabled/Disabled
> choice in mod_proxy_balancer totally ignores the concept of quiescing,
> where we are taking a server offline, but continuing to serve those
> requests targeted by session to that server. Once the number of sessions
> settles on something quite low, the user then takes that server offline
> entirely and the remaining users are subjected to 'expired session' results.
>
> A boolean Enabled/Disabled flag doesn't address this need.
>
> Does anyone feel like working on this feature, since those I had
> previously approached didn't have all that much interest, or got
> busy with other things?
>
>
There are several things that requires improvements:
1 - The default load balancing logic doesn't work well when you restart
a back-end node.
2 - Nothing from the shared information is persisted so a restart may
have weird effect.
3 - Additionally the forward and reserve proxy logic are mixed togother
so it makes hard to change anything.
With mod_cluster (http://www.jboss.org/mod_cluster) I have a balancer
which accepts dynamic configuration (and persist it if needed).
If there is enough interest I could extent code already in ASF (slotmem,
mod_serf and heartbeat as well as httpd-proxy-scoreboard) to reach a
similar features: basically a bunch of slotmems that will be filled
"dynamically" use to create the worker and the routing decisions (a kind
of dynamic ProxyPass table).
Cheers
Jean-Frederic