You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Eric Covener <co...@gmail.com> on 2016/04/12 00:12:43 UTC
Re: [Bug 53555] Scoreboard full error with event/ssl
On Mon, Apr 11, 2016 at 4:59 PM, <bu...@apache.org> wrote:
> ServerLimit >= 10 * MaxRequestWorkers / ThreadsPerChild
Hi Stefan -- I am curious -- prior to the recent patches, just having
the extra capacity in ServerLimit didn't help / didn't help as much?
Re: [Bug 53555] Scoreboard full error with event/ssl
Posted by Stefan Fritsch <sf...@sfritsch.de>.
On Tuesday, 14 June 2016 17:31:50 CEST Eric Covener wrote:
> On Wed, Apr 13, 2016 at 6:27 PM, Stefan Fritsch <sf...@sfritsch.de> wrote:
> > Maybe it would be better to remove the logic to re-use scoreboard
> > slots of processes which have already terminated some threads.
> > Instead, one could use the scoreboard area abore (MaxRequestWorkers /
> > ThreadsPerChild) more aggressively, and maybe even allocate some
> > additional slack space at startup.
>
> Been helping a user affected by this, they got here after tuning
> around some OOM issues
> and are stuck with long ProxyTimeout values (a good recipe for this problem)
>
> Without understanding this very well, using the slack space (exclusively)
> does seem like a good idea to me.
>
> >There could be these config knobs:
> > - the max number of fully active processes (ServerLimit)
>
> We could also also leave this as MaxRequestWorkers /ThreadsPerChild.
> The slack would be defined by the difference from ServerLimit. This
> is also handy because it means ServerLimit really is the max
> active+exiting.
I have implemented this now. The patch is attached to the PR:
https://bz.apache.org/bugzilla/attachment.cgi?id=34201&action=diff
In addition, no more processes are told to gracefully stop due to
MaxSpareThreads, if there are more running processes than MaxRequestWorkers /
ThreadsPerChild. This implies that some are already finishing gracefully but
have not finished yet.
This condition can be refined, but the other suggestions below have not been
implemented yet.
I don't think I will have more time to work on this in the next 3 weeks at
least. But if anyone could take a peek at the patch and comment and/or test
it, that would be nice.
Cheers,
Stefan
> > - the max number of gracefully finishing processes ("OldServerLimit"?)
>
> Maybe better to count threads here after your other changes?
>
> > - what to do if OldServerLimit is reached during a graceful restart:
> > * reduce the max number of fully active processes accordingly
> > * kill off old processes
>
> Maybe OldServerLimit could be the # of exiting processes (or threads)
> that are not counted against the limits, rather than something that is
> binary (either breached or not)
>
> GracefulShutdownTimeout could not be directly applied, but maybe we
> could have something similar applied only to get us back below
> OldServerLimit.
>
> > - what to do if OldServerLimit during normal operation:
> > * stop doing idle-cleanup of fully active processes [X]
> > * kill off old processes
> >
> > Hmm. Now that I think of it, I think we should really do [X] if there
> > are too many old processes. The only question is how to define too
> > many.
>
> I think OldServerLimit as more of a soft limit for countermeasures
> like this helps here too. The slack space itself is then the hard
> limit.
Re: [Bug 53555] Scoreboard full error with event/ssl
Posted by Yann Ylavic <yl...@gmail.com>.
On Sat, Apr 30, 2016 at 1:04 PM, Yann Ylavic <yl...@gmail.com> wrote:
>
> How about:
>
> Index: server/mpm/event/event.c
> ===================================================================
> --- server/mpm/event/event.c (revision 1741696)
> +++ server/mpm/event/event.c (working copy)
> @@ -2857,7 +2857,7 @@ static void perform_idle_server_maintenance(int ch
> (status == SERVER_GRACEFUL);
> any_dead_threads = any_dead_threads || (status == SERVER_DEAD);
> all_dead_threads = all_dead_threads &&
> - (status == SERVER_DEAD || status == SERVER_GRACEFUL);
> + (status == SERVER_DEAD);
>
> /* We consider a starting server as idle because we started it
> * at least a cycle ago, and if it still hasn't finished starting
> _
>
> AIUI, that would (re)use *in priority* scoreboard entries of processes
> *fully* idle.
> So possibly admins could configure ServerLimit = N *
> (MaxRequestWorkers / ThreadsPerChild), where N would be the number of
> graceful restarts expected to not mix worker entries of old/new
> generation processes.
Hmm, not that easy actually.
perform_idle_server_maintenance() is currently based on
ap_daemons_limit processes (as possibly are codes using
AP_MPMQ_MAX_DAEMONS), not on server_limit.
Using the spare "server_limit - ap_daemons_limit" room at runtime
would require changing ap_daemons_limit dynamically/accordingly...
Is setting "AP_MPMQ_MAX_DAEMONS = server_limit" an option with
mpm_{event,worker}, so that we could make dynamic ap_daemons_limit
internal to those MPMs, whereas AP_MPMQ_MAX_DAEMONS users would
continue to "work"?
Re: [Bug 53555] Scoreboard full error with event/ssl
Posted by Yann Ylavic <yl...@gmail.com>.
Hi Stefan,
On Thu, Apr 14, 2016 at 12:27 AM, Stefan Fritsch <sf...@sfritsch.de> wrote:
> On Monday 11 April 2016 18:12:43, Eric Covener wrote:
>> On Mon, Apr 11, 2016 at 4:59 PM, <bu...@apache.org> wrote:
>> > ServerLimit >= 10 * MaxRequestWorkers / ThreadsPerChild
>>
>> Hi Stefan -- I am curious -- prior to the recent patches, just
>> having the extra capacity in ServerLimit didn't help / didn't help
>> as much?
>
> No, without the patches attached to the PR, ServerLimit does not play
> any role for this issue. Only the scoreboard entries up to
> (MaxRequestWorkers / ThreadsPerChild) will ever be used.
>
> The state in 2.4 is that scoreboard entries of processes where some
> threads have already terminated would get re-used, but there is no
> code that terminate threads before the whole process terminates.
How about:
Index: server/mpm/event/event.c
===================================================================
--- server/mpm/event/event.c (revision 1741696)
+++ server/mpm/event/event.c (working copy)
@@ -2857,7 +2857,7 @@ static void perform_idle_server_maintenance(int ch
(status == SERVER_GRACEFUL);
any_dead_threads = any_dead_threads || (status == SERVER_DEAD);
all_dead_threads = all_dead_threads &&
- (status == SERVER_DEAD || status == SERVER_GRACEFUL);
+ (status == SERVER_DEAD);
/* We consider a starting server as idle because we started it
* at least a cycle ago, and if it still hasn't finished starting
_
AIUI, that would (re)use *in priority* scoreboard entries of processes
*fully* idle.
So possibly admins could configure ServerLimit = N *
(MaxRequestWorkers / ThreadsPerChild), where N would be the number of
graceful restarts expected to not mix worker entries of old/new
generation processes.
N could be determined based on the frequency of graceful restarts and
average connections' duration (bounded by MaxKeepAliveRequests,
*Timeout, ...).
Spare scoreboard entries (ServerLimit - MaxRequestWorkers /
ThreadsPerChild) are quite cheap with today's memory capacities...
The current mixing logic would still apply when the above is
exhausted, but admins could increase N.
Regards,
Yann.
Re: [Bug 53555] Scoreboard full error with event/ssl
Posted by Eric Covener <co...@gmail.com>.
On Wed, Apr 13, 2016 at 6:27 PM, Stefan Fritsch <sf...@sfritsch.de> wrote:
> Maybe it would be better to remove the logic to re-use scoreboard
> slots of processes which have already terminated some threads.
> Instead, one could use the scoreboard area abore (MaxRequestWorkers /
> ThreadsPerChild) more aggressively, and maybe even allocate some
> additional slack space at startup.
Been helping a user affected by this, they got here after tuning
around some OOM issues
and are stuck with long ProxyTimeout values (a good recipe for this problem)
Without understanding this very well, using the slack space (exclusively)
does seem like a good idea to me.
>There could be these config knobs:
> - the max number of fully active processes (ServerLimit)
We could also also leave this as MaxRequestWorkers /ThreadsPerChild.
The slack would be defined by the difference from ServerLimit. This
is also handy because it means ServerLimit really is the max
active+exiting.
> - the max number of gracefully finishing processes ("OldServerLimit"?)
Maybe better to count threads here after your other changes?
> - what to do if OldServerLimit is reached during a graceful restart:
> * reduce the max number of fully active processes accordingly
> * kill off old processes
Maybe OldServerLimit could be the # of exiting processes (or threads)
that are not counted against the limits, rather than something that is
binary (either breached or not)
GracefulShutdownTimeout could not be directly applied, but maybe we
could have something similar applied only to get us back below
OldServerLimit.
> - what to do if OldServerLimit during normal operation:
> * stop doing idle-cleanup of fully active processes [X]
> * kill off old processes
>
> Hmm. Now that I think of it, I think we should really do [X] if there
> are too many old processes. The only question is how to define too
> many.
I think OldServerLimit as more of a soft limit for countermeasures
like this helps here too. The slack space itself is then the hard
limit.
--
Eric Covener
covener@gmail.com
Re: [Bug 53555] Scoreboard full error with event/ssl
Posted by Stefan Fritsch <sf...@sfritsch.de>.
On Monday 11 April 2016 18:12:43, Eric Covener wrote:
> On Mon, Apr 11, 2016 at 4:59 PM, <bu...@apache.org> wrote:
> > ServerLimit >= 10 * MaxRequestWorkers / ThreadsPerChild
>
> Hi Stefan -- I am curious -- prior to the recent patches, just
> having the extra capacity in ServerLimit didn't help / didn't help
> as much?
No, without the patches attached to the PR, ServerLimit does not play
any role for this issue. Only the scoreboard entries up to
(MaxRequestWorkers / ThreadsPerChild) will ever be used.
The state in 2.4 is that scoreboard entries of processes where some
threads have already terminated would get re-used, but there is no
code that terminate threads before the whole process terminates.
In trunk, I have added logic to terminate threads as soon as possible
during graceful finishing of a process. But the reason for that is
that I want old processes to use as few resources as possible. I am
not sure that new processes re-using the scoreboard entries of those
threads is a good idea. In the worst case, you could get
ThreadsPerChild processes that each use one worker slot in the
scoreboard area of a single process.
Maybe it would be better to remove the logic to re-use scoreboard
slots of processes which have already terminated some threads.
Instead, one could use the scoreboard area abore (MaxRequestWorkers /
ThreadsPerChild) more aggressively, and maybe even allocate some
additional slack space at startup. There could be these config knobs:
- the max number of fully active processes (ServerLimit)
- the max number of gracefully finishing processes ("OldServerLimit"?)
- what to do if OldServerLimit is reached during a graceful restart:
* reduce the max number of fully active processes accordingly
* kill off old processes
- what to do if OldServerLimit during normal operation:
* stop doing idle-cleanup of fully active processes [X]
* kill off old processes
Hmm. Now that I think of it, I think we should really do [X] if there
are too many old processes. The only question is how to define too
many.
Any thoughts?