You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2018/01/17 13:57:54 UTC
[Bug 62009] New: scoreboard is full with worker MPM
https://bz.apache.org/bugzilla/show_bug.cgi?id=62009
Bug ID: 62009
Summary: scoreboard is full with worker MPM
Product: Apache httpd-2
Version: 2.4.29
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: mod_http2
Assignee: bugs@httpd.apache.org
Reporter: a.abfalterer@gmail.com
Target Milestone: ---
I am currently investigating a problem (Apache/2.4.29 with mod_http2 v1.10.12)
related to worker MPM that leads to excessive termination and starting of
processes, sometimes also up to the termination of all httpd processes, thus to
the outage of the entire proxy. An outage occurs in connection with the log
entry "scoreboard is full, not at MaxRequestWorkers".
It looks as if the behavior is caused by mod_http2.
# Reproduction
The behavior can be aggravated if few processes are started, I use following
settings
ServerLimit 4
StartServers 1
MaxRequestWorkers 12
MinSpareThreads 3
MaxSpareThreads 6
ThreadsPerChild 3
After a restart, a large number of simultaneous requests (I generate 100
simultaneous requests) must be made. This is done so often (mostly 2-4 times
with a pause in the middle of each bulk) until httpd has 4 child processes.
Then you stop so that httpd becomes idle. Let's name this state "idle-4-proc".
The misbehavior is now manifested by the fact that excess processes are not
killed. Actually, there should only be a maximum of 2 processes
(MaxSpareThreads 6).
If in this state another bulk of requests is sent off, httpd begins to stop and
start processes like crazy. Now, sending some other bulks sometimes (not
always) lead to the already mentioned "scoreboard is full" error.
# Analysis
I could find out following
## httpd without mod_http2
If httpd is built without mod_http2, the main process recognizes in idle-4-proc
state that too many processes are available and signals by means of
AP_MPM_PODX_GRACEFUL that a process must be terminated. One of the child
processes reads AP_MPM_PODX_GRACEFUL from the POD, sends the signal SIGHUP,
which is intercepted by the listener thread. The listener thread then sends
SIGTERM to its PID to terminate all worker threads. In this way, the surplus
processes are terminated cleanly.
## httpd with mod_http2
However, if httpd is build with mod_http2 the listener thread does not receive
SIGHUP and therefore cannot send SIGTERM to the worker threads. As a result,
the child process is not terminated, httpd continues to send
AP_MPM_PODX_GRACEFUL and all child processes end up in an inconsistent state
(marked as terminated, but not really terminated). When requests now arrive,
all listener threads (of semi-terminated child processes) recognize that they
have to terminate and so they finally send SIGTERM to its PID. This causes the
misbehavior of httpd, which now starts too many new child processes.
The problem that listener threads does not receive SIGHUP seems to be that the
signal is caught by an h2-worker. This can be confirmed by attaching to an
child-process before idle-4-proc state and then sending off the bulks: one can
see that it is an h2-worker thread (instead of the listener thread) that
"handles" SIGHUP in worker.c:dummy_signal_handler().
# Workaround
The problem can be workaround-ed by blocking SIGHUP for each h2-worker, for
example by adding
sigset_t sig_mask;
sigemptyset(&sig_mask);
sigaddset(&sig_mask, SIGHUP);
#if defined(SIGPROCMASK_SETS_THREAD_MASK)
sigprocmask(SIG_BLOCK, &sig_mask, NULL);
#else
pthread_sigmask(SIG_BLOCK, &sig_mask, NULL);
#endif
in h2_workers.c:slot_run(). Doing so, I can observe same behavior as like with
httpd without mod_http2.
I'd appreciate it if someone could confirm the misbehavior and my analysis.
Thanks, Armin
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62009] scoreboard is full with worker MPM
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62009
Yann Ylavic <yl...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Keywords| |FixedInTrunk
--- Comment #7 from Yann Ylavic <yl...@gmail.com> ---
Committed to trunk in r1821504, and proposed for backport to 2.4.x.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62009] scoreboard is full with worker MPM
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62009
--- Comment #6 from Yann Ylavic <yl...@gmail.com> ---
(In reply to Armin Abfalterer from comment #3)
> Yann, by applying the patch surplus processes are terminated immediately,
> great!
Nice, will commit it and propose a backport for 2.4.x.
> Not sure if the
> "scoreboard is full" error cannot still occur.
I think it can, we made improvements in the event MPM lately for a better reuse
of scoreboard entries on graceful stop/restart, but AFAIK not in the worker
MPM.
I think there is a workaround for this though, I'm mainly copying a comment I
made in another PR regarding this:
You probably want to "oversize" ServerLimit w.r.t. MaxRequestWorkers and
ThreadsPerChild, such that there is room in the scoreboard for gracefully
restarting processes (without interfering with the new generation ones).
This relaxes the usual/strict "MaxRequestWorkers = ServerLimit *
ThreadsPerChild" formula to take reloads into account. For example, if
MaxRequestWorkers should be 1000 (adapt to your needs), something like
ThreadLimit = ThreadsPerChild = 50 and ServerLimit = 50 (instead of 20) leaves
room for 2.5 simultaneous reloads, or one/two reload and some
MaxConnectionsPerChild/MaxSpareThreads kicking at the same time.
YMMV...
(In reply to Yann Ylavic from comment #1)
> Thanks Armin for the detailed report.
I think i didn't insist enough there, that's a really great report and analysis
that nailed an issue which possibly bites httpd since a while now for some
configurations, not only http2 (not negligible today) but also probably
mod_watchdog or any (third-party) that also make use of threads in children
processes.
We may have worked around this issue with some (heavier) MPM
fixes/optimizations, mod_http2 changes, and/or configuration advides
(MaxRequestsPerChild 0, large MaxSpareThreads...), but that didn't completely
close the door.
Good thing that it is now, thanks (again) a bunch Armin!
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62009] scoreboard is full with worker MPM
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62009
--- Comment #1 from Yann Ylavic <yl...@gmail.com> ---
Thanks Armin for the detailed report.
For MPM worker and event, all signals should be blocked by default for all
threads, then som specific threads unblock dedicated signals for their own
wakeup (like SIGHUP for the listener thread).
The general blocking is done by apr_setup_signal_thread() from child_main(),
that is at the very beginning of the child process when it's still single
threaded. This should be inherited by further threads, unless we should really
call sigprocmask() here instead of pthread_sigmask(), irrespective of
SIGPROCMASK_SETS_THREAD_MASK, or (more exactly) especially if
PTHREAD_SIGMASK_DOES_NOT_SET_PROC_MASK for a single threaded process.
That doesn't seem to have caused issues w/o mod_http2 so far, so probably not
the good track.
So it looks like mod_http2 is special w.r.t. threads, but I don't hink the
right fix is to play with the mask of each possible thread created there (or
outside the MPM), that'd be a nightmare for maintenance. I'd rather we find out
what's special on the mod_http2 side...
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62009] scoreboard is full with worker MPM
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62009
--- Comment #10 from adv ydv <ad...@gmail.com> ---
if you are like the online games then you are http://myspades.org go to the
website like the online more user are like the online games.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62009] scoreboard is full with worker MPM
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62009
--- Comment #5 from Eric Covener <co...@gmail.com> ---
(In reply to Armin Abfalterer from comment #4)
> (In reply to Armin Abfalterer from comment #3)
> > What I still can observe is that httpd under load forks too many children.
> > With my stated configuration ("ServerLimit 4") and under load (concurrency
> > of 256 clients) httpd sometimes forks up to 11 children.
>
> Sorry, there were 11 children with a different configuration. With the above
> mentioned it was a maximum of 5.
Children that are in the process of exiting aren't counted against the limits.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62009] scoreboard is full with worker MPM
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62009
--- Comment #9 from Armin Abfalterer <a....@gmail.com> ---
Yann, I say thank you very much, it was going incredibly fast! Thanks also for
the "oversizing ServerLimit" hint.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62009] scoreboard is full with worker MPM
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62009
--- Comment #2 from Yann Ylavic <yl...@gmail.com> ---
Created attachment 35683
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35683&action=edit
Block signals before child_init() hooks
Eric found the why on dev@, mod_http2 creates threads in its child_init() hook,
before the MPM blocks signals.
Armin, does this patch work for you?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62009] scoreboard is full with worker MPM
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62009
Yann Ylavic <yl...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |FIXED
--- Comment #8 from Yann Ylavic <yl...@gmail.com> ---
Backported to 2.4.x in r1821517, will be in upcoming 2.4.30.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62009] scoreboard is full with worker MPM
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62009
--- Comment #4 from Armin Abfalterer <a....@gmail.com> ---
(In reply to Armin Abfalterer from comment #3)
> What I still can observe is that httpd under load forks too many children.
> With my stated configuration ("ServerLimit 4") and under load (concurrency
> of 256 clients) httpd sometimes forks up to 11 children.
Sorry, there were 11 children with a different configuration. With the above
mentioned it was a maximum of 5.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 62009] scoreboard is full with worker MPM
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=62009
--- Comment #3 from Armin Abfalterer <a....@gmail.com> ---
Yann, by applying the patch surplus processes are terminated immediately,
great!
What I still can observe is that httpd under load forks too many children. With
my stated configuration ("ServerLimit 4") and under load (concurrency of 256
clients) httpd sometimes forks up to 11 children. However, after the load the
number of processes gets back to normal. Not sure if the "scoreboard is full"
error cannot still occur.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org