You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2016/10/25 21:46:14 UTC
[Bug 53555] Scoreboard full error with event/ssl
https://bz.apache.org/bugzilla/show_bug.cgi?id=53555
--- Comment #57 from Thomas Jarosch <th...@intra2net.com> ---
Hi Stefan,
thanks for trying to solve the "scoreboard full" issue :)
I've been hit by it badly today, the affected machine
is a forward proxy and stalls the traffic almost completely.
Some background info:
- event mpm on httpd 2.4.23
- forward proxy setup via mod_proxy
- 280 real users + other machines. ~370 clients
- server load is around 0.2, plenty of free RAM
- file descriptor limit is 1024
- logrotate sends a graceful restart every hour
If the problem occurs, httpd doesn't even respond
to the /server-status page reliably.
A small script logs the /server-status page every 30s to disk.
Specific case: logrotate sends a "graceful restart" at 13h.
/server-status output at 13:04:24h:
-------------------
Total accesses: 8801 - Total Traffic: 74.6 MB
75 requests currently being processed, 125 idle workers
+---------------------------------------------------------------------------+
| | Connections | Threads | Async connections |
| PID |-------------------+-------------+---------------------------------|
| | total | accepting | busy | idle | writing | keep-alive | closing ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 14906 | 7 | yes | 6 | 44 | 0 | 1 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 14959 | 9 | yes | 9 | 41 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 15014 | 3 | no | 0 | 0 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 15015 | 49 | yes | 50 | 0 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 15329 | 3 | no | 0 | 0 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 15893 | 15 | no | 0 | 0 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 17762 | 11 | yes | 10 | 40 | 0 | 1 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| Sum | 97 | | 75 | 125 | 0 | 2 | 0 ||
+---------------------------------------------------------------------------+
_________R_____R__________________R___R___R__R________R______R_R
R_____R__R_________________R__R____RGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGGGGGGGGRRRRRRRRRRRRRRRRRRRRRRRRRRRWRRRRRRRRRRRRRR
RRRRRRRRGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGR__________R__R_____
_______R_RR_________R_RR_R____
-------------------
/server-status output at 13:15:25h:
-------------------
Total accesses: 12929 - Total Traffic: 90.9 MB
87 requests currently being processed, 63 idle workers
+---------------------------------------------------------------------------+
| | Connections | Threads | Async connections |
| PID |-------------------+-------------+---------------------------------|
| | total | accepting | busy | idle | writing | keep-alive | closing ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 14906 | 18 | yes | 16 | 34 | 0 | 2 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 14959 | 27 | yes | 26 | 24 | 0 | 2 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 15014 | 2 | no | 0 | 0 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 15015 | 2 | no | 0 | 0 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 15329 | 2 | no | 0 | 0 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 18564 | 45 | yes | 45 | 5 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 17762 | 39 | no | 0 | 0 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 18078 | 44 | no | 0 | 0 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| Sum | 179 | | 87 | 63 | 0 | 4 | 0 ||
+---------------------------------------------------------------------------+
_____R__R___R_RR_RR_R_RR__R_____R_R___R_R_____R___W_RR__RR_RR__R
RR__R_RR____RRRRR_R_RR___R_RR_RR____GGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGRRRRRR
RRRRRRRRR_RRRRRRRRR_RRRR_RRRRRRRRRRR_R_RRRRRGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGG
-------------------
/server-status at 13:25:20h:
(httpd hardly responding anymore):
-------------------
Total accesses: 14630 - Total Traffic: 97.4 MB
50 requests currently being processed, 0 idle workers
+---------------------------------------------------------------------------+
| | Connections | Threads | Async connections |
| PID |-------------------+-------------+---------------------------------|
| | total | accepting | busy | idle | writing | keep-alive | closing ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 14906 | 36 | no | 0 | 0 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 14959 | 2 | yes | 0 | 0 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 15014 | 2 | no | 0 | 0 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 15015 | 2 | no | 0 | 0 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 15329 | 2 | no | 0 | 0 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 18564 | 50 | yes | 50 | 0 | 0 | 1 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 17762 | 3 | no | 0 | 0 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| 18078 | 1 | no | 0 | 0 | 0 | 0 | 0 ||
|-------+-------+-----------+------+------+---------+------------+---------||
| Sum | 98 | | 50 | 0 | 0 | 1 | 0 ||
+---------------------------------------------------------------------------+
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGWRRRRR
RRRRWRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGG
-------------------
I can provide more /server-status output if needed.
After around 30 mins, the external "mon" watchdog
kills httpd and restarts it. Traffic continues to flow.
httpd config:
-------------------
Timeout 300
KeepAliveTimeout 300
<IfModule mpm_event_module>
# Number of concurrent connections is: ServerLimit * ThreadsPerChild
# Result: 16 * 50 -> 800
#
StartServers 1
ServerLimit 16
ThreadLimit 50
ThreadsPerChild 50
MaxConnectionsPerChild 1000
</IfModule>
No other performance related settings.
-------------------
I've now increased ServerLimit to 32 and disabled
logrotate as a quick fix. It holds so far.
Occasionally I still see the "scoreboard full" message,
even though there are just ~160 active connections and some processes
are (still?) in the graceful shutdown state.
I'll put the patch from #55 on the productive machine tomorrow :o)
It already runs on my own proxy and the one from my department.
Anything else to watch out for?
I can provide gdb backtraces if you tell
me to look for something specific, too.
Triggering a graceful restart during peak traffic might be a good test...
Cheers,
Thomas
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org