You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Mark Jacquet <ma...@yahoo.com.INVALID> on 2015/06/02 01:58:13 UTC
[users@httpd] AH00485: scoreboard is full, not at MaxRequestWorkers
Sorry for the bandwidth, this is a repeat send from January.We STILL have this issue.
Perhaps a new look might jolt your memory and help me resolve this issue!!!
Problem: Apache server will stay up for random amount of time, usually days, but eventually enters a hung state. When hung the CPU load gradually spikes on the machine and new web server requests are unresponsive. Only a full server stop/kill and restart will resolve the issue, at which time the host load average drops quickly to normal.
Error logs typically contain many many lines of:
Wed Jan 28 16:06:58.667188 2015] [mpm_event:error] [pid 25336:tid 1] AH00485: scoreboard is full, not at MaxRequestWorkers
I have done a lot of web research on this top and have found many cases where others have had the same/similar issue but no real solutions. Seem very close to this bug report: https://issues.apache.org/bugzilla/show_bug.cgi?id=53555
Environment:
LDOM (VM) SunOS myhostname 5.10 Generic_118833-36 sun4v sparc SUNW,Sun-Fire-T200
8G RAM
http Conf:
StartServers 8
MinSpareServers Not set
MaxSpareServers Not set
ServerLimit 256
MaxRequestWorkers 200
MaxConnectionsPerChild 200
KeepAlive On
TimeOut 3000
MaxKeepAliveRequests 50
KeepAliveTimeout 2RequestReadTimeout header=10-20,MinRate=500 body=20,MinRate=500
We use LDAP to authenitcate most of our web tools, and this is configured like:
#
# Enable the LDAP connection pool and shared
# memory cache. Enable the LDAP cache status
# handler. Requires that mod_ldap and mod_authnz_ldap be loaded.
#
LDAPSharedCacheSize 500000
LDAPCacheEntries 1024
LDAPCacheTTL 600
LDAPOpCacheEntries 1024
LDAPOpCacheTTL 600
Current non-hung Score Board:
Apache Server Status for myserver.domain (via 5.5.5.5)
- Server Version: Apache/2.4.12 (Unix)
- Server MPM: event
- Server Built: Feb 20 2015 13:47:55
- Current Time: Monday, 01-Jun-2015 16:52:14 PDT
- Restart Time: Monday, 01-Jun-2015 00:35:23 PDT
- Parent Server Config. Generation: 1
- Parent Server MPM Generation: 0
- Server uptime: 16 hours 16 minutes 51 seconds
- Server load: 1.06 0.84 0.60
- Total accesses: 8263 - Total Traffic: 414.7 MB
- CPU Usage: u20.9 s32 cu660.69 cs121.41 - 1.42% CPU load
- .141 requests/sec - 7.2 kB/second - 51.4 kB/request
- 2 requests currently being processed, 98 idle workers
| PID | Connections | Threads | Async connections |
| total | accepting | busy | idle | writing | keep-alive | closing |
| 18793 | 0 | yes | 0 | 25 | 0 | 0 | 0 |
| 9525 | 3 | yes | 1 | 24 | 0 | 0 | 3 |
| 13484 | 0 | yes | 0 | 25 | 0 | 0 | 0 |
| 19506 | 3 | yes | 1 | 24 | 0 | 0 | 2 |
| Sum | 6 | | 2 | 98 | 0 | 0 | 5 |
........................._______________________________________
________W__________________________________W_________________...
................................................................
........
Any thoughts/comments on http conf tuning, OS patches, apache bug fixes appreciated.
This is a production server, so you can imagine, having it go down at random times (usually when I am asleep) is not fun!
Thanks.
MJ
RE: [users@httpd] AH00485: scoreboard is full, not at
MaxRequestWorkers
Posted by Scot Russell <sc...@hotmail.com>.
I think the directives are similar for event and worker and any config changes would not be too bad, but the worker documentation would give better info. To switch I think you would load the worker module instead of event or recompile with worker depending on how you are doing it.
Date: Tue, 2 Jun 2015 00:36:41 +0000
From: mark_jacquet@yahoo.com.INVALID
To: users@httpd.apache.org
Subject: Re: [users@httpd] AH00485: scoreboard is full, not at MaxRequestWorkers
I read that as well but the following statement he made did not make me feel like it totally resolved the issue.
"However, what I did notice is that the same server periodically had all of its workers taken up with requests, so that may be relevant to the problem as well."
If I did switch to "Worker" I wonder how much config file change would be needed?
Any thoughts on that?
MJ
On Monday, June 1, 2015 5:33 PM, Scot Russell <sc...@hotmail.com> wrote:
I don't know much on this, but for the bug link you included, one comment mentioned it being a problem with event mpm, that went away when using worker mpm. Is it possible to try the worker mpm?Date: Mon, 1 Jun 2015 23:58:13 +0000From: mark_jacquet@yahoo.com.INVALIDTo: users@httpd.apache.orgSubject: [users@httpd] AH00485: scoreboard is full, not at MaxRequestWorkersSorry for the bandwidth, this is a repeat send from January.We STILL have this issue. Perhaps a new look might jolt your memory and help me resolve this issue!!!Problem: Apache
server will stay up for random amount of time, usually days, but
eventually enters a hung state. When hung the CPU load gradually spikes
on the machine and new web server requests are unresponsive. Only a full server stop/kill and restart will resolve the issue, at which time the host load average drops quickly to normal.Error logs typically contain many many lines of: Wed Jan 28 16:06:58.667188 2015] [mpm_event:error] [pid 25336:tid 1] AH00485: scoreboard is full, not at MaxRequestWorkersI
have done a lot of web research on this top and have found many cases
where others have had the same/similar issue but no real solutions.
Seem very close to this bug report: https://issues.apache.org/bugzilla/show_bug.cgi?id=53555Environment:LDOM (VM) SunOS myhostname 5.10 Generic_118833-36 sun4v sparc SUNW,Sun-Fire-T200 8G RAMhttp Conf:StartServers 8MinSpareServers Not setMaxSpareServers Not setServerLimit 256MaxRequestWorkers 200MaxConnectionsPerChild 200KeepAlive OnTimeOut 3000MaxKeepAliveRequests 50KeepAliveTimeout 2RequestReadTimeout header=10-20,MinRate=500 body=20,MinRate=500We use LDAP to authenitcate most of our web tools, and this is configured like:## Enable the LDAP connection pool and shared# memory cache. Enable the LDAP cache status# handler. Requires that mod_ldap and mod_authnz_ldap be loaded.#LDAPSharedCacheSize 500000LDAPCacheEntries 1024LDAPCacheTTL 600LDAPOpCacheEntries 1024LDAPOpCacheTTL 600Current non-hung Score Board:Apache Server Status for myserver.domain (via 5.5.5.5)
Server Version: Apache/2.4.12 (Unix)Server MPM: eventServer Built: Feb 20 2015 13:47:55
Current Time: Monday, 01-Jun-2015 16:52:14 PDTRestart Time: Monday, 01-Jun-2015 00:35:23 PDTParent Server Config. Generation: 1Parent Server MPM Generation: 0Server uptime: 16 hours 16 minutes 51 secondsServer load: 1.06 0.84 0.60Total accesses: 8263 - Total Traffic: 414.7 MBCPU Usage: u20.9 s32 cu660.69 cs121.41 - 1.42% CPU load.141 requests/sec - 7.2 kB/second - 51.4 kB/request2 requests currently being processed, 98 idle workers
PIDConnectionsThreadsAsync connectionstotalacceptingbusyidlewritingkeep-aliveclosing187930yes02500095253yes124003134840yes025000195063yes124002Sum6 298005
........................._______________________________________
________W__________________________________W_________________...
................................................................
........Any thoughts/comments on http conf tuning, OS patches, apache bug fixes appreciated.This is a production server, so you can imagine, having it go down at random times (usually when I am asleep) is not fun!Thanks.MJ
Re: [users@httpd] AH00485: scoreboard is full, not at
MaxRequestWorkers
Posted by Mark Jacquet <ma...@yahoo.com.INVALID>.
I read that as well but the following statement he made did not make me feel like it totally resolved the issue.
"However, what I did notice is that the same server periodically had all of its workers taken up with requests, so that may be relevant to the problem as well."
If I did switch to "Worker" I wonder how much config file change would be needed?
Any thoughts on that?
MJ
On Monday, June 1, 2015 5:33 PM, Scot Russell <sc...@hotmail.com> wrote:
#yiv1467884072 #yiv1467884072 --.yiv1467884072hmmessage P{margin:0px;padding:0px;}#yiv1467884072 body.yiv1467884072hmmessage{font-size:12pt;font-family:Calibri;}#yiv1467884072 I don't know much on this, but for the bug link you included, one comment mentioned it being a problem with event mpm, that went away when using worker mpm. Is it possible to try the worker mpm?
Date: Mon, 1 Jun 2015 23:58:13 +0000
From: mark_jacquet@yahoo.com.INVALID
To: users@httpd.apache.org
Subject: [users@httpd] AH00485: scoreboard is full, not at MaxRequestWorkers
Sorry for the bandwidth, this is a repeat send from January.We STILL have this issue.
Perhaps a new look might jolt your memory and help me resolve this issue!!!
Problem: Apache server will stay up for random amount of time, usually days, but eventually enters a hung state. When hung the CPU load gradually spikes on the machine and new web server requests are unresponsive. Only a full server stop/kill and restart will resolve the issue, at which time the host load average drops quickly to normal.
Error logs typically contain many many lines of:
Wed Jan 28 16:06:58.667188 2015] [mpm_event:error] [pid 25336:tid 1] AH00485: scoreboard is full, not at MaxRequestWorkers
I have done a lot of web research on this top and have found many cases where others have had the same/similar issue but no real solutions. Seem very close to this bug report: https://issues.apache.org/bugzilla/show_bug.cgi?id=53555
Environment:
LDOM (VM) SunOS myhostname 5.10 Generic_118833-36 sun4v sparc SUNW,Sun-Fire-T200
8G RAM
http Conf:
StartServers 8
MinSpareServers Not set
MaxSpareServers Not set
ServerLimit 256
MaxRequestWorkers 200
MaxConnectionsPerChild 200
KeepAlive On
TimeOut 3000
MaxKeepAliveRequests 50
KeepAliveTimeout 2RequestReadTimeout header=10-20,MinRate=500 body=20,MinRate=500
We use LDAP to authenitcate most of our web tools, and this is configured like:
#
# Enable the LDAP connection pool and shared
# memory cache. Enable the LDAP cache status
# handler. Requires that mod_ldap and mod_authnz_ldap be loaded.
#
LDAPSharedCacheSize 500000
LDAPCacheEntries 1024
LDAPCacheTTL 600
LDAPOpCacheEntries 1024
LDAPOpCacheTTL 600
Current non-hung Score Board:
Apache Server Status for myserver.domain (via 5.5.5.5)
- Server Version: Apache/2.4.12 (Unix)
- Server MPM: event
- Server Built: Feb 20 2015 13:47:55
- Current Time: Monday, 01-Jun-2015 16:52:14 PDT
- Restart Time: Monday, 01-Jun-2015 00:35:23 PDT
- Parent Server Config. Generation: 1
- Parent Server MPM Generation: 0
- Server uptime: 16 hours 16 minutes 51 seconds
- Server load: 1.06 0.84 0.60
- Total accesses: 8263 - Total Traffic: 414.7 MB
- CPU Usage: u20.9 s32 cu660.69 cs121.41 - 1.42% CPU load
- .141 requests/sec - 7.2 kB/second - 51.4 kB/request
- 2 requests currently being processed, 98 idle workers
| PID | Connections | Threads | Async connections |
| total | accepting | busy | idle | writing | keep-alive | closing |
| 18793 | 0 | yes | 0 | 25 | 0 | 0 | 0 |
| 9525 | 3 | yes | 1 | 24 | 0 | 0 | 3 |
| 13484 | 0 | yes | 0 | 25 | 0 | 0 | 0 |
| 19506 | 3 | yes | 1 | 24 | 0 | 0 | 2 |
| Sum | 6 | | 2 | 98 | 0 | 0 | 5 |
........................._______________________________________
________W__________________________________W_________________...
................................................................
........
Any thoughts/comments on http conf tuning, OS patches, apache bug fixes appreciated.
This is a production server, so you can imagine, having it go down at random times (usually when I am asleep) is not fun!
Thanks.
MJ
RE: [users@httpd] AH00485: scoreboard is full, not at
MaxRequestWorkers
Posted by Scot Russell <sc...@hotmail.com>.
I don't know much on this, but for the bug link you included, one comment mentioned it being a problem with event mpm, that went away when using worker mpm. Is it possible to try the worker mpm?
Date: Mon, 1 Jun 2015 23:58:13 +0000
From: mark_jacquet@yahoo.com.INVALID
To: users@httpd.apache.org
Subject: [users@httpd] AH00485: scoreboard is full, not at MaxRequestWorkers
Sorry for the bandwidth, this is a repeat send from January.We STILL have this issue.
Perhaps a new look might jolt your memory and help me resolve this issue!!!
Problem: Apache
server will stay up for random amount of time, usually days, but
eventually enters a hung state. When hung the CPU load gradually spikes
on the machine and new web server requests are unresponsive. Only a full server stop/kill and restart will resolve the issue, at which time the host load average drops quickly to normal.
Error logs typically contain many many lines of: Wed Jan 28 16:06:58.667188 2015] [mpm_event:error] [pid 25336:tid 1] AH00485: scoreboard is full, not at MaxRequestWorkers
I
have done a lot of web research on this top and have found many cases
where others have had the same/similar issue but no real solutions.
Seem very close to this bug report: https://issues.apache.org/bugzilla/show_bug.cgi?id=53555Environment:LDOM (VM) SunOS myhostname 5.10 Generic_118833-36 sun4v sparc SUNW,Sun-Fire-T200 8G RAMhttp Conf:StartServers 8MinSpareServers Not setMaxSpareServers Not setServerLimit 256MaxRequestWorkers 200MaxConnectionsPerChild 200KeepAlive OnTimeOut 3000MaxKeepAliveRequests 50KeepAliveTimeout 2RequestReadTimeout header=10-20,MinRate=500 body=20,MinRate=500
We use LDAP to authenitcate most of our web tools, and this is configured like:
## Enable the LDAP connection pool and shared# memory cache. Enable the LDAP cache status# handler. Requires that mod_ldap and mod_authnz_ldap be loaded.#LDAPSharedCacheSize 500000LDAPCacheEntries 1024LDAPCacheTTL 600LDAPOpCacheEntries 1024LDAPOpCacheTTL 600
Current non-hung Score Board:Apache Server Status for myserver.domain (via 5.5.5.5)
Server Version: Apache/2.4.12 (Unix)Server MPM: eventServer Built: Feb 20 2015 13:47:55
Current Time: Monday, 01-Jun-2015 16:52:14 PDTRestart Time: Monday, 01-Jun-2015 00:35:23 PDTParent Server Config. Generation: 1Parent Server MPM Generation: 0Server uptime: 16 hours 16 minutes 51 secondsServer load: 1.06 0.84 0.60Total accesses: 8263 - Total Traffic: 414.7 MBCPU Usage: u20.9 s32 cu660.69 cs121.41 - 1.42% CPU load.141 requests/sec - 7.2 kB/second - 51.4 kB/request2 requests currently being processed, 98 idle workers
PIDConnections
ThreadsAsync connections
totalacceptingbusyidlewritingkeep-aliveclosing
187930yes025000
95253yes124003
134840yes025000
195063yes124002
Sum6 298005
........................._______________________________________
________W__________________________________W_________________...
................................................................
........Any thoughts/comments on http conf tuning, OS patches, apache bug fixes appreciated.This is a production server, so you can imagine, having it go down at random times (usually when I am asleep) is not fun!Thanks.MJ