You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Sam <le...@chello.at> on 2012/09/06 11:06:19 UTC
Re: [users@httpd] scoreboard is full, not at MaxRequestWorkers
Yes, I still see such messages even after upgrading to 2.4.3
> On Sun, Aug 19, 2012 at 8:25 AM, Sam<le...@chello.at> wrote:
>>> I upgraded to Apache 2.4 and use mod_event on a server
>>> having average high load.
>>>
>>> My issue is that I am getting pretty much often the error message
>>> within the error log
>>>
>>> AH00485: scoreboard is full, not at MaxRequestWorkers
>>>
>>> What causes this? The server almost stops to handle new requests
>>> when this occurs.
>>>
>>> MaxRequestWorkers 192
>>> ServerLimit 3
>>> StartServers 2
>>>
>>> MaxMemFree 2048
>>> MinSpareThreads 64
>>> MaxSpareThreads 64
>>> ThreadLimit 64
>>> ThreadsPerChild 64
>>> MaxConnectionsPerChild 10240
>>> AsyncRequestWorkerFactor 2
> Just in case you still have this issue, we've switched to 2.4.3 after
> seeing a few bug reports that could be related, and since then I
> haven't seen this again.
>
> Regards,
>
> Martin
Re: [users@httpd] scoreboard is full, not at MaxRequestWorkers
Posted by Sam <le...@chello.at>.
Changing KeepAlive to Off does not help, the issue occurs at this
point where obviously due to MaxConnectionsPerChild a Process
should become killed.
As observed, the problem is that (in the example stated) both
PIDs 11454 and 16555 do not become killed because of these
remaining connections, which is ok, since these could be
remaining downloads or something to a client.
But then, Apache does neither fork new processes to accept
new connections nor does it kill the both PIDs (which is as just
statet, ok) - the scoreboard is full then in that case and new
connections can not be accepted.
As I see a solution to this problem is that the main httpd process
should be 1. able to fork new processes 2. I see no reason why
the scoreboard is full then, 1+2 connections is 3, not 2x64=128,
in that given case 125 free slots are wasted since they are blocked
by the 2 remaining processes which show connections which fill up
the scoreboard, those are shown as "G"s in the scoreboard for both
PIDs - that should not be the case.
Since 2.4.2 (the first I've used) I did not understand why a
finishing/killing process shows all "G"s - however, that issue did
not occur in 2.2 when I used MPM event in experimental stage, but
in 2.2 I did not have that problem, the connections were counted as
usual. Probably because lots of the code was taken from MPM worker,
where the calculations were still correct, until fully adopted to 2.4.
By this theory, also MPM worker might experience this, however, due to
the different design MPM worker also may never get to that point to
show this behavior and error messages in the log.
So my conclusion is also a question: why are finishing processes showing
all "G"s in the scoreboard, and/or why are these finishing threads all
counted in the scoreboard (here: 2x64 instead of 1+2, causing 125 threads
to be wasted).
I would expect this issue to become much more important when most people
upgrade to 2.4, also I clearly would say this is a bug.
> On Thu, Sep 6, 2012 at 11:21 PM, Sam <lennsen@chello.at
> <ma...@chello.at>> wrote:
>
> Never, the server is not even under high load. There
> are several thousand vhosts, it is about shared webhosting.
>
> This is what server-status says:
>
> PID Connections Threads Async connections
> total accepting busy idle writing keep-alive closing
> 21388 257 yes 44 20 0 171 44
> 11454 2 no 0 0 0 0 0
> 16555 1 no 0 0 0 0 0
> Sum 260 44 20 0 171 44
>
>
> How about those 171 in keep-alive? What are your KeepAlive and
> KeepAliveTimeout settings? Maybe try switching KeepAlive off? Whit
> only 192 threads and KeepAlive On you are imposing DoS on your self in
> case of long timeout.
>
> That one PID 11454has "2" connections, 16555 has "1", all other
> values are 0, I guess it should become killed? - but it does not,
> it stays like this for a while (I don't know on what this "while"
> depends on).
>
> At this point all 3 rows are filled with stuff, such as:
>
> WWW_W_WWWWWW_WW_W_W_W_WW__WWW__WW__WWWWWW__WWW__W__W_RWWWWRWWWWW
> GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
> GGGGGGGGGGGGGGGLGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
>
> But as you see, due to several _ this means that there are
> not all 192 busy, in fact:
>
> 105 requests/sec - 2.5 MB/second - 24.8 kB/request
> 44 requests currently being processed, 20 idle workers
>
> WHY can no others be used? With the statement above it means that
> 128 (which is 192-64) workers are unused because they should become
> killed already, to spawn new ones, these are the ones in the 2. and
> 3. row with "G"s.
>
> --
>
> So the rest is occupied with "G". In Apache 2.2 I used MPM_event
> as well, this obviously never occured because the both PIDs above
> which are showing G only showed just dots in that case, now after
> one PID should be finished, all slots become occupied with G,
> which obviously causes a full scoreboard.
>
> I would consider this a bug, a serious one for my case, since it
> often occurs (~1-3 times per hour) that the server stops accepting
> new requests, just because of that full scoreboard, where the server
> is neither busy, nor do I see what configuration directive should
> be configured badly. Again:
>
> KeepAlive On
> ListenBacklog 4095
> Timeout 20
> KeepAliveTimeout 8
> MaxKeepAliveRequests 16192
> MaxRanges 200
> MaxRangeOverlaps 20
> MaxRangeReversals 20
> GracefulShutdownTimeout 6
> LimitInternalRecursion 10
> LimitRequestFieldSize 4094
> LimitRequestFields 40
> LimitRequestLine 4094
> LimitXMLRequestBody 786432
>
> MaxRequestWorkers 192
> ServerLimit 3
> StartServers 2
>
> MaxMemFree 2048
> MinSpareThreads 64
> MaxSpareThreads 64
> ThreadLimit 128
> ThreadsPerChild 64
> MaxConnectionsPerChild 10240
> AsyncRequestWorkerFactor 10
>
>
> I assumed AsyncRequestWorkerFactor to have something to do
> with it, I raised it from 2 (default) to 10, no change.
>
> By the way: after some time (in this case above both) PIDs
> become killed finally, releasing 2x64 slots in the scoreboard,
> the waring messages vanish and the server can respond again
> well to requests. In the interest of finding a workaround until
> this is fixed: what configuration directives control that timeout
> or whatever, which cause these PIDs to become killed?
>
>
>>
>> On Thu, Sep 6, 2012 at 7:06 PM, Sam <lennsen@chello.at
>> <ma...@chello.at>> wrote:
>>
>>
>> Yes, I still see such messages even after upgrading to 2.4.3
>>
>>> On Sun, Aug 19, 2012 at 8:25 AM, Sam<le...@chello.at> <ma...@chello.at> wrote:
>>>>> I upgraded to Apache 2.4 and use mod_event on a server
>>>>> having average high load.
>>>>>
>>>>> My issue is that I am getting pretty much often the error message
>>>>> within the error log
>>>>>
>>>>> AH00485: scoreboard is full, not at MaxRequestWorkers
>>>>>
>>>>> What causes this? The server almost stops to handle new requests
>>>>> when this occurs.
>>>>>
>>>>> MaxRequestWorkers 192
>>>>> ServerLimit 3
>>>>> StartServers 2
>>>>>
>>>>> MaxMemFree 2048
>>>>> MinSpareThreads 64
>>>>> MaxSpareThreads 64
>>>>> ThreadLimit 64
>>>>> ThreadsPerChild 64
>>>>> MaxConnectionsPerChild 10240
>>>>> AsyncRequestWorkerFactor 2
>>
>>
>> So you never have more than 192 simultaneous requests at any
>> given time ???
>>
>
>
Re: [users@httpd] scoreboard is full, not at MaxRequestWorkers
Posted by Tom Evans <te...@googlemail.com>.
On Fri, Sep 7, 2012 at 12:57 AM, Igor Cicimov <ic...@gmail.com> wrote:
> On Thu, Sep 6, 2012 at 11:21 PM, Sam <le...@chello.at> wrote:
>>
>> Never, the server is not even under high load. There
>> are several thousand vhosts, it is about shared webhosting.
>>
>> This is what server-status says:
>>
>> PID Connections Threads Async connections
>> total accepting busy idle writing keep-alive closing
>> 21388 257 yes 44 20 0 171 44
>> 11454 2 no 0 0 0 0 0
>> 16555 1 no 0 0 0 0 0
>> Sum 260 44 20 0 171 44
>>
>
> How about those 171 in keep-alive? What are your KeepAlive and
> KeepAliveTimeout settings? Maybe try switching KeepAlive off? Whit only 192
> threads and KeepAlive On you are imposing DoS on your self in case of long
> timeout.
>
OP is using the event MPM, which uses a single thread to handle all
keep alive connections and writing connections, so the number of keep
alive sockets is not really relevant.
Cheers
Tom
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] scoreboard is full, not at MaxRequestWorkers
Posted by Igor Cicimov <ic...@gmail.com>.
On Thu, Sep 6, 2012 at 11:21 PM, Sam <le...@chello.at> wrote:
> **
> Never, the server is not even under high load. There
> are several thousand vhosts, it is about shared webhosting.
>
> This is what server-status says:
>
> PID Connections Threads Async connections
> total accepting busy idle writing keep-alive closing
> 21388 257 yes 44 20 0 171 44
> 11454 2 no 0 0 0 0 0
> 16555 1 no 0 0 0 0 0
> Sum 260 44 20 0 171 44
>
>
How about those 171 in keep-alive? What are your KeepAlive and
KeepAliveTimeout settings? Maybe try switching KeepAlive off? Whit only 192
threads and KeepAlive On you are imposing DoS on your self in case of long
timeout.
> That one PID 11454 has "2" connections, 16555 has "1", all other
> values are 0, I guess it should become killed? - but it does not,
> it stays like this for a while (I don't know on what this "while" depends
> on).
>
> At this point all 3 rows are filled with stuff, such as:
>
> WWW_W_WWWWWW_WW_W_W_W_WW__WWW__WW__WWWWWW__WWW__W__W_RWWWWRWWWWW
> GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
> GGGGGGGGGGGGGGGLGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
>
> But as you see, due to several _ this means that there are
> not all 192 busy, in fact:
>
> 105 requests/sec - 2.5 MB/second - 24.8 kB/request
> 44 requests currently being processed, 20 idle workers
>
> WHY can no others be used? With the statement above it means that
> 128 (which is 192-64) workers are unused because they should become
> killed already, to spawn new ones, these are the ones in the 2. and
> 3. row with "G"s.
>
> --
>
> So the rest is occupied with "G". In Apache 2.2 I used MPM_event
> as well, this obviously never occured because the both PIDs above
> which are showing G only showed just dots in that case, now after
> one PID should be finished, all slots become occupied with G,
> which obviously causes a full scoreboard.
>
> I would consider this a bug, a serious one for my case, since it
> often occurs (~1-3 times per hour) that the server stops accepting
> new requests, just because of that full scoreboard, where the server
> is neither busy, nor do I see what configuration directive should
> be configured badly. Again:
>
> KeepAlive On
> ListenBacklog 4095
> Timeout 20
> KeepAliveTimeout 8
> MaxKeepAliveRequests 16192
> MaxRanges 200
> MaxRangeOverlaps 20
> MaxRangeReversals 20
> GracefulShutdownTimeout 6
> LimitInternalRecursion 10
> LimitRequestFieldSize 4094
> LimitRequestFields 40
> LimitRequestLine 4094
> LimitXMLRequestBody 786432
>
> MaxRequestWorkers 192
> ServerLimit 3
> StartServers 2
>
> MaxMemFree 2048
> MinSpareThreads 64
> MaxSpareThreads 64
> ThreadLimit 128
> ThreadsPerChild 64
> MaxConnectionsPerChild 10240
> AsyncRequestWorkerFactor 10
>
>
> I assumed AsyncRequestWorkerFactor to have something to do
> with it, I raised it from 2 (default) to 10, no change.
>
> By the way: after some time (in this case above both) PIDs
> become killed finally, releasing 2x64 slots in the scoreboard,
> the waring messages vanish and the server can respond again
> well to requests. In the interest of finding a workaround until
> this is fixed: what configuration directives control that timeout
> or whatever, which cause these PIDs to become killed?
>
>
>
> On Thu, Sep 6, 2012 at 7:06 PM, Sam <le...@chello.at> wrote:
>
>>
>> Yes, I still see such messages even after upgrading to 2.4.3
>>
>> On Sun, Aug 19, 2012 at 8:25 AM, Sam <le...@chello.at> <le...@chello.at> wrote:
>>
>> I upgraded to Apache 2.4 and use mod_event on a server
>> having average high load.
>>
>> My issue is that I am getting pretty much often the error message
>> within the error log
>>
>> AH00485: scoreboard is full, not at MaxRequestWorkers
>>
>> What causes this? The server almost stops to handle new requests
>> when this occurs.
>>
>> MaxRequestWorkers 192
>> ServerLimit 3
>> StartServers 2
>>
>> MaxMemFree 2048
>> MinSpareThreads 64
>> MaxSpareThreads 64
>> ThreadLimit 64
>> ThreadsPerChild 64
>> MaxConnectionsPerChild 10240
>> AsyncRequestWorkerFactor 2
>>
>>
> So you never have more than 192 simultaneous requests at any given time ???
>
>
>
Re: [users@httpd] scoreboard is full, not at MaxRequestWorkers
Posted by Sam <le...@chello.at>.
Never, the server is not even under high load. There
are several thousand vhosts, it is about shared webhosting.
This is what server-status says:
PID Connections Threads Async connections
total accepting busy idle writing keep-alive closing
21388 257 yes 44 20 0 171 44
11454 2 no 0 0 0 0 0
16555 1 no 0 0 0 0 0
Sum 260 44 20 0 171 44
That one PID 11454has "2" connections, 16555 has "1", all other
values are 0, I guess it should become killed? - but it does not,
it stays like this for a while (I don't know on what this "while"
depends on).
At this point all 3 rows are filled with stuff, such as:
WWW_W_WWWWWW_WW_W_W_W_WW__WWW__WW__WWWWWW__WWW__W__W_RWWWWRWWWWW
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGLGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
But as you see, due to several _ this means that there are
not all 192 busy, in fact:
105 requests/sec - 2.5 MB/second - 24.8 kB/request
44 requests currently being processed, 20 idle workers
WHY can no others be used? With the statement above it means that
128 (which is 192-64) workers are unused because they should become
killed already, to spawn new ones, these are the ones in the 2. and
3. row with "G"s.
--
So the rest is occupied with "G". In Apache 2.2 I used MPM_event
as well, this obviously never occured because the both PIDs above
which are showing G only showed just dots in that case, now after
one PID should be finished, all slots become occupied with G,
which obviously causes a full scoreboard.
I would consider this a bug, a serious one for my case, since it
often occurs (~1-3 times per hour) that the server stops accepting
new requests, just because of that full scoreboard, where the server
is neither busy, nor do I see what configuration directive should
be configured badly. Again:
KeepAlive On
ListenBacklog 4095
Timeout 20
KeepAliveTimeout 8
MaxKeepAliveRequests 16192
MaxRanges 200
MaxRangeOverlaps 20
MaxRangeReversals 20
GracefulShutdownTimeout 6
LimitInternalRecursion 10
LimitRequestFieldSize 4094
LimitRequestFields 40
LimitRequestLine 4094
LimitXMLRequestBody 786432
MaxRequestWorkers 192
ServerLimit 3
StartServers 2
MaxMemFree 2048
MinSpareThreads 64
MaxSpareThreads 64
ThreadLimit 128
ThreadsPerChild 64
MaxConnectionsPerChild 10240
AsyncRequestWorkerFactor 10
I assumed AsyncRequestWorkerFactor to have something to do
with it, I raised it from 2 (default) to 10, no change.
By the way: after some time (in this case above both) PIDs
become killed finally, releasing 2x64 slots in the scoreboard,
the waring messages vanish and the server can respond again
well to requests. In the interest of finding a workaround until
this is fixed: what configuration directives control that timeout
or whatever, which cause these PIDs to become killed?
>
> On Thu, Sep 6, 2012 at 7:06 PM, Sam <lennsen@chello.at
> <ma...@chello.at>> wrote:
>
>
> Yes, I still see such messages even after upgrading to 2.4.3
>
>> On Sun, Aug 19, 2012 at 8:25 AM, Sam<le...@chello.at> <ma...@chello.at> wrote:
>>>> I upgraded to Apache 2.4 and use mod_event on a server
>>>> having average high load.
>>>>
>>>> My issue is that I am getting pretty much often the error message
>>>> within the error log
>>>>
>>>> AH00485: scoreboard is full, not at MaxRequestWorkers
>>>>
>>>> What causes this? The server almost stops to handle new requests
>>>> when this occurs.
>>>>
>>>> MaxRequestWorkers 192
>>>> ServerLimit 3
>>>> StartServers 2
>>>>
>>>> MaxMemFree 2048
>>>> MinSpareThreads 64
>>>> MaxSpareThreads 64
>>>> ThreadLimit 64
>>>> ThreadsPerChild 64
>>>> MaxConnectionsPerChild 10240
>>>> AsyncRequestWorkerFactor 2
>
>
> So you never have more than 192 simultaneous requests at any given
> time ???
>
Re: [users@httpd] scoreboard is full, not at MaxRequestWorkers
Posted by Igor Cicimov <ic...@gmail.com>.
On Thu, Sep 6, 2012 at 7:06 PM, Sam <le...@chello.at> wrote:
> **
>
> Yes, I still see such messages even after upgrading to 2.4.3
>
> On Sun, Aug 19, 2012 at 8:25 AM, Sam <le...@chello.at> <le...@chello.at> wrote:
>
> I upgraded to Apache 2.4 and use mod_event on a server
> having average high load.
>
> My issue is that I am getting pretty much often the error message
> within the error log
>
> AH00485: scoreboard is full, not at MaxRequestWorkers
>
> What causes this? The server almost stops to handle new requests
> when this occurs.
>
> MaxRequestWorkers 192
> ServerLimit 3
> StartServers 2
>
> MaxMemFree 2048
> MinSpareThreads 64
> MaxSpareThreads 64
> ThreadLimit 64
> ThreadsPerChild 64
> MaxConnectionsPerChild 10240
> AsyncRequestWorkerFactor 2
>
>
So you never have more than 192 simultaneous requests at any given time ???
Re: [users@httpd] scoreboard is full, not at MaxRequestWorkers
Posted by Andre Catao <ac...@gmail.com>.
baby baby
On Thu, Sep 6, 2012 at 10:06 AM, Sam <le...@chello.at> wrote:
> **
>
> Yes, I still see such messages even after upgrading to 2.4.3
>
>
> On Sun, Aug 19, 2012 at 8:25 AM, Sam <le...@chello.at> <le...@chello.at> wrote:
>
> I upgraded to Apache 2.4 and use mod_event on a server
> having average high load.
>
> My issue is that I am getting pretty much often the error message
> within the error log
>
> AH00485: scoreboard is full, not at MaxRequestWorkers
>
> What causes this? The server almost stops to handle new requests
> when this occurs.
>
> MaxRequestWorkers 192
> ServerLimit 3
> StartServers 2
>
> MaxMemFree 2048
> MinSpareThreads 64
> MaxSpareThreads 64
> ThreadLimit 64
> ThreadsPerChild 64
> MaxConnectionsPerChild 10240
> AsyncRequestWorkerFactor 2
>
> Just in case you still have this issue, we've switched to 2.4.3 after
> seeing a few bug reports that could be related, and since then I
> haven't seen this again.
>
> Regards,
>
> Martin
>
>
>