scoreboard full

[Stefan Eissing <st...@greenbytes.de>]

Hunting a bit on the "scoreboard is full, not at MaxRequestWorkers" issue,
I found that setting

    c->cs->state = CONN_STATE_LINGER;

explicitly on exiting connection processing (h2_conn.c) seems to help 
with this. As an experiment, I set

    c->cs->state = CONN_STATE_READ_REQUEST_LINE;

and observed ever increasing memory use by the processes. With setting
state to CONN_STATE_LINGER explicilty, I have stable memory use and have
not seen a full scoreboard since.

I have my theory of what goes wrong below, maybe someone with a better
understanding of mpm_event wants to run this through her head...

Maybe connection state on finishing processing needs to be checked
either better, or the code in mpm_event needs to become more robust.


//Stefan

------------------------------------------------------------------------------
Although I had not configured MaxRequestsPerChild, someone decided that
the child process should close its listener gracefully. In event.c
listener_may_exit is set and the listener is woken up.
The listener sees this, sets all to close and since it is gracefully,
if checks if any connections are still open.

This is the case if the connection pools are not cleaned up properly!
Without cleanup, the connection counter in event.c is never decremented
and the child process will never exit. In the server-status, this is
visible as all workers of a process in state 'G', which never changes.

And sometime later the "scoreboard is full" log message will come...

<green/>bytes GmbH
Hafenweg 16, 48155 Münster, Germany
Phone: +49 251 2807760. Amtsgericht Münster: HRB5782

Re: scoreboard full

[Stefan Eissing <st...@greenbytes.de>]

> Am 31.08.2015 um 22:19 schrieb Yann Ylavic <yl...@gmail.com>:
> 
> On Mon, Aug 31, 2015 at 6:11 PM, Stefan Eissing
> <st...@greenbytes.de> wrote:
>> Hunting a bit on the "scoreboard is full, not at MaxRequestWorkers" issue,
>> I found that setting
>> 
>>    c->cs->state = CONN_STATE_LINGER;
> 
> mod_http2 could possiblly return CONN_STATE_WRITE_COMPLETION (with
> c->keepalive = AP_CONN_CLOSE, unless c->aborted), so that blocking
> writes (if any) can be scheduled (in event).

Yes, you're right. Thanks for the suggestion.

>> explicitly on exiting connection processing (h2_conn.c) seems to help
>> with this. As an experiment, I set
>> 
>>    c->cs->state = CONN_STATE_READ_REQUEST_LINE;
> 
> This state looks unhandled after ap_run_process_connecton, shouldn't happen :)
> The case should probably be handled like CONN_STATE_LINGER (or
> DEBUG_ASSERT?) since ptrans leaks...

Such an unhandeled case leads to similar behaviour as noticed in PR 53555. That
might be accidental, but experience suggests it is worth looking at. ;-)

If, for some reason, ap_run_process_connection(c) (event.c, line 1137) returns
with a connection state other than:
CONN_STATE_WRITE_COMPLETION
CONN_STATE_LINGER
CONN_STATE_SUSPENDED

it looks to me as if process_socket(...) does not schedule any further event
handling on the socket. That seems, in my reading, to lead to
- no EOC written
- no c->pool cleanup
- no scoreboard connection count decrease
which seems to have the effect that
- the process will never exit gracefully on its own
- the scoreboard slot remains occupied although the process has not reached max workers

If this continues to happen, the server will become unresponsive.

*Why* ap_run_process_connection(c) might return with such a connection state, I do
not know. I know that mod_h2 has caused such (and is now being fixed). Other connection
hooks could do the same, or a socket read error at the wrong time may cause this?

Also a failed ap_run_pre_connection(c, sock) (event.c, line 1087) will set the state
to CONN_STATE_READ_REQUEST_LINE, never enter ap_run_process_connection(c) and "fall through".

So, I remain unconvinced that this is not related... ;-)

> PR 53555 seems not related though, mpm_event and mod_http call
> ap_process_http_async_connection::ap_process_async_request::ap_process_request_after_handler()
> which do the right thing wrt cs->state.
> 
> Regards,
> Yann.

<green/>bytes GmbH
Hafenweg 16, 48155 Münster, Germany
Phone: +49 251 2807760. Amtsgericht Münster: HRB5782

Re: scoreboard full

[Yann Ylavic <yl...@gmail.com>]

On Mon, Aug 31, 2015 at 6:11 PM, Stefan Eissing
<st...@greenbytes.de> wrote:
> Hunting a bit on the "scoreboard is full, not at MaxRequestWorkers" issue,
> I found that setting
>
>     c->cs->state = CONN_STATE_LINGER;

mod_http2 could possiblly return CONN_STATE_WRITE_COMPLETION (with
c->keepalive = AP_CONN_CLOSE, unless c->aborted), so that blocking
writes (if any) can be scheduled (in event).

>
> explicitly on exiting connection processing (h2_conn.c) seems to help
> with this. As an experiment, I set
>
>     c->cs->state = CONN_STATE_READ_REQUEST_LINE;

This state looks unhandled after ap_run_process_connecton, shouldn't happen :)
The case should probably be handled like CONN_STATE_LINGER (or
DEBUG_ASSERT?) since ptrans leaks...

PR 53555 seems not related though, mpm_event and mod_http call
ap_process_http_async_connection::ap_process_async_request::ap_process_request_after_handler()
which do the right thing wrt cs->state.

Regards,
Yann.