svn commit: r1801655 - /httpd/httpd/trunk/server/mpm/winnt/child.c

[ko...@apache.org]

Author: kotkov
Date: Tue Jul 11 21:07:45 2017
New Revision: 1801655

URL: http://svn.apache.org/viewvc?rev=1801655&view=rev
Log:
mpm_winnt: Use a LIFO stack instead of a FIFO queue to hold unused
completion contexts, as that may significantly reduce the memory usage.

This simple change can have a noticeable impact on the amount of memory
consumed by the child process in various cases.  Every completion context
in the queue has an associated allocator, and every allocator has it's
ap_max_mem_free memory limit which is not given back to the operating
system.  Once the queue grows, it cannot shrink back, and every allocator
in each of the queued completion contexts keeps up to its max_free amount
of memory.  The queue can only grow when a server has to serve multiple
concurrent connections at once.

With that in mind, consider a case with a server that doesn't encounter many
concurrent connections most of the time, but has occasional spikes when
it has to serve multiple concurrent connections.  During such spikes, the
size of the completion context queue grows.

The actual difference between using LIFO and FIFO orders shows up after
such spikes, when the server is back to light load and doesn't see a lot
of concurrency.  With FIFO order, every completion context in the queue
will be used in a round-robin manner, thus using *every* available allocator
one by one and ultimately claiming up to (N * ap_max_mem_free memory) from
the OS.  With LIFO order, only the completion contexts that are close to
the top of the stack will be used and reused for subsequent connections.
Hence, only a small part of the allocators will be used, and this can
prevent all other allocators from unnecessarily acquiring memory from
the OS (and keeping it), and this reduces the overall memory footprint.

Please note that this change doesn't affect the worst case behavior, as
it's still (N * ap_max_mem_free memory), but tends to behave better in
practice, for the reasons described above.

Another thing worth considering is the new behavior when the OS decides
to swap out pages of the child process, for example, in a close-to-OOM
condition.  Handling every new connection after the swap requires the OS
to load the memory pages for the allocator from the completion context that
is used for this connection.  With FIFO order, the completion contexts are
used one by one, and this would cause page loads for every new connection.
With LIFO order, there will be almost no swapping, since the same completion
context is going to be reused for subsequent new connections.

Modified:
    httpd/httpd/trunk/server/mpm/winnt/child.c

Modified: httpd/httpd/trunk/server/mpm/winnt/child.c
URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/server/mpm/winnt/child.c?rev=1801655&r1=1801654&r2=1801655&view=diff
==============================================================================
--- httpd/httpd/trunk/server/mpm/winnt/child.c (original)
+++ httpd/httpd/trunk/server/mpm/winnt/child.c Tue Jul 11 21:07:45 2017
@@ -119,7 +119,6 @@ static HANDLE max_requests_per_child_eve
 static apr_thread_mutex_t  *child_lock;
 static apr_thread_mutex_t  *qlock;
 static winnt_conn_ctx_t *qhead = NULL;
-static winnt_conn_ctx_t *qtail = NULL;
 static apr_uint32_t num_completion_contexts = 0;
 static apr_uint32_t max_num_completion_contexts = 0;
 static HANDLE ThreadDispatchIOCP = NULL;
@@ -147,13 +146,11 @@ static void mpm_recycle_completion_conte
         ResetEvent(context->overlapped.hEvent);
 
         apr_thread_mutex_lock(qlock);
-        if (qtail) {
-            qtail->next = context;
-        } else {
-            qhead = context;
+        if (!qhead) {
             SetEvent(qwait_event);
         }
-        qtail = context;
+        context->next = qhead;
+        qhead = context;
         apr_thread_mutex_unlock(qlock);
     }
 }
@@ -170,8 +167,6 @@ static apr_status_t mpm_get_completion_c
         if (qhead) {
             context = qhead;
             qhead = qhead->next;
-            if (!qhead)
-                qtail = NULL;
         } else {
             ResetEvent(qwait_event);
         }