Cleanup/desctruction of connection pool and associated bucket_alloc

[Bojan Smojver <bo...@rexursive.com>]

I noticed that if a large number of buckets in a brigade are sent out,
the resident memory footprint of httpd process (been playing with 2.2.6
for now) will go up significantly.

For instance, one could replicate this behaviour by having a file
processed by the INCLUDES filter, which contains a lot of references
(say a thousand) to something like this:

<!--#include virtual='somefile.html' -->

The size of the somefile.html does not matter (it can actually be zero).
In this particular example, resident size of httpd jumped from 3 to 11
MB. The served file in question was about 40 kB in size (i.e. the SHTML
file containing the virtual directive). Quite a bit for such a small
chunk of HTML being pushed out.

What appears to be happening is that conn->pool and conn->bucket_alloc
do not get destroyed (but rather just cleaned), which then causes the
footprint of the process to go up, given that a lot of buckets were
allocated. If fact, even destroying conn->pool does not help, because it
would appear that conn->pool is not the owner of its allocator.
Destroying conn->pool->parent brings the memory footprint of httpd back
in check.

Now imagine someone (like yours truly :-) writing a handler/filter that
sends many, many buckets inside a brigade down the filter chain. This
causes the httpd process to start consuming many, many megabytes (in
some instances I measured almost 500 MB in my tests), which are never
returned. Then imagine multiple httpd processes doing the same thing and
not releasing any of that memory back. The machine goes into DOS
quickly, due to excessive swapping.

Sure, I could fix my code to slam buckets together to reduce the number
of them, but that would not fix any other handler/filter (e.g.
mod_include). So, I'm guessing the correct fix would be to:

- make conn->pool have/own its own allocator
- destroy, rather then clear conn->pool on connection close

Thoughts?

-- 
Bojan

Re: Cleanup/desctruction of connection pool and associated bucket_alloc

[Bojan Smojver <bo...@rexursive.com>]

On Fri, 2007-10-05 at 15:37 +1000, Bojan Smojver wrote:
> I noticed that if a large number of buckets in a brigade are sent out,
> the resident memory footprint of httpd process (been playing with 2.2.6
> for now) will go up significantly.

I didn't mention this before, but MaxMemFree directive doesn't seem to
have any effect on this.

-- 
Bojan

Re: Cleanup/desctruction of connection pool and associated bucket_alloc

[Bojan Smojver <bo...@rexursive.com>]

Quoting Aleksey Midenkov <as...@uezku.kemsu.ru>:

> And what if a large file is downloaded and processed by filters? Did the
> buckets allocated by filters will not be deallocated until the connection
> end? This can be a cause of DOS. The buckets should be freed after they have
> flushed out of ap_core_output_filter.

All buckets are allocated from conn->bucket_alloc and are freed when  
the brigade is destroyed. If that weren't the case, subsequent  
requests would be increasing the allocation (this is not happening).

The problem is that the allocator gets its blocks back, but never  
releases that back to the OS.

-- 
Bojan

Re: Cleanup/desctruction of connection pool and associated bucket_alloc

[Aleksey Midenkov <as...@uezku.kemsu.ru>]

And what if a large file is downloaded and processed by filters? Did the 
buckets allocated by filters will not be deallocated until the connection 
end? This can be a cause of DOS. The buckets should be freed after they have 
flushed out of ap_core_output_filter.

On Friday 05 October 2007 09:37:57 Bojan Smojver wrote:
> I noticed that if a large number of buckets in a brigade are sent out,
> the resident memory footprint of httpd process (been playing with 2.2.6
> for now) will go up significantly.
>
> For instance, one could replicate this behaviour by having a file
> processed by the INCLUDES filter, which contains a lot of references
> (say a thousand) to something like this:
>
> <!--#include virtual='somefile.html' -->
>
> The size of the somefile.html does not matter (it can actually be zero).
> In this particular example, resident size of httpd jumped from 3 to 11
> MB. The served file in question was about 40 kB in size (i.e. the SHTML
> file containing the virtual directive). Quite a bit for such a small
> chunk of HTML being pushed out.
>
> What appears to be happening is that conn->pool and conn->bucket_alloc
> do not get destroyed (but rather just cleaned), which then causes the
> footprint of the process to go up, given that a lot of buckets were
> allocated. If fact, even destroying conn->pool does not help, because it
> would appear that conn->pool is not the owner of its allocator.
> Destroying conn->pool->parent brings the memory footprint of httpd back
> in check.
>
> Now imagine someone (like yours truly :-) writing a handler/filter that
> sends many, many buckets inside a brigade down the filter chain. This
> causes the httpd process to start consuming many, many megabytes (in
> some instances I measured almost 500 MB in my tests), which are never
> returned. Then imagine multiple httpd processes doing the same thing and
> not releasing any of that memory back. The machine goes into DOS
> quickly, due to excessive swapping.
>
> Sure, I could fix my code to slam buckets together to reduce the number
> of them, but that would not fix any other handler/filter (e.g.
> mod_include). So, I'm guessing the correct fix would be to:
>
> - make conn->pool have/own its own allocator
> - destroy, rather then clear conn->pool on connection close
>
> Thoughts?

Re: Cleanup/desctruction of connection pool and associated bucket_alloc

[Bojan Smojver <bo...@rexursive.com>]

Quoting Joe Orton <jo...@redhat.com>:

> It sounds like that is the root cause.  If you create a brigade with N
> buckets in for arbitrary values of N, expect maximum memory consumption
> to be O(N).  The output filtering guide touches on this:
>
> http://httpd.apache.org/docs/trunk/developer/output-filters.html
>
> Filters need to be written to pass processed buckets down the filter
> chain ASAP rather than buffering them up into big brigades.  Likewise
> for a content generator - buffering up a hundred 1MB HEAP buckets in a
> brigade will obviously give you a maximum heap usage of ~100MB; instead,
> pass each HEAP bucket down the filter chain as its generated and you get
> maximum of ~1MB.
>
> Most of the shipped filters do behave correctly in this respect (though
> there are problems with the handling of FLUSH buckets); see e.g. the
> AP_MIN_BYTES_TO_WRITE handling in mod_include.

All true. The problem in both cases was created by the number of  
buckets, rather than by the size of them. At first I was suspecting I  
was doing something stupid in my code, so I checked out mod_include to  
see what happens there and to my surprise, it was eating memory and  
not returning it to OS too.

What I was worried about here is that after the connection is closed,  
the memory used for all this is never reclaimed, which is what  
actually caused DOS (i.e. multiple huge processes started being  
swapped out). Not sure how far mod_include would push this (I can test  
with say 10,000+ includes and see what happens), but maybe it would be  
better to destroy the pool, just like the request pool gets destroyed,  
to be on the safe side. But then again, if a module is buggy, it's not  
Apache's problem...

Not sure why MaxMemFree never kicked in to saved the day. Gdb says  
that the value was passed in correctly and all and yet, with no  
effect...

-- 
Bojan

Re: Cleanup/desctruction of connection pool and associated bucket_alloc

[Joe Orton <jo...@redhat.com>]

On Fri, Oct 05, 2007 at 03:37:57PM +1000, Bojan Smojver wrote:
> Now imagine someone (like yours truly :-) writing a handler/filter that
> sends many, many buckets inside a brigade down the filter chain. This
> causes the httpd process to start consuming many, many megabytes (in
> some instances I measured almost 500 MB in my tests), which are never
> returned. Then imagine multiple httpd processes doing the same thing and
> not releasing any of that memory back. The machine goes into DOS
> quickly, due to excessive swapping.

It sounds like that is the root cause.  If you create a brigade with N 
buckets in for arbitrary values of N, expect maximum memory consumption 
to be O(N).  The output filtering guide touches on this:

http://httpd.apache.org/docs/trunk/developer/output-filters.html

Filters need to be written to pass processed buckets down the filter 
chain ASAP rather than buffering them up into big brigades.  Likewise 
for a content generator - buffering up a hundred 1MB HEAP buckets in a 
brigade will obviously give you a maximum heap usage of ~100MB; instead, 
pass each HEAP bucket down the filter chain as its generated and you get 
maximum of ~1MB.

Most of the shipped filters do behave correctly in this respect (though 
there are problems with the handling of FLUSH buckets); see e.g. the 
AP_MIN_BYTES_TO_WRITE handling in mod_include.

joe

Re: Cleanup/desctruction of connection pool and associated bucket_alloc

[Bojan Smojver <bo...@rexursive.com>]

On Fri, 2007-10-05 at 16:56 +1000, Bojan Smojver wrote:

> Keep you posted...

Another approach attached.

-- 
Bojan

Re: Cleanup/desctruction of connection pool and associated bucket_alloc

[Bojan Smojver <bo...@rexursive.com>]

On Fri, 2007-10-05 at 16:06 +1000, Bojan Smojver wrote:

> Example patch attached.

Sorry, that patch is seriously brain dead. I'll try modifying the
prefork.c instead, which seems like a better place to do it.

Keep you posted...

-- 
Bojan

Re: Cleanup/desctruction of connection pool and associated bucket_alloc

[Bojan Smojver <bo...@rexursive.com>]

On Fri, 2007-10-05 at 15:37 +1000, Bojan Smojver wrote:

> - make conn->pool have/own its own allocator
> - destroy, rather then clear conn->pool on connection close

Example patch attached.

-- 
Bojan