You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Stefan Fritsch <sf...@sfritsch.de> on 2007/12/03 23:13:46 UTC

Apache memory usage

Hi,

there is still the problem that during a request, many bucket brigades 
being created which are only cleaned up after the request is 
finished, see 
http://issues.apache.org/bugzilla/show_bug.cgi?id=23567 . There was 
some discussion about retaining ownership of a brigade when 
ap_pass_brigade() is called, and then reusing the brigades [1]. But 
this does not seem to be implemented in 2.2, yet.

But I found two locations where the creation of a new brigade could be 
avoided:

- In buffer_output()/ap_old_write_filter(), it is possible to keep the 
brigade around and reuse it after the next flush.

- In ap_http_chunk_filter(), a new brigade is created for every flush 
bucket. But it is not really necessary if the flush bucket is the 
last bucket in the brigade.

I don't know yet whether the saving is significant in practice. But it
saves two brigades on every combination of ap_rwrite()/ap_rflush()  
and mod_php uses ap_r*() quite a lot.

A patch is attached. It's against 2.2.6 but applies to trunk.

Cheers,
Stefan

[1] 
http://mail-archives.apache.org/mod_mbox/httpd-dev/200504.mbox/<9b5d906c83b6650b9512eb9e7ac96be1%40ricilake.net>

Re: Apache memory usage

Posted by Plüm, Rüdiger, VF-Group <ru...@vodafone.com>.

> -----Ursprüngliche Nachricht-----
> 
> Im Auftrag von Justin Erenkrantz
> Gesendet: Dienstag, 11. Dezember 2007 11:05
> An: dev@httpd.apache.org
> Betreff: Re: Apache memory usage
> 
> 
> On Dec 10, 2007 10:42 PM, Stefan Fritsch  wrote:
> > Yes, that works as well. I wanted to avoid calling 
> APR_BRIGADE_PREPEND
> > when the temporary brigade is empty. But I don't really know whether
> > APR_BRIGADE_PREPEND is so expensive that this makes sense. An
> > alternative would be to check this using APR_BRIGADE_EMPTY.
> 
> FWIW, as the brigade is a ring (read: dual-linked list), prepend or
> append are both constant-time operations - on the order of maybe 2-3
> instructions.
> 
> IOW, it's cheap.  -- justin

Additionally, the first thing the macro does is checking whether the
brigade is empty :-).

Regards

Rüdiger


Re: Apache memory usage

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Dec 10, 2007 10:42 PM, Stefan Fritsch <sf...@sfritsch.de> wrote:
> Yes, that works as well. I wanted to avoid calling APR_BRIGADE_PREPEND
> when the temporary brigade is empty. But I don't really know whether
> APR_BRIGADE_PREPEND is so expensive that this makes sense. An
> alternative would be to check this using APR_BRIGADE_EMPTY.

FWIW, as the brigade is a ring (read: dual-linked list), prepend or
append are both constant-time operations - on the order of maybe 2-3
instructions.

IOW, it's cheap.  -- justin

Re: Apache memory usage

Posted by Plüm, Rüdiger, VF-Group <ru...@vodafone.com>.

> -----Ursprüngliche Nachricht-----
> Von: Stefan Fritsch 
> Gesendet: Montag, 10. Dezember 2007 22:43
> An: dev@httpd.apache.org
> Betreff: Re: Apache memory usage
> 
> 
> On Sunday 09 December 2007, Ruediger Pluem wrote:
> > But I think your patch to server/protocol.c can be done much
> > simpler. Can you try the following and let us know if this helps as
> > well:
> >
> >
> > Index: server/protocol.c
> > ===================================================================
> > --- server/protocol.c   (Revision 602730)
> > +++ server/protocol.c   (Arbeitskopie)
> > @@ -1397,9 +1397,7 @@
> >           * can simply insert our buffered data at the front and
> >           * pass the whole bundle down the chain.
> >           */
> > -        APR_BRIGADE_CONCAT(ctx->bb, bb);
> > -        bb = ctx->bb;
> > -        ctx->bb = NULL;
> > +        APR_BRIGADE_PREPEND(bb, ctx->bb);
> >      }
> >
> >      return ap_pass_brigade(f->next, bb);
> >
> 
> Yes, that works as well. I wanted to avoid calling 
> APR_BRIGADE_PREPEND 
> when the temporary brigade is empty. But I don't really know whether 
> APR_BRIGADE_PREPEND is so expensive that this makes sense. An 
> alternative would be to check this using APR_BRIGADE_EMPTY.

Committed patches for both issues to trunk

http://svn.apache.org/viewvc?view=rev&revision=602735
http://svn.apache.org/viewvc?view=rev&revision=603227

and proposed them for backport

http://svn.apache.org/viewvc?view=rev&revision=603231

Regards

Rüdiger


Re: Apache memory usage

Posted by Stefan Fritsch <sf...@sfritsch.de>.
On Sunday 09 December 2007, Ruediger Pluem wrote:
> But I think your patch to server/protocol.c can be done much
> simpler. Can you try the following and let us know if this helps as
> well:
>
>
> Index: server/protocol.c
> ===================================================================
> --- server/protocol.c   (Revision 602730)
> +++ server/protocol.c   (Arbeitskopie)
> @@ -1397,9 +1397,7 @@
>           * can simply insert our buffered data at the front and
>           * pass the whole bundle down the chain.
>           */
> -        APR_BRIGADE_CONCAT(ctx->bb, bb);
> -        bb = ctx->bb;
> -        ctx->bb = NULL;
> +        APR_BRIGADE_PREPEND(bb, ctx->bb);
>      }
>
>      return ap_pass_brigade(f->next, bb);
>

Yes, that works as well. I wanted to avoid calling APR_BRIGADE_PREPEND 
when the temporary brigade is empty. But I don't really know whether 
APR_BRIGADE_PREPEND is so expensive that this makes sense. An 
alternative would be to check this using APR_BRIGADE_EMPTY.

Thanks for the review.

Cheers,
Stefan

Re: Apache memory usage

Posted by Ruediger Pluem <rp...@apache.org>.

On 12/09/2007 10:03 PM, Stefan Fritsch wrote:
> Hi,
> 
> On Monday 03 December 2007, Stefan Fritsch wrote:
>> But I found two locations where the creation of a new brigade could
>> be avoided:
>>
>> - In buffer_output()/ap_old_write_filter(), it is possible to keep
>> the brigade around and reuse it after the next flush.
>>
>> - In ap_http_chunk_filter(), a new brigade is created for every
>> flush bucket. But it is not really necessary if the flush bucket is
>> the last bucket in the brigade.
>>
>> I don't know yet whether the saving is significant in practice. But
>> it saves two brigades on every combination of
>> ap_rwrite()/ap_rflush() and mod_php uses ap_r*() quite a lot.
>>
>> A patch is attached. It's against 2.2.6 but applies to trunk.
> 
> I now got a report that this patch helped in one case where the system 
> used to run out of memory pretty quickly [1]. Could someone please 
> review the patch? Maybe it's not too late for 2.2.7 yet?

Your first patch to chunk_filter.c seems reasonable to me (committed
in r602735, http://svn.apache.org/viewvc?view=rev&revision=602735 to trunk).
Thanks for this good catch.

But I think your patch to server/protocol.c can be done much simpler.
Can you try the following and let us know if this helps as well:


Index: server/protocol.c
===================================================================
--- server/protocol.c   (Revision 602730)
+++ server/protocol.c   (Arbeitskopie)
@@ -1397,9 +1397,7 @@
          * can simply insert our buffered data at the front and
          * pass the whole bundle down the chain.
          */
-        APR_BRIGADE_CONCAT(ctx->bb, bb);
-        bb = ctx->bb;
-        ctx->bb = NULL;
+        APR_BRIGADE_PREPEND(bb, ctx->bb);
     }

     return ap_pass_brigade(f->next, bb);


Regards

Rüdiger

Re: Apache memory usage

Posted by Stefan Fritsch <sf...@sfritsch.de>.
Hi,

On Monday 03 December 2007, Stefan Fritsch wrote:
> But I found two locations where the creation of a new brigade could
> be avoided:
>
> - In buffer_output()/ap_old_write_filter(), it is possible to keep
> the brigade around and reuse it after the next flush.
>
> - In ap_http_chunk_filter(), a new brigade is created for every
> flush bucket. But it is not really necessary if the flush bucket is
> the last bucket in the brigade.
>
> I don't know yet whether the saving is significant in practice. But
> it saves two brigades on every combination of
> ap_rwrite()/ap_rflush() and mod_php uses ap_r*() quite a lot.
>
> A patch is attached. It's against 2.2.6 but applies to trunk.

I now got a report that this patch helped in one case where the system 
used to run out of memory pretty quickly [1]. Could someone please 
review the patch? Maybe it's not too late for 2.2.7 yet?

Cheers,
Stefan

[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=421557