You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Graham Leggett <mi...@sharp.fm> on 2002/03/01 05:23:52 UTC
Allocating a buffer efficiently...?
Hi all,
In a recent patch to mod_proxy, a static buffer used to store data read
from backend before it was given to frontend was changed to be allocated
dynamically from a pool like so:
+ /* allocate a buffer to store the bytes in */
+ /* make sure it is at least IOBUFSIZE, as recv_buffer_size may be
zero for
system default */
+ buf_size = MAX(recv_buffer_size, IOBUFSIZE);
+ buf = ap_palloc(r->pool, buf_size);
This change allows for a dynamically configurable buffer size, and fixes
the code to be thread safe.
However: it has been pointed out that this new code makes the Apache
footprint significantly larger like so:
> There is one drawback in this code. ap_palloc() is not good for
> big allocations (I think > 16K) because it stores data and meta-data
> together. I had found this when try to allocate memory from pool
> for zlib in mod_deflate. zlib needs about 390K - 2*128K + 2*64K + 6K.
> After this change Apache had grown up about 2M after about hour
> with 50 requests/s. I'm not sure that this growing could continue but
> I did not want additional 2M on each Apache.
>
> I use malloc for big allocations, store addresses in array
> allocated from pool and set cleanup for this array.
> In cleanup I free addresses if they is not free already.
Comments...?
Regards,
Graham
--
-----------------------------------------
minfrin@sharp.fm "There's a moon
over Bourbon Street
tonight..."
Re: Allocating a buffer efficiently...?
Posted by Graham Leggett <mi...@sharp.fm>.
Sander Striker wrote:
> Can you point me to the original post? I'd like to see the context.
> Specifically which pool is being used.
The original was <Pi...@is>.
Regards,
Graham
--
-----------------------------------------
minfrin@sharp.fm "There's a moon
over Bourbon Street
tonight..."
RE: Allocating a buffer efficiently...?
Posted by Sander Striker <st...@apache.org>.
> From: Igor Sysoev [mailto:is@rambler-co.ru]
> Sent: 02 March 2002 16:41
> On Sat, 2 Mar 2002, Sander Striker wrote:
>
> > > In a recent patch to mod_proxy, a static buffer used to store data read
> > > from backend before it was given to frontend was changed to be allocated
> > > dynamically from a pool like so:
> > >
> > > + /* allocate a buffer to store the bytes in */
> > > + /* make sure it is at least IOBUFSIZE, as recv_buffer_size may be
> > > zero for
> > > system default */
> > > + buf_size = MAX(recv_buffer_size, IOBUFSIZE);
> > > + buf = ap_palloc(r->pool, buf_size);
> > >
> > > This change allows for a dynamically configurable buffer size, and fixes
> > > the code to be thread safe.
> > >
> > > However: it has been pointed out that this new code makes the Apache
> > > footprint significantly larger like so:
> > >
> > > > There is one drawback in this code. ap_palloc() is not good for
> > > > big allocations (I think > 16K) because it stores data and meta-data
> > > > together. I had found this when try to allocate memory from pool
> > > > for zlib in mod_deflate. zlib needs about 390K - 2*128K + 2*64K + 6K.
> > > > After this change Apache had grown up about 2M after about hour
> > > > with 50 requests/s. I'm not sure that this growing could continue but
> > > > I did not want additional 2M on each Apache.
> >
> > Can you point me to the original post? I'd like to see the context.
> > Specifically which pool is being used.
>
> You see all context - Graham have quoted almost whole my email.
> As to pool I had tried to make big allocation from
> r->connection->client->pool. Keep-alives were off.
What you are seeing is the big allocation being pushed on the freelist.
You can prevent this by creating a subpool of the pool you were using
with a new allocator. When the subpool is destroyed (and the allocator
to go with it) the memory is handed back to the system.
> > > > I use malloc for big allocations, store addresses in array
> > > > allocated from pool and set cleanup for this array.
> > > > In cleanup I free addresses if they is not free already.
> > >
> > > Comments...?
We discussed this and came up with apr_tracker. This basically
does malloc/free and can free all mallocs in one go aswell. We (*cough* I)
never came round to finishing it.
> Igor Sysoev
Sander
RE: Allocating a buffer efficiently...?
Posted by Igor Sysoev <is...@rambler-co.ru>.
On Sat, 2 Mar 2002, Sander Striker wrote:
> > In a recent patch to mod_proxy, a static buffer used to store data read
> > from backend before it was given to frontend was changed to be allocated
> > dynamically from a pool like so:
> >
> > + /* allocate a buffer to store the bytes in */
> > + /* make sure it is at least IOBUFSIZE, as recv_buffer_size may be
> > zero for
> > system default */
> > + buf_size = MAX(recv_buffer_size, IOBUFSIZE);
> > + buf = ap_palloc(r->pool, buf_size);
> >
> > This change allows for a dynamically configurable buffer size, and fixes
> > the code to be thread safe.
> >
> > However: it has been pointed out that this new code makes the Apache
> > footprint significantly larger like so:
> >
> > > There is one drawback in this code. ap_palloc() is not good for
> > > big allocations (I think > 16K) because it stores data and meta-data
> > > together. I had found this when try to allocate memory from pool
> > > for zlib in mod_deflate. zlib needs about 390K - 2*128K + 2*64K + 6K.
> > > After this change Apache had grown up about 2M after about hour
> > > with 50 requests/s. I'm not sure that this growing could continue but
> > > I did not want additional 2M on each Apache.
>
> Can you point me to the original post? I'd like to see the context.
> Specifically which pool is being used.
You see all context - Graham have quoted almost whole my email.
As to pool I had tried to make big allocation from
r->connection->client->pool. Keep-alives were off.
> > > I use malloc for big allocations, store addresses in array
> > > allocated from pool and set cleanup for this array.
> > > In cleanup I free addresses if they is not free already.
> >
> > Comments...?
Igor Sysoev
RE: Allocating a buffer efficiently...?
Posted by Sander Striker <st...@apache.org>.
> -----Original Message-----
> From: minfrin [mailto:minfrin]On Behalf Of Graham Leggett
> Sent: 01 March 2002 05:24
> To: Apache Developers List
> Subject: Allocating a buffer efficiently...?
>
>
> Hi all,
>
> In a recent patch to mod_proxy, a static buffer used to store data read
> from backend before it was given to frontend was changed to be allocated
> dynamically from a pool like so:
>
> + /* allocate a buffer to store the bytes in */
> + /* make sure it is at least IOBUFSIZE, as recv_buffer_size may be
> zero for
> system default */
> + buf_size = MAX(recv_buffer_size, IOBUFSIZE);
> + buf = ap_palloc(r->pool, buf_size);
>
> This change allows for a dynamically configurable buffer size, and fixes
> the code to be thread safe.
>
> However: it has been pointed out that this new code makes the Apache
> footprint significantly larger like so:
>
> > There is one drawback in this code. ap_palloc() is not good for
> > big allocations (I think > 16K) because it stores data and meta-data
> > together. I had found this when try to allocate memory from pool
> > for zlib in mod_deflate. zlib needs about 390K - 2*128K + 2*64K + 6K.
> > After this change Apache had grown up about 2M after about hour
> > with 50 requests/s. I'm not sure that this growing could continue but
> > I did not want additional 2M on each Apache.
Can you point me to the original post? I'd like to see the context.
Specifically which pool is being used.
> > I use malloc for big allocations, store addresses in array
> > allocated from pool and set cleanup for this array.
> > In cleanup I free addresses if they is not free already.
>
> Comments...?
>
> Regards,
> Graham
Sander