You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apr.apache.org by Jeff Trawick <tr...@attglobal.net> on 2004/07/28 13:33:27 UTC
[PROOF-OF-CONCEPT?] logging memory used by an allocator
A couple of questions come up from an application perspective:
am I leaking memory? if so, on what operation?
how much memory does it take to perform a certain operation?
If the application can find out how much heap memory is presently owned by a
certain allocator, it can be easier to address such questions.
The attached apr.patch adds apr_allocator_memsize_get() to find the amount of
heap memory presently owned by the allocator. (debug pool flavor not
implemented; what is implemented isn't tested much ;) )
The attached httpd.patch adds %Z log format to mod_log_config to log the memory
size of the allocator used by the request pool. (I would lean towards
implementing this feature in a debug module instead of in mod_log_config.)
The last two fields in the access log below are PID/TID and MEMSIZE.
> "GET / HTTP/1.1" 200 517 23818/5 57448
> "GET / HTTP/1.1" 304 - 23818/5 57448
> "GET /cgi-bin/printenv HTTP/1.1" 200 1241 23818/5 65640
> "GET /cgi-bin/printenv HTTP/1.1" 200 1272 23818/5 65640
> "GET /manual/mod/mod_rewrite.html HTTP/1.1" 200 84992 23819/27 73832
> "GET /manual/style/css/manual.css HTTP/1.1" 200 18172 23819/27 73832
> "GET /manual/style/css/manual-loose-100pc.css HTTP/1.1" 200 2882 23819/27 73832
> "GET /manual/style/css/manual-print.css HTTP/1.1" 200 12977 23818/5 49256
> "GET /manual/images/favicon.ico HTTP/1.1" 200 1078 23819/27 73832
> "GET /manual/images/feather.gif HTTP/1.1" 200 6471 23818/5 49256
> "GET /manual/images/left.gif HTTP/1.1" 200 60 23819/27 73832
> "GET /manual/images/down.gif HTTP/1.1" 200 56 23818/5 49256 <- lower; bug in tracking?
> "GET /manual/images/up.gif HTTP/1.1" 200 57 23819/27 73832
> "GET /manual/images/mod_rewrite_fig1.gif HTTP/1.1" 200 3525 23818/5 49256
> "GET /manual/images/mod_rewrite_fig2.gif HTTP/1.1" 200 2553 23819/27 73832
httpd isn't such a great example of an app where this would be interesting, as
its pool usage has relatively few surprises. Other applications in earlier
stages of development might benefit a lot more.
W.r.t. httpd, there is also an issue that this idea may not help understand bad
memory behavior when some arbitrary plug-in is active. Some commercial
plug-ins are written for multiple web servers and have a relatively small piece
which is specific to Apache httpd (i.e., which utilizes APR pools), so you
could still have lots of memory allocated during the processing of certain
requests without it showing up in the mod_log_config memory trace above. Any
ideas for tracking memory usage regardless of how the memory is allocated from
the heap? The best I can think of is pretty poor for a threaded server:
logging the current break value (sbrk()) and hoping for a rough pattern to
emerge out of lots of data.
Fwd: [PROOF-OF-CONCEPT?] logging memory used by an allocator
Posted by Jeff Trawick <tr...@gmail.com>.
---------- Forwarded message ----------
From: Jeff Trawick <tr...@gmail.com>
Date: Tue, 3 Aug 2004 14:45:16 -0400
Subject: Re: [PROOF-OF-CONCEPT?] logging memory used by an allocator
To: Sander Striker <st...@apache.org>
On Sun, 1 Aug 2004 19:46:14 +0200, Sander Striker <st...@apache.org> wrote:
> > From: Jeff Trawick [mailto:trawick@attglobal.net]
> > Sent: Wednesday, July 28, 2004 1:33 PM
> > To: dev@apr.apache.org; dev@httpd.apache.org
> > Subject: [PROOF-OF-CONCEPT?] logging memory used by an allocator
> >
> > A couple of questions come up from an application perspective:
> >
> > am I leaking memory? if so, on what operation?
> > how much memory does it take to perform a certain operation?
> >
> > If the application can find out how much heap memory is
> > presently owned by a certain allocator, it can be easier to
> > address such questions.
>
> Wouldn't you want to know the memory currently being held by
> the allocator as well as all the memory the allocator dished
> out to it's pools?
I'm sure that could be useful to some folks; I can't think of a use at
the present with Apache unless some logging of the allocator shows
that some request uses a bunch of memory, and we know that there are
various pools in use on that request so we add some temporary debug
code to add pool granularity to the measurement
> > The attached apr.patch adds apr_allocator_memsize_get() to
> > find the amount of heap memory presently owned by the
> > allocator. (debug pool flavor not implemented; what is
> > implemented isn't tested much ;) )
>
> Given that memory management is on the critical path we need
> to be careful what we add. But this patch seems pretty harmless
> in that respect.
but doesn't "memsize_get" suck as a name? any better ideas?
> > The attached httpd.patch adds %Z log format to mod_log_config
> > to log the memory size of the allocator used by the request
> > pool. (I would lean towards implementing this feature in a
> > debug module instead of in mod_log_config.)
>
> I assume you implemented this because of an itch? :)
biggest itch was, in the context of a suspected storage leak, wanting
to draw a line between Apache memory use and some potential
third-party module or library issue; hard without some such
measurements to even understand the normal Apache storage growth in a
threaded server as more and more threads in the process eventually
handle expensive requests
more specifically:
good for a *rough* idea of memory requirements; I haven't a clue how
much heap memory it takes to process some request; 10K? 100K?
good for determining when it is useful to use MaxMemFree by
identifying situations where an infrequent request takes a lot more
pool memory than normal to process; in such case, it is practical to
set MaxMemFree to approximately the "normal" amount of memory
required, since malloc()/free() overhead won't be a killer
Fwd: [PROOF-OF-CONCEPT?] logging memory used by an allocator
Posted by Jeff Trawick <tr...@gmail.com>.
---------- Forwarded message ----------
From: Jeff Trawick <tr...@gmail.com>
Date: Tue, 3 Aug 2004 14:45:16 -0400
Subject: Re: [PROOF-OF-CONCEPT?] logging memory used by an allocator
To: Sander Striker <st...@apache.org>
On Sun, 1 Aug 2004 19:46:14 +0200, Sander Striker <st...@apache.org> wrote:
> > From: Jeff Trawick [mailto:trawick@attglobal.net]
> > Sent: Wednesday, July 28, 2004 1:33 PM
> > To: dev@apr.apache.org; dev@httpd.apache.org
> > Subject: [PROOF-OF-CONCEPT?] logging memory used by an allocator
> >
> > A couple of questions come up from an application perspective:
> >
> > am I leaking memory? if so, on what operation?
> > how much memory does it take to perform a certain operation?
> >
> > If the application can find out how much heap memory is
> > presently owned by a certain allocator, it can be easier to
> > address such questions.
>
> Wouldn't you want to know the memory currently being held by
> the allocator as well as all the memory the allocator dished
> out to it's pools?
I'm sure that could be useful to some folks; I can't think of a use at
the present with Apache unless some logging of the allocator shows
that some request uses a bunch of memory, and we know that there are
various pools in use on that request so we add some temporary debug
code to add pool granularity to the measurement
> > The attached apr.patch adds apr_allocator_memsize_get() to
> > find the amount of heap memory presently owned by the
> > allocator. (debug pool flavor not implemented; what is
> > implemented isn't tested much ;) )
>
> Given that memory management is on the critical path we need
> to be careful what we add. But this patch seems pretty harmless
> in that respect.
but doesn't "memsize_get" suck as a name? any better ideas?
> > The attached httpd.patch adds %Z log format to mod_log_config
> > to log the memory size of the allocator used by the request
> > pool. (I would lean towards implementing this feature in a
> > debug module instead of in mod_log_config.)
>
> I assume you implemented this because of an itch? :)
biggest itch was, in the context of a suspected storage leak, wanting
to draw a line between Apache memory use and some potential
third-party module or library issue; hard without some such
measurements to even understand the normal Apache storage growth in a
threaded server as more and more threads in the process eventually
handle expensive requests
more specifically:
good for a *rough* idea of memory requirements; I haven't a clue how
much heap memory it takes to process some request; 10K? 100K?
good for determining when it is useful to use MaxMemFree by
identifying situations where an infrequent request takes a lot more
pool memory than normal to process; in such case, it is practical to
set MaxMemFree to approximately the "normal" amount of memory
required, since malloc()/free() overhead won't be a killer
RE: [PROOF-OF-CONCEPT?] logging memory used by an allocator
Posted by Sander Striker <st...@apache.org>.
> From: Jeff Trawick [mailto:trawick@attglobal.net]
> Sent: Wednesday, July 28, 2004 1:33 PM
> To: dev@apr.apache.org; dev@httpd.apache.org
> Subject: [PROOF-OF-CONCEPT?] logging memory used by an allocator
>
> A couple of questions come up from an application perspective:
>
> am I leaking memory? if so, on what operation?
> how much memory does it take to perform a certain operation?
>
> If the application can find out how much heap memory is
> presently owned by a certain allocator, it can be easier to
> address such questions.
Wouldn't you want to know the memory currently being held by
the allocator as well as all the memory the allocator dished
out to it's pools?
> The attached apr.patch adds apr_allocator_memsize_get() to
> find the amount of heap memory presently owned by the
> allocator. (debug pool flavor not implemented; what is
> implemented isn't tested much ;) )
Given that memory management is on the critical path we need
to be careful what we add. But this patch seems pretty harmless
in that respect.
> The attached httpd.patch adds %Z log format to mod_log_config
> to log the memory size of the allocator used by the request
> pool. (I would lean towards implementing this feature in a
> debug module instead of in mod_log_config.)
I assume you implemented this because of an itch? :)
Sander
RE: [PROOF-OF-CONCEPT?] logging memory used by an allocator
Posted by Sander Striker <st...@apache.org>.
> From: Jeff Trawick [mailto:trawick@attglobal.net]
> Sent: Wednesday, July 28, 2004 1:33 PM
> To: dev@apr.apache.org; dev@httpd.apache.org
> Subject: [PROOF-OF-CONCEPT?] logging memory used by an allocator
>
> A couple of questions come up from an application perspective:
>
> am I leaking memory? if so, on what operation?
> how much memory does it take to perform a certain operation?
>
> If the application can find out how much heap memory is
> presently owned by a certain allocator, it can be easier to
> address such questions.
Wouldn't you want to know the memory currently being held by
the allocator as well as all the memory the allocator dished
out to it's pools?
> The attached apr.patch adds apr_allocator_memsize_get() to
> find the amount of heap memory presently owned by the
> allocator. (debug pool flavor not implemented; what is
> implemented isn't tested much ;) )
Given that memory management is on the critical path we need
to be careful what we add. But this patch seems pretty harmless
in that respect.
> The attached httpd.patch adds %Z log format to mod_log_config
> to log the memory size of the allocator used by the request
> pool. (I would lean towards implementing this feature in a
> debug module instead of in mod_log_config.)
I assume you implemented this because of an itch? :)
Sander