You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Marc Perkel <ma...@perkel.com> on 2005/10/21 18:19:14 UTC
[users@httpd] Serious Memory Leak Problem
I've asked about this before and never got an answer. I used to run my
server on a dual xeon computer and it was very memory efficient. I moved
to the 64 bit version of of Fedora Core 4 and now it's filling up memory
really fast.
Both servers had 4 gigs of ram. And basically the same configuration
file using the same modules for both. It is especially bad where perl is
involved.
Now I have to set MaxRequestsPerChild to no more than 10 to keep ram
from filling up. The server is rather busy serving 67 requests per
second. If I set MaxRequestsPerChild to 30 - it quickly fills up
another gig of ram.
I really need to fix this. Sure could use some help on trying to figure
out where the problem is.
Thanks in advance.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Serious Memory Leak Problem
Posted by Marc Perkel <ma...@perkel.com>.
Nick Kew wrote:
>On Wednesday 26 October 2005 13:44, Marc Perkel wrote:
>
>
>>[chop]
>>
>>
>
>Did you ever tall us whether you compiled it yourself or got it
>from a package?
>
>If the latter, I suggest compiling it on and for your hardware,
>in case you simply have a binary that's not quite compatible.
>
>
>
>
Used stock Fedora Core 4 RPMS.
--
Marc Perkel - marc@perkel.com
Spam Filter: http://www.junkemailfilter.com
My Blog: http://marc.perkel.com
Re: [users@httpd] Serious Memory Leak Problem
Posted by Nick Kew <ni...@webthing.com>.
On Wednesday 26 October 2005 13:44, Marc Perkel wrote:
> [chop]
Did you ever tall us whether you compiled it yourself or got it
from a package?
If the latter, I suggest compiling it on and for your hardware,
in case you simply have a binary that's not quite compatible.
--
Nick Kew
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Serious Memory Leak Problem
Posted by Marc Perkel <ma...@perkel.com>.
David Tonhofer, m-plify S.A. wrote:
>
>
> --On Friday, October 21, 2005 11:16 AM -0700 Marc Perkel
> <ma...@perkel.com> wrote:
>
>>
>>
>> David Tonhofer, m-plify S.A. wrote:
>>
>>> I really don't know what the problem could be but let's
>>> start a discussion:
>>>
>>> 1) How many children are there?
>>> 2) What is the sum of the processes RSS size?
>>> 3) What is the sum of the processes VSIZE size?
>>>
>> Thanks for your perl script. Here's the results:
>>
>> 1029308 minor faults so far
>> 227 major faults so far
>> 1532 user time jiffies burnt so far
>> 25812 kernel time jiffies burnt so far
>> 55062589440 bytes of allocated virtual memory
>> 9328185344 bytes of resident memory allocated
>> 294 children active right now
>>
>> Excuse my ignorance but what does the mean?
>>
>> Thanks in advance
>
>
>
> Well,
>
> "minor faults"/"major faults" are about paging (if I remember
> correctly, a minor fault means "page has to be read from disk,
> scratching a page in memory" and major fault means "page has to
> be read from disk, but before that an existing page has to be written")
>
> The jiffies express how many CPU has been used on the program,
> they used to be 1/100 of second units, not sure whether that still
> holds.
>
> What is interesting here is:
>
>> 55'062'589'440 bytes of allocated virtual memory
>> 9'328'185'344 bytes of resident memory allocated
>
>
> The first number is just the sum of the processes' virtual memory
> size - they indicate that they want 55 GByte in toto, but this being
> just 'virtual', it's not a problem.
>
> The second number is the sum of the processes' effective memory size -
> 9 GByte of RAM... more than you actually have? (scratches head) That
> shouldn't be possible, except if some RAM is counted twice. Damn.
>
> Here are my number (light charge on an dual XEON on RH ES4, only
> static serves, some PHP though but mostly Tomcat running the show):
>
> 87862 minor faults so far
> 9291 major faults so far
> 869 user time jiffies burnt so far
> 379 kernel time jiffies burnt so far
> 1271640064 bytes of allocated virtual memory
> 50085888 bytes of resident memory allocated
> 10 children active right now
>
> Acid test: ~ 5 MByte resident memory per child for me
> ~31 MByte resident memory per child for you
>
> This does not sound too unreasonable if there is a lot of Perl going on.
>
> We need more numbers...
>
Thanks for your help. I was using mod_perl and I uninstalled it trying
to isolate that in case it was somehow the problem. No difference. And,
like you - I used to run a dual xeon system and at that time I was
amazed by how little memory it was using. The difference that I know of is:
1) I'm running a dual core Athlon instead of 2 xeon processors.
2) I'm running the 64 bit version of fedora Core 4 instead of the 32 bt
version.
I also had a second server that had a P4 processor - not hyperthreaded -
and memory usage was also very low on that.
Here's some data I saved:
On the new server running top it shows *httpd* processes using:
virt 201m
res 53m
shr 11m
On the old server the *httpd* process show:
virt: 50k
res 14m
shr 10k
That's quite a difference.
What can I do to nail this down. Or - is the 64 bit version just broken?
Or is the a Dual Core Athlon related problem?
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Serious Memory Leak Problem
Posted by Marc Perkel <ma...@perkel.com>.
David Tonhofer, m-plify S.A. wrote:
>
>
> --On Friday, October 21, 2005 11:16 AM -0700 Marc Perkel
> <ma...@perkel.com> wrote:
>
>>
>>
>> David Tonhofer, m-plify S.A. wrote:
>>
>>> I really don't know what the problem could be but let's
>>> start a discussion:
>>>
>>> 1) How many children are there?
>>> 2) What is the sum of the processes RSS size?
>>> 3) What is the sum of the processes VSIZE size?
>>>
>> Thanks for your perl script. Here's the results:
>>
>> 1029308 minor faults so far
>> 227 major faults so far
>> 1532 user time jiffies burnt so far
>> 25812 kernel time jiffies burnt so far
>> 55062589440 bytes of allocated virtual memory
>> 9328185344 bytes of resident memory allocated
>> 294 children active right now
>>
>> Excuse my ignorance but what does the mean?
>>
>> Thanks in advance
>
>
>
> Well,
>
> "minor faults"/"major faults" are about paging (if I remember
> correctly, a minor fault means "page has to be read from disk,
> scratching a page in memory" and major fault means "page has to
> be read from disk, but before that an existing page has to be written")
>
> The jiffies express how many CPU has been used on the program,
> they used to be 1/100 of second units, not sure whether that still
> holds.
>
> What is interesting here is:
>
>> 55'062'589'440 bytes of allocated virtual memory
>> 9'328'185'344 bytes of resident memory allocated
>
>
> The first number is just the sum of the processes' virtual memory
> size - they indicate that they want 55 GByte in toto, but this being
> just 'virtual', it's not a problem.
>
> The second number is the sum of the processes' effective memory size -
> 9 GByte of RAM... more than you actually have? (scratches head) That
> shouldn't be possible, except if some RAM is counted twice. Damn.
Well - I have 4 gigs and usually running 1/2 full so the 9g can't be
right. But maybe if we trace how it is wrong it might lead to the problem
>
> Here are my number (light charge on an dual XEON on RH ES4, only
> static serves, some PHP though but mostly Tomcat running the show):
>
> 87862 minor faults so far
> 9291 major faults so far
> 869 user time jiffies burnt so far
> 379 kernel time jiffies burnt so far
> 1271640064 bytes of allocated virtual memory
> 50085888 bytes of resident memory allocated
> 10 children active right now
>
> Acid test: ~ 5 MByte resident memory per child for me
> ~31 MByte resident memory per child for you
>
> This does not sound too unreasonable if there is a lot of Perl going on.
>
> We need more numbers...
>
ok - how do we get more numbers. And - the real problem is the memory
growth. I have to limit my MaxRequestsPerChild to 10 because if I change
it to 30 total system memory usage grows withing minutes swelling by a
full gig. I really need to solve this. Thanks for your help.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Serious Memory Leak Problem
Posted by Marc Perkel <ma...@perkel.com>.
David Tonhofer, m-plify S.A. wrote:
>
>
> --On Friday, October 21, 2005 11:16 AM -0700 Marc Perkel
> <ma...@perkel.com> wrote:
>
>>
>>
>> David Tonhofer, m-plify S.A. wrote:
>>
>>> I really don't know what the problem could be but let's
>>> start a discussion:
>>>
>>> 1) How many children are there?
>>> 2) What is the sum of the processes RSS size?
>>> 3) What is the sum of the processes VSIZE size?
>>>
>> Thanks for your perl script. Here's the results:
>>
>> 1029308 minor faults so far
>> 227 major faults so far
>> 1532 user time jiffies burnt so far
>> 25812 kernel time jiffies burnt so far
>> 55062589440 bytes of allocated virtual memory
>> 9328185344 bytes of resident memory allocated
>> 294 children active right now
>>
>> Excuse my ignorance but what does the mean?
>>
>> Thanks in advance
>
>
>
> Well,
>
> "minor faults"/"major faults" are about paging (if I remember
> correctly, a minor fault means "page has to be read from disk,
> scratching a page in memory" and major fault means "page has to
> be read from disk, but before that an existing page has to be written")
>
> The jiffies express how many CPU has been used on the program,
> they used to be 1/100 of second units, not sure whether that still
> holds.
>
> What is interesting here is:
>
>> 55'062'589'440 bytes of allocated virtual memory
>> 9'328'185'344 bytes of resident memory allocated
>
>
> The first number is just the sum of the processes' virtual memory
> size - they indicate that they want 55 GByte in toto, but this being
> just 'virtual', it's not a problem.
>
> The second number is the sum of the processes' effective memory size -
> 9 GByte of RAM... more than you actually have? (scratches head) That
> shouldn't be possible, except if some RAM is counted twice. Damn.
>
> Here are my number (light charge on an dual XEON on RH ES4, only
> static serves, some PHP though but mostly Tomcat running the show):
>
> 87862 minor faults so far
> 9291 major faults so far
> 869 user time jiffies burnt so far
> 379 kernel time jiffies burnt so far
> 1271640064 bytes of allocated virtual memory
> 50085888 bytes of resident memory allocated
> 10 children active right now
>
> Acid test: ~ 5 MByte resident memory per child for me
> ~31 MByte resident memory per child for you
>
> This does not sound too unreasonable if there is a lot of Perl going on.
>
> We need more numbers...
>
>
So - what should I test next to track this down? Also - if I set
MaxRequestsPerChild to 30 - it quickly fills up another gig of ram. It's
currently set at 10.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Serious Memory Leak Problem
Posted by "David Tonhofer, m-plify S.A." <d....@m-plify.com>.
--On Friday, October 21, 2005 11:16 AM -0700 Marc Perkel <ma...@perkel.com> wrote:
>
>
> David Tonhofer, m-plify S.A. wrote:
>
>> I really don't know what the problem could be but let's
>> start a discussion:
>>
>> 1) How many children are there?
>> 2) What is the sum of the processes RSS size?
>> 3) What is the sum of the processes VSIZE size?
>>
> Thanks for your perl script. Here's the results:
>
> 1029308 minor faults so far
> 227 major faults so far
> 1532 user time jiffies burnt so far
> 25812 kernel time jiffies burnt so far
> 55062589440 bytes of allocated virtual memory
> 9328185344 bytes of resident memory allocated
> 294 children active right now
>
> Excuse my ignorance but what does the mean?
>
> Thanks in advance
Well,
"minor faults"/"major faults" are about paging (if I remember
correctly, a minor fault means "page has to be read from disk,
scratching a page in memory" and major fault means "page has to
be read from disk, but before that an existing page has to be written")
The jiffies express how many CPU has been used on the program,
they used to be 1/100 of second units, not sure whether that still
holds.
What is interesting here is:
> 55'062'589'440 bytes of allocated virtual memory
> 9'328'185'344 bytes of resident memory allocated
The first number is just the sum of the processes' virtual memory
size - they indicate that they want 55 GByte in toto, but this being
just 'virtual', it's not a problem.
The second number is the sum of the processes' effective memory size -
9 GByte of RAM... more than you actually have? (scratches head) That
shouldn't be possible, except if some RAM is counted twice. Damn.
Here are my number (light charge on an dual XEON on RH ES4, only
static serves, some PHP though but mostly Tomcat running the show):
87862 minor faults so far
9291 major faults so far
869 user time jiffies burnt so far
379 kernel time jiffies burnt so far
1271640064 bytes of allocated virtual memory
50085888 bytes of resident memory allocated
10 children active right now
Acid test: ~ 5 MByte resident memory per child for me
~31 MByte resident memory per child for you
This does not sound too unreasonable if there is a lot of Perl going on.
We need more numbers...
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Serious Memory Leak Problem
Posted by Marc Perkel <ma...@perkel.com>.
David Tonhofer, m-plify S.A. wrote:
> I really don't know what the problem could be but let's
> start a discussion:
>
> 1) How many children are there?
> 2) What is the sum of the processes RSS size?
> 3) What is the sum of the processes VSIZE size?
>
Thanks for your perl script. Here's the results:
1029308 minor faults so far
227 major faults so far
1532 user time jiffies burnt so far
25812 kernel time jiffies burnt so far
55062589440 bytes of allocated virtual memory
9328185344 bytes of resident memory allocated
294 children active right now
Excuse my ignorance but what does the mean?
Thanks in advance
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Serious Memory Leak Problem
Posted by "David Tonhofer, m-plify S.A." <d....@m-plify.com>.
I really don't know what the problem could be but let's
start a discussion:
1) How many children are there?
2) What is the sum of the processes RSS size?
3) What is the sum of the processes VSIZE size?
In other words, run this:
#!/usr/bin/perl -w
###################################################################################
# Hashtable holding the result of /proc inspection
###################################################################################
%extendedStatus = ();
$extendedStatus{"minor_faults"} = 0; # Sum over minor faults
$extendedStatus{"major_faults"} = 0; # Sum over major faults
$extendedStatus{"utime"} = 0; # Sum over user time jiffies burnt
$extendedStatus{"stime"} = 0; # Sum over kernel time jiffies burnt
$extendedStatus{"vsize"} = 0; # Sum over virtual memory size in byte
$extendedStatus{"rss"} = 0; # Sum over resident set sizes in byte (orig. in pages = 4KB)
$extendedStatus{"children"} = 0; # Will receive number of apache children
###################################################################################
# Subroutine that calls syslog, then calls die
###################################################################################
sub croak {
my($msg) = @_;
die "$msg\n";
}
###################################################################################
# Query the /proc filesystem. Return 1 if all went well, 0 otherwise. If a major
# problem occurs, the problem is logged to syslog, then die is called.
# Upon successful return, all the obtained values have been filled into
# %extendedStatus.
###################################################################################
sub obtainFromProc {
my($server) = @_;
my($pidfile) = "/var/run/httpd.pid"; # System-specific
if (!(-f $pidfile)) {
croak("Could not find pidfile '$pidfile' - maybe the Apache server is not running?");
}
my($apachePid) = `cat $pidfile` or croak("Could not cat the pidfile '$pidfile': $!");
chomp($apachePid);
#
# Obtain mother process values, using the same method as for the children
#
handleChild($apachePid);
#
# Apache does not use a threading model; instead it has a number of children that are
# processes. Let's find all the processes that have as parent $pid and sum the required
# values over these. We get the PIDs of these processes using an appropriate pd line
#
if (!open(PS,"ps --no-headers -eo ppid,pid |")) {
croak(LOG_ERR,"Could not open pipe from 'ps'");
}
my(@ps) = <PS>;
if (!close(PS)) {
croak(LOG_ERR,"Could not close pipe from 'ps'");
}
my($line);
foreach $line (@ps) {
chomp($line);
if ($line =~ /^\s*$apachePid\s+(\d+)/) {
my($childPid) = $1;
$extendedStatus{"children"}++;
handleChild($childPid);
}
}
}
###################################################################################
# Process an apache child process.
# The values in %extendedStatus are updated
###################################################################################
sub handleChild {
my($childpid) = @_;
my($statfile) = "/proc/$childpid/stat";
open(STATS,$statfile) or return; # Could happen that the process dies before we can read
my($statline) = <STATS>;
close(STATS) or return; # Could happen that the process dies before we can read
chomp($statline);
# Decompose the line of 'stat' into an array of values
my(@statline) = ();
while ($statline =~ /^\s*(\S+)(.*)$/) {
push(@statline,$1);
$statline = $2;
}
# Extract and add
$extendedStatus{"vsize"} += $statline[22]; # Virtual memory size in byte
$extendedStatus{"rss"} += $statline[23]*4*1024; # Resident set size in byte (orig. in pages = 4KB)
$extendedStatus{"minor_faults"} += $statline[9];
$extendedStatus{"major_faults"} += $statline[11];
$extendedStatus{"utime"} += $statline[13];
$extendedStatus{"stime"} += $statline[14];
}
#
# Gather from /proc as needed, may die - or fills in %extendedStatus
#
obtainFromProc();
print $extendedStatus{"minor_faults"} . " minor faults so far\n";
print $extendedStatus{"major_faults"} . " major faults so far\n";
print $extendedStatus{"utime"} . " user time jiffies burnt so far\n";
print $extendedStatus{"stime"} . " kernel time jiffies burnt so far\n";
print $extendedStatus{"vsize"} . " bytes of allocated virtual memory\n";
print $extendedStatus{"rss"} . " bytes of resident memory allocated\n";
print $extendedStatus{"children"} . " children active right now\n";
--On Friday, October 21, 2005 9:19 AM -0700 Marc Perkel <ma...@perkel.com> wrote:
> I've asked about this before and never got an answer. I used to run my server on a dual xeon computer and it was very memory efficient. I moved to the 64 bit version of of Fedora Core 4 and now it's filling up memory really fast.
>
> Both servers had 4 gigs of ram. And basically the same configuration file using the same modules for both. It is especially bad where perl is involved.
>
> Now I have to set MaxRequestsPerChild to no more than 10 to keep ram from filling up. The server is rather busy serving 67 requests per second. If I set MaxRequestsPerChild to 30 - it quickly fills up another gig of ram.
>
> I really need to fix this. Sure could use some help on trying to figure out where the problem is.
>
> Thanks in advance.
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org