[users@httpd] Re: httpd is creating large number of shared memory segments.

["Chandranshu ." <ch...@gmail.com>]

Hi

We changed our script that used to do a graceful restart to also record the
number of shared memory segments before and after the restart. Plotting the
number of shm segments vs. the number of times the server was restarted
gracefully was almost a straight line.

Even doing a `apachectl -k restart` didn't solve the problem. Rather, the
shared memory count again went up by 1.

Doing an `apachectl -k stop` removed all shared memory segments.

On starting it again using `apachectl -k start`, the server created 3 shared
memory segments, two of which had status as dest. However, repeatedly doing
ipcs -m -i <shmid> even on these shm segments showed that httpd and php-cgi
processes were attaching and detaching these segments as well.

So, is this the expected behavior of the server? And does the apache server
needs to be periodically stopped and started in case of multiple graceful
restarts?

Thanks for your kind consideration,
Chandranshu


On Thu, Jun 25, 2009 at 5:43 PM, Chandranshu . <ch...@gmail.com>wrote:

> Hi
>
> We're running http-2.2.3 with preform MPM and host around 500 sites for
> many users. We always do a graceful restart of the server lest some users
> lose their session. The server has created a large number of shared memory
> segments. We have seen it creating as many as 4030 segments and this number
> consistently stays above 1500.
>
> Doing ipcs -m shows that many of these segments have been marked for
> destruction but are still attached to some processes. Doing a grep on the
> pid of the process that last attached the segment returns nothing. Doing it
> in a while loop revealed that there were numerous httpd processes being
> spawned that attached the shm segment for a brief time before vanishing.
>
> Why are there so many shared memory segments and are they not freed up by
> the main process?Also, could someone please guide me as to why are the
> processes dying so fast and replaced by new processes? I guess it has
> something to do with the following config:
> <IfModule prefork.c>
>   StartServers      2
>   MinSpareServers   10
>   MaxSpareServers   25
>   ServerLimit     1024
>   MaxClients      1024
>   MaxRequestsPerChild  4000
> </IfModule>
>
> Thanks and regards
> Chandranshu
>

Re: [users@httpd] Re: httpd is creating large number of shared memory segments.

[André Warnier <aw...@ice-sa.com>]

Chandranshu . wrote:
> Hi André
> 
> Our server is handling around 20 requests per second. Please note that this
> is not a single site. There are many sites that are hosted on this server.
> As requests to host new site keep coming in, we keep reloading the virtual
> host config through a graceful restart.
> 
> Here are my KeepAlive settings:
>    KeepAlive On
>    MaxKeepAliveRequests 100
>    KeepAliveTimeout 15
> 
> Unfortunately, I can't answer why the MaxRequestsPerChild was modified as it
> was done by the earlier sysad and there is no documentation left around by
> him. I have been reading up a few articles on tuning the apache performance
> and am planning to change them as per the requirements of the server.
> 
> Please let me know if there is any other config/statistics I can provide
> which will help in debugging the issue.
> 
There are 2 things I would do first of all :

1) try to get an idea of the time needed, on average, to process 1 
request.  You can get this in the access log, with the parameters 
explained here :
http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats
(%D parameter)
(While you are at it, also add the %k parameter, it will be useful)

2) change the following setting :
KeepAliveTimeout 15
to
KeepAliveTimeout 2

and look at the results.

Subjectively, I have the impression that the number of Apache processes 
(MaxServers 1024), is a total overkill for 20 requests/seconds, unless 
your requests take a very long time to be serviced.
But we'll see when there are some numbers.


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [users@httpd] Re: httpd is creating large number of shared memory segments.

["Chandranshu ." <ch...@gmail.com>]

Hi André

Our server is handling around 20 requests per second. Please note that this
is not a single site. There are many sites that are hosted on this server.
As requests to host new site keep coming in, we keep reloading the virtual
host config through a graceful restart.

Here are my KeepAlive settings:
   KeepAlive On
   MaxKeepAliveRequests 100
   KeepAliveTimeout 15

Unfortunately, I can't answer why the MaxRequestsPerChild was modified as it
was done by the earlier sysad and there is no documentation left around by
him. I have been reading up a few articles on tuning the apache performance
and am planning to change them as per the requirements of the server.

Please let me know if there is any other config/statistics I can provide
which will help in debugging the issue.

Thanks much for your time,
Chandranshu

On Fri, Jun 26, 2009 at 3:47 PM, André Warnier <aw...@ice-sa.com> wrote:

> Chandranshu . wrote:
> ...
> Just to help in getting an idea of your site :
> - how many requests is your site typically handling (per second, per
> minute..) ?
> - what is your KeepAlive setting ?
>
> and, why did you change the default for MaxRequestsPerChild ?
> any particular reason, or just for the sake of it ?
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>  "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>

Re: [users@httpd] Re: httpd is creating large number of shared memory segments.

[André Warnier <aw...@ice-sa.com>]

Chandranshu . wrote:
...
Just to help in getting an idea of your site :
- how many requests is your site typically handling (per second, per 
minute..) ?
- what is your KeepAlive setting ?

and, why did you change the default for MaxRequestsPerChild ?
any particular reason, or just for the sake of it ?

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [users@httpd] Re: httpd is creating large number of shared memory segments.

["Chandranshu ." <ch...@gmail.com>]

Hello Scott,

Many many thanks for your guidance. I followed the binary search among
modules and was quickly able to pinpoint mod_cband (
http://sourceforge.net/projects/cband/) as the reason of why the number of
shared memory segments keeps increasing.

Again, many thanks to you and everyone who helped me out.

Regards
Chandranshu

On Fri, Jun 26, 2009 at 8:36 PM, Scott Gifford <sg...@suspectclass.com>wrote:

> "Chandranshu ." <ch...@gmail.com> writes:
>
> > We changed our script that used to do a graceful restart to also
> > record the number of shared memory segments before and after the
> > restart. Plotting the number of shm segments vs. the number of times
> > the server was restarted gracefully was almost a straight line.
>
> Hello Chandranshu,
>
> When you do a graceful server restart, Apache asks the children to
> exit when it's convenient and starts up new children.  So it would not
> surprise me to see a short spike in resource consumption until the old
> child processes complete.  It would surprise me if it didn't return to
> normal after a few minutes, though, and you're seeing the increased
> shared memory segments until a server restart, right?  And you suspect
> it's a leak somewhere?
>
> A strategy for figuring out what's causing this would be to start off
> with a minimal Apache configuration.  You could create a new conf file
> with all modules disabled listening on another port and look for the
> leak there.  If you still see it, it must be in the Apache core;
> otherwise it's in a module.  Keep enabling modules until you start
> seeing it, then you know what module it is (you can also enable and
> disable groups of modules to do a sort of binary search).  If with the
> same set of modules as your production server you still don't see a
> leak, it must be some code running under your server, and you can use
> the same sort of process to bring in different pieces of code until
> you see the leak.
>
> Hope this helps, and good luck!
>
> ----Scott.
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>   "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>

Re: [users@httpd] Re: httpd is creating large number of shared memory segments.

[Scott Gifford <sg...@suspectclass.com>]

"Chandranshu ." <ch...@gmail.com> writes:

> We changed our script that used to do a graceful restart to also
> record the number of shared memory segments before and after the
> restart. Plotting the number of shm segments vs. the number of times
> the server was restarted gracefully was almost a straight line.

Hello Chandranshu,

When you do a graceful server restart, Apache asks the children to
exit when it's convenient and starts up new children.  So it would not
surprise me to see a short spike in resource consumption until the old
child processes complete.  It would surprise me if it didn't return to
normal after a few minutes, though, and you're seeing the increased
shared memory segments until a server restart, right?  And you suspect
it's a leak somewhere?

A strategy for figuring out what's causing this would be to start off
with a minimal Apache configuration.  You could create a new conf file
with all modules disabled listening on another port and look for the
leak there.  If you still see it, it must be in the Apache core;
otherwise it's in a module.  Keep enabling modules until you start
seeing it, then you know what module it is (you can also enable and
disable groups of modules to do a sort of binary search).  If with the
same set of modules as your production server you still don't see a
leak, it must be some code running under your server, and you can use
the same sort of process to bring in different pieces of code until
you see the leak.

Hope this helps, and good luck!

----Scott.

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org