You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mod_python-dev@quetz.apache.org by Eric Brunson <br...@brunson.com> on 2006/12/10 17:21:30 UTC
Apache log messages
I'm seeing a lot of these messages in the httpd error logs:
[Sun Dec 10 06:46:50 2006] [notice] mod_python: Creating 8 session
mutexes based on 150 max processes and 0 max threads.
[Sun Dec 10 06:46:50 2006] [notice] mod_python: using mutex_directory /tmp
[Sun Dec 10 06:46:50 2006] [error] (17)File exists: mod_python: Failed
to create global mutex 1 of 8 (/tmp/mpmtx6661).
Configuration Failed
We're having some severe problems on the server and mod_python is
getting blamed for it because of these logs. I'm going through the
source code right now to get a better understanding, but i thought
someone may be able to shed a little light.
Thanks,
e.
Re: Apache log messages
Posted by Jim Gallacher <jp...@jgassociates.ca>.
Jorey Bump wrote:
> I hate it when a module source distribution presumes that it can edit
> httpd.conf or insert itself in the loading process automatically. I
> understand the desire to be helpful, but I like to control this myself.
> PHP does this, so I must backup httpd.conf and restore it afterwards.
>
> AFAIK, mod_python doesn't do this, but I only use it on Linux. Does the
> install procedure insert the module for other platforms? We should avoid
> this.
For the stock source distribution "make install" doesn't touch the
Apache configuration files. Instead it just outputs the message
"Now don't forget to edit your main config and add"
"LoadModule python_module $(LIBEXECDIR)/mod_python.so"
"and if your configuration uses ClearModuleList, then also"
"AddModule mod_python.c"
I don't know if Nicolas has some other magic in his Windows binary
installers however.
Jim
Re: Apache log messages
Posted by Jeff Hinrichs - DM&T <je...@dundeemt.com>.
On 12/10/06, Eric Brunson <br...@brunson.com> wrote:
> Jorey Bump wrote:
> > Eric: I've CC'd the list with this response in case it proves helpful
> > to other FreeBSD users. I hope you don't mind.
> >
> > Eric Brunson wrote:
> >> Jorey Bump wrote:
> >>
> >>> Are you loading the module twice?
> >>
> >> Good point, I just checked to make sure it wasn't loading twice, and it
> >> is a problem on FreeBSD. The default install procedure (or maybe it's
> >> just the way I do things, but anyway) puts a python specific boot script
> >> in /usr/local/etc/apache/Include/python.conf, but the freebsd packages
> >> edit the LoadModule directives directly into the httpd.conf. So if you
> >> install from source, then try to install a ports package later, you end
> >> up loading the module twice.
> >
> > I hate it when a module source distribution presumes that it can edit
> > httpd.conf or insert itself in the loading process automatically. I
> > understand the desire to be helpful, but I like to control this
> > myself. PHP does this, so I must backup httpd.conf and restore it
> > afterwards.
> >
> > AFAIK, mod_python doesn't do this, but I only use it on Linux. Does
> > the install procedure insert the module for other platforms? We should
> > avoid this.
>
> The edits are done by the FreeBSD ports package when you install their
> binary build of mod_python, not the mod_python installation itself.
>
> >
> > FWIW, I like to control all local customizations in a local.conf file
> > included at the end of httpd.conf, where I selectively include
> > additional configuration files in the order I want. A similar approach
> > has been adopted in the default Apache 2.2 installation, and I've been
> > doing it since 1.3. As it's difficult to predict the scheme used by an
> > administrator, we should make it a point not to interfere. Of course,
> > package maintainers will be forced to support any 'enhancements'
> > developed for the target distribution.
> >
>
>
If you are running a server on FreeBSD, ports are your friend. Don't
use packages, imho, they are preconfigured and work best for desktop
type installs not servers. If you haven't already -- man portsnap and
the portupgrade/install utilities. (I only install mod_python from
source for testing the beta and reporting and that is on a dev machine
) I'm cutting this short here as this is a FreeBSD thing and not a
specific mp issue. Checkout the mailing lists and the manuals are
your best friend (best docs around)
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/index.html
--
Jeff Hinrichs
Re: Apache log messages
Posted by Eric Brunson <br...@brunson.com>.
Jorey Bump wrote:
> Eric: I've CC'd the list with this response in case it proves helpful
> to other FreeBSD users. I hope you don't mind.
>
> Eric Brunson wrote:
>> Jorey Bump wrote:
>>
>>> Are you loading the module twice?
>>
>> Good point, I just checked to make sure it wasn't loading twice, and it
>> is a problem on FreeBSD. The default install procedure (or maybe it's
>> just the way I do things, but anyway) puts a python specific boot script
>> in /usr/local/etc/apache/Include/python.conf, but the freebsd packages
>> edit the LoadModule directives directly into the httpd.conf. So if you
>> install from source, then try to install a ports package later, you end
>> up loading the module twice.
>
> I hate it when a module source distribution presumes that it can edit
> httpd.conf or insert itself in the loading process automatically. I
> understand the desire to be helpful, but I like to control this
> myself. PHP does this, so I must backup httpd.conf and restore it
> afterwards.
>
> AFAIK, mod_python doesn't do this, but I only use it on Linux. Does
> the install procedure insert the module for other platforms? We should
> avoid this.
The edits are done by the FreeBSD ports package when you install their
binary build of mod_python, not the mod_python installation itself.
>
> FWIW, I like to control all local customizations in a local.conf file
> included at the end of httpd.conf, where I selectively include
> additional configuration files in the order I want. A similar approach
> has been adopted in the default Apache 2.2 installation, and I've been
> doing it since 1.3. As it's difficult to predict the scheme used by an
> administrator, we should make it a point not to interfere. Of course,
> package maintainers will be forced to support any 'enhancements'
> developed for the target distribution.
>
Re: Apache log messages
Posted by Jorey Bump <li...@joreybump.com>.
Eric: I've CC'd the list with this response in case it proves helpful to
other FreeBSD users. I hope you don't mind.
Eric Brunson wrote:
> Jorey Bump wrote:
>
>> Are you loading the module twice?
>
> Good point, I just checked to make sure it wasn't loading twice, and it
> is a problem on FreeBSD. The default install procedure (or maybe it's
> just the way I do things, but anyway) puts a python specific boot script
> in /usr/local/etc/apache/Include/python.conf, but the freebsd packages
> edit the LoadModule directives directly into the httpd.conf. So if you
> install from source, then try to install a ports package later, you end
> up loading the module twice.
I hate it when a module source distribution presumes that it can edit
httpd.conf or insert itself in the loading process automatically. I
understand the desire to be helpful, but I like to control this myself.
PHP does this, so I must backup httpd.conf and restore it afterwards.
AFAIK, mod_python doesn't do this, but I only use it on Linux. Does the
install procedure insert the module for other platforms? We should avoid
this.
FWIW, I like to control all local customizations in a local.conf file
included at the end of httpd.conf, where I selectively include
additional configuration files in the order I want. A similar approach
has been adopted in the default Apache 2.2 installation, and I've been
doing it since 1.3. As it's difficult to predict the scheme used by an
administrator, we should make it a point not to interfere. Of course,
package maintainers will be forced to support any 'enhancements'
developed for the target distribution.
Re: Apache log messages
Posted by Eric Brunson <br...@brunson.com>.
Jorey Bump wrote:
> Eric Brunson wrote:
>> I'm seeing a lot of these messages in the httpd error logs:
>>
>> [Sun Dec 10 06:46:50 2006] [notice] mod_python: Creating 8 session
>> mutexes based on 150 max processes and 0 max threads.
>> [Sun Dec 10 06:46:50 2006] [notice] mod_python: using mutex_directory
>> /tmp
>> [Sun Dec 10 06:46:50 2006] [error] (17)File exists: mod_python: Failed
>> to create global mutex 1 of 8 (/tmp/mpmtx6661).
>> Configuration Failed
>>
>>
>> We're having some severe problems on the server and mod_python is
>> getting blamed for it because of these logs. I'm going through the
>> source code right now to get a better understanding, but i thought
>> someone may be able to shed a little light.
>
> Are you loading the module twice? Try running:
>
> apachectl configtest
>
> If that turns up nothing, I'd stop apache and remove any /tmp/mpm*
> files created by the user apache runs as, then try again. And then
> restart apache once more, to see if the problem persists.
>
Good point, I just checked to make sure it wasn't loading twice, and it
is a problem on FreeBSD. The default install procedure (or maybe it's
just the way I do things, but anyway) puts a python specific boot script
in /usr/local/etc/apache/Include/python.conf, but the freebsd packages
edit the LoadModule directives directly into the httpd.conf. So if you
install from source, then try to install a ports package later, you end
up loading the module twice.
Thanks for the input,
e.
Re: Apache log messages
Posted by Jorey Bump <li...@joreybump.com>.
Eric Brunson wrote:
> I'm seeing a lot of these messages in the httpd error logs:
>
> [Sun Dec 10 06:46:50 2006] [notice] mod_python: Creating 8 session
> mutexes based on 150 max processes and 0 max threads.
> [Sun Dec 10 06:46:50 2006] [notice] mod_python: using mutex_directory /tmp
> [Sun Dec 10 06:46:50 2006] [error] (17)File exists: mod_python: Failed
> to create global mutex 1 of 8 (/tmp/mpmtx6661).
> Configuration Failed
>
>
> We're having some severe problems on the server and mod_python is
> getting blamed for it because of these logs. I'm going through the
> source code right now to get a better understanding, but i thought
> someone may be able to shed a little light.
Are you loading the module twice? Try running:
apachectl configtest
If that turns up nothing, I'd stop apache and remove any /tmp/mpm* files
created by the user apache runs as, then try again. And then restart
apache once more, to see if the problem persists.
Re: Apache log messages
Posted by Eric Brunson <br...@brunson.com>.
Jim Gallacher wrote:
> Eric Brunson wrote:
>> Platform: FreeBSD 6.1, Apache 2.0.55 (mpm_prefork), Python 2.4.3,
>> mod_python 3.3 (20061130232253 snapshot).
>>
>> Well, I see the problem is that httpd is getting started up close enough
>> to the same time every boot cycle that the worker processes are getting
>> the same semaphore file names. For some reason FreeBSD doesn't clean
>> out /tmp on boot by default. I can't seem to find any cleanup mechanism
>> for the files on shutdown, but the 80 or so mutex files seem to
>> correspond to server crashes
>
> Yep. As a rule that is were the mutexes get consumed. I'm sure I don't
> need to tell you the ultimate solution is to figure out why apache is
> crashing and make it stop. :)
It's actually the server server that's crashing, not the apache server.
We're trying to get some new hardware on deck to make sure it's not
physical before we do too much digging into FreeBSD. We kind of had it
foisted upon is (more secure, blah, blah, blah, better networking, blah,
blah, you know the arguments), but no one here actually knows as much
about FreeBSD as the rest of us know about Linux, so we end up
scratching our heads and staring a lot when it comes to some of these
problems.
>
> My notes on this issue follow. They apply to Linux and may not be of
> any use on FreeBSD.
I've definitely hit that one on Linux before. In fact, I just had a
support call for one of our linux boxes and that's the first thing I
thought of. It turned out to be a missing log directory, but still. I
always have to Google for that ipcs fix below, I think I'll just store
your email away for future reference. :-)
> -------------------------------------------------------------------------
>
> Under some conditions Apache may crap out and the semaphores it
> reserved for the mod_python mutexes may not have been released. This
> may happen if it does not cleanly shutdown due to a hung mod_python
> instance.
>
> Each time apache is restarted it will grab additional semaphores until
> the server is starved for this resource.
>
> Check the number of semaphores available:
>
> # sysctl kernel.sem
> kernel.sem = 250 32000 32 128
>
>
> Get information about the semaphores
> # ipcs -s
>
> ------ Semaphore Arrays --------
> key semid owner perms nsems
> 0x0052e2c1 0 postgres 600 17
> 0x0052e2c2 32769 postgres 600 17
> 0x0052e2c3 65538 postgres 600 17
> 0x0052e2c4 98307 postgres 600 17
> 0x0052e2c5 131076 postgres 600 17
> 0x0052e2c6 163845 postgres 600 17
> 0x0052e2c7 196614 postgres 600 17
> 0x00000000 3145735 www-data 600 1
> 0x00000000 3178504 www-data 600 1
> 0x00000000 3211273 www-data 600 1
> 0x00000000 3244042 www-data 600 1
> 0x00000000 3276811 www-data 600 1
> 0x00000000 3309580 www-data 600 1
> 0x00000000 3342349 www-data 600 1
> 0x00000000 3375118 www-data 600 1
> 0x00000000 3407887 www-data 600 1
> 0x00000000 3440656 www-data 600 1
>
>
> The solution is to stop apache and clean out the dead semaphores,
> substituting the the correct apache user name for apache-user. Make
> sure apache is stopped before running this - we only want to remove
> the dead semaphores. On debian apache is run as the "www-data" user.
>
> ~localhost# ipcs -s | awk '/www-data/ { system("ipcrm sem " $2) }'
>
> Another one liner using perl to do the cleanup:
>
> ~localhost# ipcs -s | perl -ane '/^0x00000000/ && `ipcrm -s $F[1]`'
>
> Jim
Re: Apache log messages
Posted by Jim Gallacher <jp...@jgassociates.ca>.
Eric Brunson wrote:
> Platform: FreeBSD 6.1, Apache 2.0.55 (mpm_prefork), Python 2.4.3,
> mod_python 3.3 (20061130232253 snapshot).
>
> Well, I see the problem is that httpd is getting started up close enough
> to the same time every boot cycle that the worker processes are getting
> the same semaphore file names. For some reason FreeBSD doesn't clean
> out /tmp on boot by default. I can't seem to find any cleanup mechanism
> for the files on shutdown, but the 80 or so mutex files seem to
> correspond to server crashes
Yep. As a rule that is were the mutexes get consumed. I'm sure I don't
need to tell you the ultimate solution is to figure out why apache is
crashing and make it stop. :)
My notes on this issue follow. They apply to Linux and may not be of any
use on FreeBSD.
-------------------------------------------------------------------------
Under some conditions Apache may crap out and the semaphores it reserved
for the mod_python mutexes may not have been released. This may happen
if it does not cleanly shutdown due to a hung mod_python instance.
Each time apache is restarted it will grab additional semaphores until
the server is starved for this resource.
Check the number of semaphores available:
# sysctl kernel.sem
kernel.sem = 250 32000 32 128
Get information about the semaphores
# ipcs -s
------ Semaphore Arrays --------
key semid owner perms nsems
0x0052e2c1 0 postgres 600 17
0x0052e2c2 32769 postgres 600 17
0x0052e2c3 65538 postgres 600 17
0x0052e2c4 98307 postgres 600 17
0x0052e2c5 131076 postgres 600 17
0x0052e2c6 163845 postgres 600 17
0x0052e2c7 196614 postgres 600 17
0x00000000 3145735 www-data 600 1
0x00000000 3178504 www-data 600 1
0x00000000 3211273 www-data 600 1
0x00000000 3244042 www-data 600 1
0x00000000 3276811 www-data 600 1
0x00000000 3309580 www-data 600 1
0x00000000 3342349 www-data 600 1
0x00000000 3375118 www-data 600 1
0x00000000 3407887 www-data 600 1
0x00000000 3440656 www-data 600 1
The solution is to stop apache and clean out the dead semaphores,
substituting the the correct apache user name for apache-user. Make sure
apache is stopped before running this - we only want to remove the dead
semaphores. On debian apache is run as the "www-data" user.
~localhost# ipcs -s | awk '/www-data/ { system("ipcrm sem " $2) }'
Another one liner using perl to do the cleanup:
~localhost# ipcs -s | perl -ane '/^0x00000000/ && `ipcrm -s $F[1]`'
Jim
Re: Apache log messages
Posted by Eric Brunson <br...@brunson.com>.
Platform: FreeBSD 6.1, Apache 2.0.55 (mpm_prefork), Python 2.4.3,
mod_python 3.3 (20061130232253 snapshot).
Well, I see the problem is that httpd is getting started up close enough
to the same time every boot cycle that the worker processes are getting
the same semaphore file names. For some reason FreeBSD doesn't clean
out /tmp on boot by default. I can't seem to find any cleanup mechanism
for the files on shutdown, but the 80 or so mutex files seem to
correspond to server crashes and stopping and starting apache normally
doesn't result in orphaned files.
Sorry to bug you all, the answers were all there in the source.
Eric Brunson wrote:
> I'm seeing a lot of these messages in the httpd error logs:
>
> [Sun Dec 10 06:46:50 2006] [notice] mod_python: Creating 8 session
> mutexes based on 150 max processes and 0 max threads.
> [Sun Dec 10 06:46:50 2006] [notice] mod_python: using mutex_directory /tmp
> [Sun Dec 10 06:46:50 2006] [error] (17)File exists: mod_python: Failed
> to create global mutex 1 of 8 (/tmp/mpmtx6661).
> Configuration Failed
>
>
> We're having some severe problems on the server and mod_python is
> getting blamed for it because of these logs. I'm going through the
> source code right now to get a better understanding, but i thought
> someone may be able to shed a little light.
>
> Thanks,
> e.
>
>
>