Hung httpds process consuming 100% CPU

["Mark W. Shead" <mw...@gmail.com>]

Has anyone run into problems with httpds.prefork hanging and  
consuming 100% of a CPU?  This is on a 4 processor box.  Every few  
days a process will start consuming 100% of one of the available CPUs  
and just sit their indefinately. Once a few of the CPUs are running  
these hung processes, subversion becomes very slow and unusable.

Any suggestions?

OS: RHEL 3
SVN: 1.3.2
Apache: 2.0.54
Server: 4 processors

Mark Shead

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: Hung httpds process consuming 100% CPU

[Jerry McAllister <je...@msu.edu>]

On Mon, Apr 16, 2007 at 09:15:37AM -0500, Mark W. Shead wrote:

> Has anyone run into problems with httpds.prefork hanging and  
> consuming 100% of a CPU?  This is on a 4 processor box.  Every few  
> days a process will start consuming 100% of one of the available CPUs  
> and just sit their indefinately. Once a few of the CPUs are running  
> these hung processes, subversion becomes very slow and unusable.

I have had httpd.prefork hang and go nowhere on SuSE 10.1.   I 
nuked everything and did it all over (including installing susie)
and then used /usr/sbin/apache2ctl to start it and it worked fine and the
install from the bootup startup script now works too.   

Unfortunately after cleaning it up, I can't reproduce the problem
but I remember that when it failed, some part of starting mangled
the config files and, for example my line that sets ServerAdmin
  ServerAdmin jerrymc@xxx.xxx
would get broken in to two pieces and rewritten in the composite config
it creates on two separate lines as:
  ServerAdmin
  jerrymc@xxx.xxx

There may have been some other junk happening as well.
After repeatedly fixing it up several times and starting again
and seeing the same result, I just nuked everything - used 
a FreeBSD fixit CD to overwrite the hard disk and reinstalled SuSE
and everything.   You may ask why I didn't just use FreeBSD and
run from there.   I would have preferred that, but this system
was being installed according to someone elses requirements which
included running it on Enterprise SuSE.   I have never seen anything
like that on BSD.

> 
> Any suggestions?

I guess, given my experience, clean up the machine, reinstall and
then start Apache the first time using the apache2ctl to start it
manually.   Since I changed several things all at once, I can't
say which was what fixed it or cleaned up the problem.

////jerry

> 
> OS: RHEL 3
> SVN: 1.3.2
> Apache: 2.0.54
> Server: 4 processors
> 
> Mark Shead
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: Hung httpds process consuming 100% CPU

[Tony Butt <tj...@cea.com.au>]

Mark W. Shead wrote:
> Has anyone run into problems with httpds.prefork hanging and consuming 
> 100% of a CPU?  This is on a 4 processor box.  Every few days a 
> process will start consuming 100% of one of the available CPUs and 
> just sit their indefinately. Once a few of the CPUs are running these 
> hung processes, subversion becomes very slow and unusable.
>
> Any suggestions?
>
> OS: RHEL 3
> SVN: 1.3.2
> Apache: 2.0.54
> Server: 4 processors
>
> Mark Shead
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
We recently had that problem - our problem seemed to be mod_auth_ldap.

Our history is as follows:

We originally set up on SuSE Enterprise Linux 9 running apache 2.0.49, 
and built mod_auth_krb to authenticate against a Windows 2000 Domain Server.
All was good with this setup, except that mod_auth_krb really hammered 
the DNS when used with subversion, as it doesn't cache credentials, and 
the Kerberos configuration file used hostnames.
I changed the Kerberos configuration to use IP addresses, and we ran 
happily like that for about 2 years.

Then, our IT people 'upgraded' the Windows 2000 servers to Windows 2003, 
and mod_auth_krb failed (2003 is fussier about some details). When we 
got the credential setup correct, it worked fine for basic web access, 
but an order of magnitude slower for subversion access, which was 
unacceptable to us.

As an alternative, I managed to configure mod_auth_ldap and mod_ldap to 
authenticate against the 2003 servers for subversion, and all was well 
for a week or so.
When we built up some confidence, we integrated the mod_auth_ldap 
configuration into the rest of the web server (trac, websvn, TWiki, etc).
Soon after this, we started to notice the http process spinning, and 
consuming all available CPU. After undoing every change we could find 
(to no effect), and some frantic web research, I found some references 
to this relating to the mod_ldap caching of credentials. I disabled the 
caching by setting
  LDAPCacheEntries = 0
in the apache configuration. We have not seen the problem since.

Tony Butt
CEA Technologies
Canberra, Australia