You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Kevin Slean <ke...@wctravel.com> on 2001/05/03 16:37:35 UTC
Apache Processes hanging
Mod_perlers,
I have a problem with Apache and was looking for your thoughts on my problem
or some additional mailing lists more focused on just Apache.
I have 70 httpd daemons running and some of them just appear to hang. As
time goes by, the number of hung processes increases such that there are no
working ones left to perform real work. Consequently transaction processing
performance drops eventually to zero.
Our web transactions running through these httpd daemons should not take
more than 60 seconds in a worst case scenario. Yet, some of these 'hung'
processes have been on the same transaction for over 30 hours. I
originally thought that this was a mod_perl problem and was buried in the
CGI/Perl routines performing the transactions. However, upon closer
inspection, I have found that these hanging processes are also running JAVA
servlets and gif gets. This makes me suspect that it is an Apache problem.
I ran truss on the hung processes and found that they fell into two camps.
The first group was sitting at a read system call. The second group was in
a loop with sigalarm going off every 10 seconds.
We are running the following:
Hardware: Sun Ultra-250
OS: Solaris 5.7 patch level 106541-12
Apache: Apache/1.3.11 (Unix) ApacheJServ/1.1.2 mod_perl/1.24
secured_by_Raven/1.4.2
Any ideas, thoughts, and comments are welcome. Thanks.
Kevin
Re: Apache Processes hanging
Posted by Paul Cotter <pr...@bellsouth.net>.
This answer has nothing to do with modperl - sorry. I have had the same
problem
on a Sybase database with a 'normal' application.
This situation can occur due to a database (b)locks, particularly if a
transaction is
composed of more than one update, or it fires a referential trigger which
has the same
effect.
The first question is what is the lock-level. Is it at the row or at the
page - I presume
the row. If the insert causes a 'fault' such that an index page becomes full
and has to
split then the whole index page will be locked regardless of row-level
locking. If the
second part of transaction is waiting on someone else then we can get the
deadly
embrace situation. However, it can normally be cleared on time-outs. If
however
you are acquiring these blocks faster then they time out, then in very short
order
you will be .. er.. screwed.
Sometimes it can help to do dirty reads if the data you need to present does
not need to be up to date.
One of the cures for this is an update pipe. Instead of each 'program' doing
the insert
on the database they funnel the updates to a single threading process. Reads
of
course can be be done by the individual 'programs'.
Can you confirm that the connection is freed if you kill the process that is
blocked?
If so this gives you another way out.
----- Original Message -----
From: "Robert Landrum" <rl...@capitoladvantage.com>
To: "Kevin Slean" <ke...@wctravel.com>; <mo...@apache.org>
Sent: Thursday, 03 May, 2001 03:40 PM
Subject: Re: Apache Processes hanging
> At 10:37 AM -0400 5/3/01, Kevin Slean wrote:
> >Mod_perlers,
> >
> >I have a problem with Apache and was looking for your thoughts on my
problem
> >or some additional mailing lists more focused on just Apache.
> >
> >I have 70 httpd daemons running and some of them just appear to hang. As
> >time goes by, the number of hung processes increases such that there are
no
> >working ones left to perform real work. Consequently transaction
processing
> >performance drops eventually to zero.
> >
> >Our web transactions running through these httpd daemons should not take
> >more than 60 seconds in a worst case scenario. Yet, some of these 'hung'
> >processes have been on the same transaction for over 30 hours. I
> >originally thought that this was a mod_perl problem and was buried in the
> >CGI/Perl routines performing the transactions. However, upon closer
> >inspection, I have found that these hanging processes are also running
JAVA
> >servlets and gif gets. This makes me suspect that it is an Apache
problem.
> >
> >I ran truss on the hung processes and found that they fell into two
camps.
> >The first group was sitting at a read system call. The second group was
in
> >a loop with sigalarm going off every 10 seconds.
> >
> I'm having similar problems, but we think it's directly related to
> Oracle. Basically, a connection is made to the Oracle database, a
> transaction is started and finished, but the connection to the
> database doesn't go away and the statement (at least from the oracle
> side) never seems to finish. The data is present in the database
> (these are insert statement, btw). Over time, every process collects
> one of these hanging statements and it eventually overwhelms our
> oracle database. The only solution is to restart apache every 5
> minutes to eliminate the built-up non-finished transactions.
>
> Has anyone seen this before?
>
> Rob
>
> --
> As soon as you make something foolproof, someone will create a better
fool.
>
RE: Apache Processes hanging
Posted by Rob Bloodgood <ro...@empire2.com>.
> I'm having similar problems, but we think it's directly related to
> Oracle. Basically, a connection is made to the Oracle database, a
> transaction is started and finished, but the connection to the
> database doesn't go away and the statement (at least from the oracle
> side) never seems to finish. The data is present in the database
> (these are insert statement, btw). Over time, every process collects
> one of these hanging statements and it eventually overwhelms our
> oracle database. The only solution is to restart apache every 5
> minutes to eliminate the built-up non-finished transactions.
Yeah... two things: CONCURRENCY and TRANSACTIONS.
Concurrency: Are there any other processes/reports/queries running at the
time of insert? That will lock ALL of them, waiting for the insert to
complete so the lock is released. Or, Another Interesting Way To Lock A
Really Buff Linux Server (tm).
Transactions: how's this one for fun? I started experimenting with
Apache::Session::Oracle to see what I could see. Usually I run w/
$dbh->{AutoCommit} = 1, which is the default, because most of the time I'm
just running SELECT's. But ::Oracle wouldn't ever complete the transaction,
hanging that server process and eventually most of the httpd system, all
waiting for the commit() on the INSERT (from the new Session) that doesn't
complete. <sigh> I ended up having to do a local block, with Commit => 1:
{
local $dbh->{AutoCommit} = 0;
tie %session, 'Apache::Session::Oracle', $session_id, { Handle => $dbh,
Commit => 1};
$session_id = $session{_session_id}; # save a copy
_set_cookie( $r, SESSION_COOKIE, $session{_session_id} );
$session{referer} ||= $referer; # preserve prior entries
untie %session;
}
HTH!
L8r,
Rob
Re: Apache Processes hanging
Posted by Robert Landrum <rl...@capitoladvantage.com>.
At 10:37 AM -0400 5/3/01, Kevin Slean wrote:
>Mod_perlers,
>
>I have a problem with Apache and was looking for your thoughts on my problem
>or some additional mailing lists more focused on just Apache.
>
>I have 70 httpd daemons running and some of them just appear to hang. As
>time goes by, the number of hung processes increases such that there are no
>working ones left to perform real work. Consequently transaction processing
>performance drops eventually to zero.
>
>Our web transactions running through these httpd daemons should not take
>more than 60 seconds in a worst case scenario. Yet, some of these 'hung'
>processes have been on the same transaction for over 30 hours. I
>originally thought that this was a mod_perl problem and was buried in the
>CGI/Perl routines performing the transactions. However, upon closer
>inspection, I have found that these hanging processes are also running JAVA
>servlets and gif gets. This makes me suspect that it is an Apache problem.
>
>I ran truss on the hung processes and found that they fell into two camps.
>The first group was sitting at a read system call. The second group was in
>a loop with sigalarm going off every 10 seconds.
>
>We are running the following:
>
> Hardware: Sun Ultra-250
> OS: Solaris 5.7 patch level 106541-12
> Apache: Apache/1.3.11 (Unix) ApacheJServ/1.1.2 mod_perl/1.24
> secured_by_Raven/1.4.2
>
>Any ideas, thoughts, and comments are welcome. Thanks.
>
>Kevin
I'm having similar problems, but we think it's directly related to
Oracle. Basically, a connection is made to the Oracle database, a
transaction is started and finished, but the connection to the
database doesn't go away and the statement (at least from the oracle
side) never seems to finish. The data is present in the database
(these are insert statement, btw). Over time, every process collects
one of these hanging statements and it eventually overwhelms our
oracle database. The only solution is to restart apache every 5
minutes to eliminate the built-up non-finished transactions.
Has anyone seen this before?
Rob
--
As soon as you make something foolproof, someone will create a better fool.
Re: Apache Processes hanging
Posted by Stas Bekman <st...@stason.org>.
On Thu, 3 May 2001, Kevin Slean wrote:
>
> Mod_perlers,
>
> I have a problem with Apache and was looking for your thoughts on my problem
> or some additional mailing lists more focused on just Apache.
>
> I have 70 httpd daemons running and some of them just appear to hang. As
> time goes by, the number of hung processes increases such that there are no
> working ones left to perform real work. Consequently transaction processing
> performance drops eventually to zero.
>
> Our web transactions running through these httpd daemons should not take
> more than 60 seconds in a worst case scenario. Yet, some of these 'hung'
> processes have been on the same transaction for over 30 hours. I
> originally thought that this was a mod_perl problem and was buried in the
> CGI/Perl routines performing the transactions. However, upon closer
> inspection, I have found that these hanging processes are also running JAVA
> servlets and gif gets. This makes me suspect that it is an Apache problem.
>
> I ran truss on the hung processes and found that they fell into two camps.
> The first group was sitting at a read system call. The second group was in
> a loop with sigalarm going off every 10 seconds.
>
> We are running the following:
>
> Hardware: Sun Ultra-250
> OS: Solaris 5.7 patch level 106541-12
> Apache: Apache/1.3.11 (Unix) ApacheJServ/1.1.2 mod_perl/1.24
> secured_by_Raven/1.4.2
>
> Any ideas, thoughts, and comments are welcome. Thanks.
Since you already truss()ed the processes and know that it's not a
mod_perl problem, you probably want to take the question to the httpd
mailing list. Meanwhile you can use Apache::Watchdog::RunAway to monitor
and kill the processes that hang.
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://apachetoday.com http://logilune.com/
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/