Decaying Performance under newer apache/mp versions

["J.e. Turcotte" <je...@emagery.com>]

We have a pretty large apache/modperl setup on our original server that has 
worked quite well despite periods of heavy load... this setup is sitting on 
some older versions;

Now we are developing new replacement content for large parts of this site 
on a pair of development only servers which are trying to catch up by using 
higher versions of all three; apache, modperl, and perl.  These machines 
have very low load... no more than 4 or 5 people developing and testing, and 
rarely all at the same time.  However, we have noticed some pretty scary 
behavior on these machines, and feel that it is linked somehow to the newer 
versions... since in most cases, the new mp/perl modules we're writing are 
better made and more efficient than the old... but despite this, there is an 
intrinsic decay in BOTH of the dev machines that is not present under the 
public machine which takes the vast brunt of the labor without a problem.

The versions on the two dev machines are:
httpd-2.0.52-9, mod_perl-1.99_16-4, perl-5.8.5-12.1

At first, content works as expected on the dev machines... then, after a 
couple days or so, some of the error screens we have built in (like, 'we're 
sorry, the database is out of reach at the moment, please try again' and 
various 'you are not authorized to be here' screens) start creeping in, even 
when we can observe our database and warns from the code and see no 
problems... what is odd is that when these screens appear, there is NO HIT 
to either the error or access logs for the site.  This behavior continues to 
decay to the point where sometimes you have to click a link 2 or 3 times to 
get past the error screens, and submitted forms have great difficulty 
getting through.  After a bit of this, a rare case occurs where an 
ultimately unrelated screen appears ... a search page when you click to an 
administration link, and so on... no possible connected logic between the 
two, and yet it appears (though this is rare even under the worst conditions 
we've seen)... and then finally the system begins either to ignore the 
user's cookie, or sees some phantom cookie, and all pages that check it 
react as if it is wrong, no matter how many times you refresh.  By the time 
it does this, no action will cure it... logging out and in again 
(destroy,recreate cookie), no effect, still acts as if their either is none 
or it stores info other than what it physically contains... getting error 
screens 3/4s of the clicks, etc.

And what cures this technoschizophrenic server behavior?  A restart of 
apache (httpd) ... then everything is peachy keen again and the decay begins 
again, taking a day or two to accumulate.

We have an underlaying set of modules which interact with apache through 
modperl ($r), interpreting authentification, setting up higher level modules 
based on URI request, database-to-object, etc... then tons of upper level 
modules which are all the various URI sections of the site, using 
module-per-table data structures to be wholly OO, etc.  This is how it works 
on the older versioned but wholly behaving machine, and how we've continued 
to think on the newer machines...

Has anyone encountered similar problems?  Does anyone have a deeper 
understanding of the whole apache/mp/perl setup to just ~know~ what these 
symptoms are indicitive of?  I'm struggling since I'm more on the upper perl 
end and less an OS person.  Our OS guys feel like it's gotta be a code 
problem, while I feel that perhaps our apache and/or modperl installs on 
these machines are not ideal, or that in the move from mp1 to mp2, some new 
capacities in modperl2 are allowing a breakdown of otherwise good structures 
under mp1.

Any ideas?  suggestions?  ... heck, even psychic vibes?? 

Re: Decaying Performance under newer apache/mp versions

[Ricardo Ramos Massaro <ri...@gmail.com>]

On 5/9/05, J.e. Turcotte <je...@emagery.com> wrote:
> Has anyone encountered similar problems?  Does anyone have a deeper
> understanding of the whole apache/mp/perl setup to just ~know~ what these
> symptoms are indicitive of?

I'm sorry if this is completely unrelated, I am somewhat of a mp newbie.

I've been recently bitten by some very similar problems on some
scripts (not mp handlers) where CGI parameters and cookies were
not being read correctly -- sometimes my script was seeing GET
and POST parameters completely unrelated to what the browser sent.
My scripts all looked like:

use strict;
use CGI;
my $cgi = new CGI();
sub process {
  my $x = $cgi->param('x');
  # ...
}
process();

Because the problem only appears when an apache process
is executing a script for the second time, it is very hard to track
(reducing the number of apache children to 1 is very helpful
for debugging). This problem and many possible solutions
are described in great detail in

http://perl.apache.org/docs/general/perl_reference/perl_reference.html#my___Scoped_Variable_in_Nested_Subroutines

To understand why it is related to scripting in mp, see

http://perl.apache.org/docs/general/perl_reference/perl_reference.html#When_You_Cannot_Get_Rid_of_The_Inner_Subroutine

I changed "my $cgi = ..." to "local our $cgi = ..." and everything
started to work magically.

Ricardo

Re: Decaying Performance under newer apache/mp versions

["J.e. Turcotte" <je...@emagery.com>]

here's our connect:
$n is defined earlier, a central module that contains secure informations

Apache::DBI->connect_on_init (
            $n->database, $n->DBI_username, $n->DBI_password,
            { AutoCommit => 1, 'RaiseError' => 1
            }
         );

and we connect to MySQL (4.1.7)... though there is a separate database (set 
of tables) for the in-dev content, it is served by the same machine that 
successfully works without incident on the public machine (which conntects 
the same way, as well, connect_on_init).


----- Original Message ----- 
From: "Gedanken" <ge...@io.com>
To: "J.e. Turcotte" <je...@emagery.com>
Cc: <mo...@perl.apache.org>
Sent: Monday, May 09, 2005 3:33 PM
Subject: Re: Decaying Performance under newer apache/mp versions


>
> What database are you connecting to, and how are you connecting to it?
>
> -- 
> gedanken 

Re: Decaying Performance under newer apache/mp versions

[Gedanken <ge...@io.com>]

What database are you connecting to, and how are you connecting to it?

-- 
gedanken

Re: Decaying Performance under newer apache/mp versions

[Perrin Harkins <pe...@elem.com>]

On Mon, 2005-05-09 at 15:22 -0400, J.e. Turcotte wrote:
> Our OS guys feel like it's gotta be a code 
> problem, while I feel that perhaps our apache and/or modperl installs on 
> these machines are not ideal, or that in the move from mp1 to mp2, some new 
> capacities in modperl2 are allowing a breakdown of otherwise good structures 
> under mp1.

If possible, I'd suggest you try to run your code on these machines with
mp1 as well, to determine if that is the source of your problems.
However, if you're doing lots of work with the Apache API and had to
change it to make things work under mp2, that won't be much help.

Problems of this nature usually result from one of these:

- A database handle that is opened prior to forking, or inadvertently
stashed in a closure so that when the database connection times out the
reconnection doesn't work.

- Some kind of socket or file access that is opened before forking and
then used simultaneously from multiple processes.

- An XS module with C code that gets flaky when used in a forking
scenario or in certain error conditions.

To track it down, I would suggest you add significant logging and/or
assertions to your application.  This would allow you to check for
various things (a valid database handle that responds to ping(), a valid
$r object, expected input, etc.) along the way, and see where things
fall apart when they do.  This will lead you to the problem.

You could also see what happens when you set a really short timeout on
connections in the database, to check if the problem involved
reconnecting.

- Perrin