You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Dorian Taylor <do...@foobarsystems.com> on 2005/05/09 20:22:56 UTC

[mp2] [PATCH] implement ap_save_brigade

no tests currently, but there's a module on cpan that uses it.

http://search.cpan.org/~DORIAN/Apache-TrapSubRequest-0.01/

Re: Decaying Performance under newer apache/mp versions

Posted by Ricardo Ramos Massaro <ri...@gmail.com>.
On 5/9/05, J.e. Turcotte <je...@emagery.com> wrote:
> Has anyone encountered similar problems?  Does anyone have a deeper
> understanding of the whole apache/mp/perl setup to just ~know~ what these
> symptoms are indicitive of?

I'm sorry if this is completely unrelated, I am somewhat of a mp newbie.

I've been recently bitten by some very similar problems on some
scripts (not mp handlers) where CGI parameters and cookies were
not being read correctly -- sometimes my script was seeing GET
and POST parameters completely unrelated to what the browser sent.
My scripts all looked like:

use strict;
use CGI;
my $cgi = new CGI();
sub process {
  my $x = $cgi->param('x');
  # ...
}
process();

Because the problem only appears when an apache process
is executing a script for the second time, it is very hard to track
(reducing the number of apache children to 1 is very helpful
for debugging). This problem and many possible solutions
are described in great detail in

http://perl.apache.org/docs/general/perl_reference/perl_reference.html#my___Scoped_Variable_in_Nested_Subroutines

To understand why it is related to scripting in mp, see

http://perl.apache.org/docs/general/perl_reference/perl_reference.html#When_You_Cannot_Get_Rid_of_The_Inner_Subroutine

I changed "my $cgi = ..." to "local our $cgi = ..." and everything
started to work magically.

Ricardo

Re: Decaying Performance under newer apache/mp versions

Posted by "J.e. Turcotte" <je...@emagery.com>.
here's our connect:
$n is defined earlier, a central module that contains secure informations

Apache::DBI->connect_on_init (
            $n->database, $n->DBI_username, $n->DBI_password,
            { AutoCommit => 1, 'RaiseError' => 1
            }
         );

and we connect to MySQL (4.1.7)... though there is a separate database (set 
of tables) for the in-dev content, it is served by the same machine that 
successfully works without incident on the public machine (which conntects 
the same way, as well, connect_on_init).


----- Original Message ----- 
From: "Gedanken" <ge...@io.com>
To: "J.e. Turcotte" <je...@emagery.com>
Cc: <mo...@perl.apache.org>
Sent: Monday, May 09, 2005 3:33 PM
Subject: Re: Decaying Performance under newer apache/mp versions


>
> What database are you connecting to, and how are you connecting to it?
>
> -- 
> gedanken 


Re: Decaying Performance under newer apache/mp versions

Posted by Gedanken <ge...@io.com>.
What database are you connecting to, and how are you connecting to it?

-- 
gedanken

Re: Decaying Performance under newer apache/mp versions

Posted by Perrin Harkins <pe...@elem.com>.
On Mon, 2005-05-09 at 15:22 -0400, J.e. Turcotte wrote:
> Our OS guys feel like it's gotta be a code 
> problem, while I feel that perhaps our apache and/or modperl installs on 
> these machines are not ideal, or that in the move from mp1 to mp2, some new 
> capacities in modperl2 are allowing a breakdown of otherwise good structures 
> under mp1.

If possible, I'd suggest you try to run your code on these machines with
mp1 as well, to determine if that is the source of your problems.
However, if you're doing lots of work with the Apache API and had to
change it to make things work under mp2, that won't be much help.

Problems of this nature usually result from one of these:

- A database handle that is opened prior to forking, or inadvertently
stashed in a closure so that when the database connection times out the
reconnection doesn't work.

- Some kind of socket or file access that is opened before forking and
then used simultaneously from multiple processes.

- An XS module with C code that gets flaky when used in a forking
scenario or in certain error conditions.

To track it down, I would suggest you add significant logging and/or
assertions to your application.  This would allow you to check for
various things (a valid database handle that responds to ping(), a valid
$r object, expected input, etc.) along the way, and see where things
fall apart when they do.  This will lead you to the problem.

You could also see what happens when you set a really short timeout on
connections in the database, to check if the problem involved
reconnecting.

- Perrin


Decaying Performance under newer apache/mp versions

Posted by "J.e. Turcotte" <je...@emagery.com>.
We have a pretty large apache/modperl setup on our original server that has 
worked quite well despite periods of heavy load... this setup is sitting on 
some older versions;

Now we are developing new replacement content for large parts of this site 
on a pair of development only servers which are trying to catch up by using 
higher versions of all three; apache, modperl, and perl.  These machines 
have very low load... no more than 4 or 5 people developing and testing, and 
rarely all at the same time.  However, we have noticed some pretty scary 
behavior on these machines, and feel that it is linked somehow to the newer 
versions... since in most cases, the new mp/perl modules we're writing are 
better made and more efficient than the old... but despite this, there is an 
intrinsic decay in BOTH of the dev machines that is not present under the 
public machine which takes the vast brunt of the labor without a problem.

The versions on the two dev machines are:
httpd-2.0.52-9, mod_perl-1.99_16-4, perl-5.8.5-12.1

At first, content works as expected on the dev machines... then, after a 
couple days or so, some of the error screens we have built in (like, 'we're 
sorry, the database is out of reach at the moment, please try again' and 
various 'you are not authorized to be here' screens) start creeping in, even 
when we can observe our database and warns from the code and see no 
problems... what is odd is that when these screens appear, there is NO HIT 
to either the error or access logs for the site.  This behavior continues to 
decay to the point where sometimes you have to click a link 2 or 3 times to 
get past the error screens, and submitted forms have great difficulty 
getting through.  After a bit of this, a rare case occurs where an 
ultimately unrelated screen appears ... a search page when you click to an 
administration link, and so on... no possible connected logic between the 
two, and yet it appears (though this is rare even under the worst conditions 
we've seen)... and then finally the system begins either to ignore the 
user's cookie, or sees some phantom cookie, and all pages that check it 
react as if it is wrong, no matter how many times you refresh.  By the time 
it does this, no action will cure it... logging out and in again 
(destroy,recreate cookie), no effect, still acts as if their either is none 
or it stores info other than what it physically contains... getting error 
screens 3/4s of the clicks, etc.

And what cures this technoschizophrenic server behavior?  A restart of 
apache (httpd) ... then everything is peachy keen again and the decay begins 
again, taking a day or two to accumulate.

We have an underlaying set of modules which interact with apache through 
modperl ($r), interpreting authentification, setting up higher level modules 
based on URI request, database-to-object, etc... then tons of upper level 
modules which are all the various URI sections of the site, using 
module-per-table data structures to be wholly OO, etc.  This is how it works 
on the older versioned but wholly behaving machine, and how we've continued 
to think on the newer machines...

Has anyone encountered similar problems?  Does anyone have a deeper 
understanding of the whole apache/mp/perl setup to just ~know~ what these 
symptoms are indicitive of?  I'm struggling since I'm more on the upper perl 
end and less an OS person.  Our OS guys feel like it's gotta be a code 
problem, while I feel that perhaps our apache and/or modperl installs on 
these machines are not ideal, or that in the move from mp1 to mp2, some new 
capacities in modperl2 are allowing a breakdown of otherwise good structures 
under mp1.

Any ideas?  suggestions?  ... heck, even psychic vibes?? 


Re: [mp2] [PATCH] implement ap_save_brigade

Posted by Stas Bekman <st...@stason.org>.
Dorian Taylor wrote:
>>I'm not getting very far with this, Dorian. Neither httpd-dev nor apr-dev 
>>are taking it anywhere. So at the moment ap_save_brigade is sort of an 
>>unstable API, so other developers think we shouldn't expose it in the core 
>>API. So at the moment you have two options:
> 
> 
> hm, this is unfortunate.

Feel free to jump into the thread I've started on both apr-dev and 
httpd-dev list and ask for this to happen.

>>1) write an XS code in your module which will add this code.
> 
> 
> as long as it's relatively straight-forward, i think i'd opt to do this.

May be Geoff's Apache-SSLLookup will help:
http://search.cpan.org/dist/Apache-SSLLookup/

>>2) rewrite the code not to use this API, instead use a scalar buffer or an 
>>array and append the data their flattening the brigades on each invocation.
> 
> 
> ap_save_brigade was unfortunately the only way i could come up with
> for intercepting the body of a response so that it doesn't get sent
> to the client, and that its contents may be used for other things.

Why? Take a look for example at t/filter/TestFilter/out_str_buffer.pm (I 
should rename it to out_bbs_buffer.pm)


-- 
__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://ticketmaster.com

Re: [mp2] [PATCH] implement ap_save_brigade

Posted by Dorian Taylor <do...@foobarsystems.com>.
> I'm not getting very far with this, Dorian. Neither httpd-dev nor apr-dev 
> are taking it anywhere. So at the moment ap_save_brigade is sort of an 
> unstable API, so other developers think we shouldn't expose it in the core 
> API. So at the moment you have two options:

hm, this is unfortunate.

> 1) write an XS code in your module which will add this code.

as long as it's relatively straight-forward, i think i'd opt to do this.

> 2) rewrite the code not to use this API, instead use a scalar buffer or an 
> array and append the data their flattening the brigades on each invocation.

ap_save_brigade was unfortunately the only way i could come up with
for intercepting the body of a response so that it doesn't get sent
to the client, and that its contents may be used for other things.

> Please let us know whether this works for you. Thanks.

i can get on this in a few days.

cheers

.d

Re: [mp2] [PATCH] implement ap_save_brigade

Posted by Stas Bekman <st...@stason.org>.
Stas Bekman wrote:
> Dorian Taylor wrote:
> 
>>> it'd be nice to run a benchmark. I wonder why ap_save_brigade was 
>>> marked as ! in the map file. Which normally means it's not going to 
>>> be exposed. Did you by chance look at the archives for possible 
>>> references to it?
>>
>>
>>
>> there's a mention of it in the APR::Bucket manpage. i couldn't find
>> anything else aside from a thread on dev@httpd.apache.org:
>>
>> http://www.mail-archive.com/dev@httpd.apache.org/msg22884.html
>>
>> and something else i wrote to this list asking about it.
>>
>> it's been labeled ! since its initial revision in svn though, apparently.
>>
>> perhaps dougm could shed some light?
>>
>> i came across the function via
>> http://www.outoforder.cc/projects/apache/mod_transform/ , where the
>> author uses it to retrieve arbitrary output from what may or may
>> not actually be a file on the filesystem. i've been using it to do
>> things like feeding subversion PROPFIND responses into xslt processors.
> 
> 
> It will work just the same when flattening the data on each request. I'm 
> not sure what happens when a filehandle bucket is setaside, but normal 
> buckets will be copied, that's why it'd be interesting to see a 
> benchmark whether it's a better idea to use that approach.
> 
> Joe, do you have any insights on this topic?

I'm not getting very far with this, Dorian. Neither httpd-dev nor apr-dev 
are taking it anywhere. So at the moment ap_save_brigade is sort of an 
unstable API, so other developers think we shouldn't expose it in the core 
API. So at the moment you have two options:

1) write an XS code in your module which will add this code.

2) rewrite the code not to use this API, instead use a scalar buffer or an 
array and append the data their flattening the brigades on each invocation.

Please let us know whether this works for you. Thanks.

-- 
__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://ticketmaster.com

Re: [mp2] [PATCH] implement ap_save_brigade

Posted by Stas Bekman <st...@stason.org>.
Dorian Taylor wrote:
>>it'd be nice to run a benchmark. I wonder why ap_save_brigade was marked 
>>as ! in the map file. Which normally means it's not going to be exposed. 
>>Did you by chance look at the archives for possible references to it?
> 
> 
> there's a mention of it in the APR::Bucket manpage. i couldn't find
> anything else aside from a thread on dev@httpd.apache.org:
> 
> http://www.mail-archive.com/dev@httpd.apache.org/msg22884.html
> 
> and something else i wrote to this list asking about it.
> 
> it's been labeled ! since its initial revision in svn though, apparently.
> 
> perhaps dougm could shed some light?
> 
> i came across the function via
> http://www.outoforder.cc/projects/apache/mod_transform/ , where the
> author uses it to retrieve arbitrary output from what may or may
> not actually be a file on the filesystem. i've been using it to do
> things like feeding subversion PROPFIND responses into xslt processors.

It will work just the same when flattening the data on each request. I'm 
not sure what happens when a filehandle bucket is setaside, but normal 
buckets will be copied, that's why it'd be interesting to see a benchmark 
whether it's a better idea to use that approach.

Joe, do you have any insights on this topic?

-- 
__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://ticketmaster.com

Re: [mp2] [PATCH] implement ap_save_brigade

Posted by Dorian Taylor <do...@foobarsystems.com>.
> it'd be nice to run a benchmark. I wonder why ap_save_brigade was marked 
> as ! in the map file. Which normally means it's not going to be exposed. 
> Did you by chance look at the archives for possible references to it?

there's a mention of it in the APR::Bucket manpage. i couldn't find
anything else aside from a thread on dev@httpd.apache.org:

http://www.mail-archive.com/dev@httpd.apache.org/msg22884.html

and something else i wrote to this list asking about it.

it's been labeled ! since its initial revision in svn though, apparently.

perhaps dougm could shed some light?

i came across the function via
http://www.outoforder.cc/projects/apache/mod_transform/ , where the
author uses it to retrieve arbitrary output from what may or may
not actually be a file on the filesystem. i've been using it to do
things like feeding subversion PROPFIND responses into xslt processors.

.dorian

Re: [mp2] [PATCH] implement ap_save_brigade

Posted by Stas Bekman <st...@stason.org>.
Dorian Taylor wrote:
>>How is that related?
> 
> 
> i think that's some cargo culting that i didn't clean up. that can
> be disregarded.

sure

>>So you think this approach will be faster than flattening bb on each 
>>filter invocation and concatenating the scalar?
> 
> 
> honestly i didn't give it any thought. when i wrote this i just
> wanted to make it go.

it'd be nice to run a benchmark. I wonder why ap_save_brigade was marked 
as ! in the map file. Which normally means it's not going to be exposed. 
Did you by chance look at the archives for possible references to it?

-- 
__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://ticketmaster.com

Re: [mp2] [PATCH] implement ap_save_brigade

Posted by Dorian Taylor <do...@foobarsystems.com>.
> How is that related?

i think that's some cargo culting that i didn't clean up. that can
be disregarded.

> So you think this approach will be faster than flattening bb on each 
> filter invocation and concatenating the scalar?

honestly i didn't give it any thought. when i wrote this i just
wanted to make it go.

> sub Apache::SubRequest::run_trapped {
>     my ($r, $sref) = @_;
> [...]
>     $bb->flatten(my $data);
>     $$sref = $data;
>     $rv;
> 
> why not make the API that does run_trapped($data) and save the copying of 
> what could be a huge data with:
> 
> sub Apache::SubRequest::run_trapped {
>     my $r = shift;
> [...]
>     $bb->flatten($_[1]);

yeah that makes significantly more sense. i'll update that soon.

.d

Re: [mp2] [PATCH] implement ap_save_brigade

Posted by Stas Bekman <st...@stason.org>.
Dorian Taylor wrote:
> no tests currently, but there's a module on cpan that uses it.
> 
> http://search.cpan.org/~DORIAN/Apache-TrapSubRequest-0.01/

Thanks Dorian, please see comments below.

> ------------------------------------------------------------------------
> 
> diff -ur mod_perl-2.0.0-RC4/lib/Apache/ParseSource.pm mod_perl-2.0.0-RC4-djt/lib/Apache/ParseSource.pm
> --- mod_perl-2.0.0-RC4/lib/Apache/ParseSource.pm	Fri Jan  7 01:17:50 2005
> +++ mod_perl-2.0.0-RC4-djt/lib/Apache/ParseSource.pm	Fri Feb 11 08:22:37 2005
> @@ -138,7 +138,7 @@
>                                      apr_optional mod_include mod_cgi
>                                      mod_proxy mod_ssl ssl_ apr_anylock
>                                      apr_rmm ap_config mod_log_config
> -                                    mod_perl modperl_ apreq);
> +                                    mod_perl modperl_ apreq mod_transform mod_.*svn);
>          $unwanted = qr|^$unwanted|;
>          my $wanted = '';

How is that related?

> diff -ur mod_perl-2.0.0-RC4/xs/Apache/Filter/Apache__Filter.h mod_perl-2.0.0-RC4-djt/xs/Apache/Filter/Apache__Filter.h
> --- mod_perl-2.0.0-RC4/xs/Apache/Filter/Apache__Filter.h	Fri Jan  7 01:17:40 2005
> +++ mod_perl-2.0.0-RC4-djt/xs/Apache/Filter/Apache__Filter.h	Thu Feb 10 20:33:58 2005
> @@ -365,3 +365,20 @@
>  
>      return rc;
>  }
> +
> +static MP_INLINE
> +apr_status_t mpxs_Apache__Filter_save_brigade(pTHX_ ap_filter_t *f,
> +                                              apr_bucket_brigade *saveto,
> +                                              apr_bucket_brigade *bb,
> +                                              apr_pool_t *p)
> +{
> +    apr_status_t rc = ap_save_brigade(f, &saveto, &bb, p);
> +    /* if users don't bother to check the success, do it on their
> +     * behalf */
> +    if (GIMME_V == G_VOID && rc != APR_SUCCESS) {
> +        modperl_croak(aTHX_ rc, "Apache::Filter::save_brigade");
> +    }
> +
> +    return rc;
> +}
> +
> diff -ur mod_perl-2.0.0-RC4/xs/maps/apache_functions.map mod_perl-2.0.0-RC4-djt/xs/maps/apache_functions.map
> --- mod_perl-2.0.0-RC4/xs/maps/apache_functions.map	Fri Jan  7 01:17:39 2005
> +++ mod_perl-2.0.0-RC4-djt/xs/maps/apache_functions.map	Thu Feb 10 22:59:18 2005
> @@ -245,7 +245,9 @@
>  !ap_register_output_filter
>  -ap_remove_output_filter
>  -ap_remove_input_filter
> -!ap_save_brigade
> +~ap_save_brigade
> + mpxs_Apache__Filter_save_brigade | | \
> +    f, saveto, bb, p=(f->r ? f->r->pool : NULL) | save_brigade
>   ap_filter_flush
>  ~ap_fflush
>   mpxs_Apache__Filter_fflush

So you think this approach will be faster than flattening bb on each 
filter invocation and concatenating the scalar?

sub Apache::SubRequest::run_trapped {
     my ($r, $sref) = @_;
[...]
     $bb->flatten(my $data);
     $$sref = $data;
     $rv;

why not make the API that does run_trapped($data) and save the copying of 
what could be a huge data with:

sub Apache::SubRequest::run_trapped {
     my $r = shift;
[...]
     $bb->flatten($_[1]);

-- 
__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://ticketmaster.com