You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by ad...@utoronto.ca on 2008/07/10 07:28:18 UTC
some flawed benchmarks
A couple of months ago i was going through slides from gozers "From
CGI to mod_perl 2.0, Fast!" talk, which has some benchmarks comparing
CGI, perlrun and registry to each other. At which point i realized
that i've never really known how much faster using straight handlers
is than using one of the CGI emulation layers. I also didn't have any
idea how much faster SetHandler modperl was vs SetHandler perl-script.
So i decided to see what i could figure out. I took gozers CGI from
the slides (slightly modified) and ran it through the paces on my
laptop, then converted the script to run as a straight handler.
here's the CGI version:
#!/usr/bin/perl
print qq[Content-Type: text/html\r\n\r\n];
print(qq[
<html><body>
<h1>Hello Worlds</h1>
<pre>
GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}
MOD_PERL: $ENV{MOD_PERL}
</pre>
</body></html>
]);
Here's the Handler version
package Kabob::HelloWorld;
use strict;
use warnings;
use Apache2::RequestRec ();
use Apache2::Const -compile =>qw(:common);
sub handler {
my $r = shift;
$r->content_type('text/html');
$r->print(qq[
<html><body>
<h1>Hello Worlds</h1>
<pre>
GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}
MOD_PERL: $ENV{MOD_PERL}
</pre>
</body></html>
]);
return Apache2::Const::OK;
}
1;
and here's the conf (these tests were all running through a light
mod_proxy front end too)
<Location /cgi/>
<IfDefine FrontEnd>
ProxyPass http://localhost:8080/cgi/
ProxyPassReverse http://localhost:8080/cgi/
</IfDefine>
</Location>
<IfDefine BackEnd>
ScriptAlias /cgi/ /www/p/
</IfDefine>
Alias /perlrun/ /www/p/
<Location /perlrun/>
<IfDefine FrontEnd>
ProxyPass http://localhost:8080/perlrun/
ProxyPassReverse http://localhost:8080/perlrun/
</IfDefine>
<IfDefine BackEnd>
SetHandler perl-script
PerlHandler ModPerl::PerlRun
Options +ExecCGI
PerlSendHeader On
</IfDefine>
</Location>
Alias /registry/ /www/p/
<Location /registry/>
<IfDefine FrontEnd>
ProxyPass http://localhost:8080/registry/
ProxyPassReverse http://localhost:8080/registry/
</IfDefine>
<IfDefine BackEnd>
SetHandler perl-script
PerlHandler ModPerl::Registry
Options +ExecCGI
PerlSendHeader On
</IfDefine>
</Location>
<Location /perlscript/>
<IfDefine FrontEnd>
ProxyPass http://localhost:8080/perlscript/
ProxyPassReverse http://localhost:8080/perlscript/
</IfDefine>
<IfDefine BackEnd>
SetHandler perl-script
PerlResponseHandler Kabob::HelloWorld
</IfDefine>
</Location>
<Location /modperl/>
<IfDefine FrontEnd>
ProxyPass http://localhost:8080/modperl/
ProxyPassReverse http://localhost:8080/modperl/
</IfDefine>
<IfDefine BackEnd>
SetHandler modperl
PerlResponseHandler Kabob::HelloWorld
</IfDefine>
</Location>
and here's the results (which are no doubt flawed for a number of reasons)
running: ab -n 10000 [url]
CGI
Requests per second: 217.80 [#/sec] (mean)
Time per request: 4.591 [ms] (mean)
Transfer rate: 53.17 [Kbytes/sec] received
PerlRun
Requests per second: 482.49 [#/sec] (mean)
Time per request: 2.073 [ms] (mean)
Transfer rate: 114.49 [Kbytes/sec] received
Registry
Requests per second: 693.33 [#/sec] (mean)
Time per request: 1.442 [ms] (mean)
Transfer rate: 164.53 [Kbytes/sec] received
SetHandler perl-script
Requests per second: 772.12 [#/sec] (mean)
Time per request: 1.295 [ms] (mean)
Transfer rate: 189.94 [Kbytes/sec] received
SetHandler modperl
Requests per second: 1048.66 [#/sec] (mean)
Time per request: 0.954 [ms] (mean)
Transfer rate: 250.84 [Kbytes/sec] received
I'm not sure how well you can really compare the CGI emulation numbers
to the PerlHandler numbers, but personally i think the 30%ish
improvement from perl-script to modperl is pretty amazing. I wouldn't
have imagined it would have been that high.
Adam
Re: some flawed benchmarks
Posted by ad...@utoronto.ca.
Quoting Perrin Harkins <pe...@elem.com>:
> Note that CGI and FastCGI don't need the proxy frontend.
The only reason I did it that way was because that's how apache was
already set up on my laptop, and i didn't feel like dorking around
with it too much. I could certainly change it around so that CGI and
FGCGI (if added) were handled by the light apache. I wasn't sure how
apples to oranges that would be, but i guess in reality, people that
are running plain CGI or fast CGI probably aren't doing it on a
mod_perl enabled server.
> It is pretty cool that it works so well. I feel like I should point
> out though, for the benefit of those using Registry, that the
> difference is only this big because the code isn't doing anything. In
> a real-world scenario you'd see an improvement, but nothing close to
> 30%.
It's probably also worth noting that under sethandler modperl the
GATEWAY_INTERFACE env variable has no value, which results in the page
being 3 bytes shorter (iirc), which is actually a pretty big deal
since the whole page is pretty short. The examples should likely be
modified to actually return pages that are the same length.
anything else stupid i'm missing?
adam
Re: some flawed benchmarks
Posted by Perrin Harkins <pe...@elem.com>.
On Thu, Jul 10, 2008 at 1:28 AM, <ad...@utoronto.ca> wrote:
> and here's the conf (these tests were all running through a light mod_proxy
> front end too)
Note that CGI and FastCGI don't need the proxy frontend.
> I'm not sure how well you can really compare the CGI emulation numbers to
> the PerlHandler numbers, but personally i think the 30%ish improvement from
> perl-script to modperl is pretty amazing. I wouldn't have imagined it would
> have been that high.
It is pretty cool that it works so well. I feel like I should point
out though, for the benefit of those using Registry, that the
difference is only this big because the code isn't doing anything. In
a real-world scenario you'd see an improvement, but nothing close to
30%.
- Perrin
Re: some flawed benchmarks
Posted by Dodger <el...@gmail.com>.
I appreciate this, as I'd been wondering.
But it also prompts me to.. I gotta ask...
No offence but...
Don't you know what a here_doc is?
--
Dodger
2008/7/9 <ad...@utoronto.ca>:
> A couple of months ago i was going through slides from gozers "From CGI to
> mod_perl 2.0, Fast!" talk, which has some benchmarks comparing CGI, perlrun
> and registry to each other. At which point i realized that i've never
> really known how much faster using straight handlers is than using one of
> the CGI emulation layers. I also didn't have any idea how much faster
> SetHandler modperl was vs SetHandler perl-script.
>
> So i decided to see what i could figure out. I took gozers CGI from the
> slides (slightly modified) and ran it through the paces on my laptop, then
> converted the script to run as a straight handler.
>
> here's the CGI version:
>
> #!/usr/bin/perl
>
>
> print qq[Content-Type: text/html\r\n\r\n];
>
> print(qq[
> <html><body>
> <h1>Hello Worlds</h1>
> <pre>
> GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}
> MOD_PERL: $ENV{MOD_PERL}
> </pre>
> </body></html>
>
> ]);
>
>
> Here's the Handler version
>
> package Kabob::HelloWorld;
>
> use strict;
> use warnings;
>
> use Apache2::RequestRec ();
>
> use Apache2::Const -compile =>qw(:common);
>
> sub handler {
> my $r = shift;
>
> $r->content_type('text/html');
> $r->print(qq[
> <html><body>
> <h1>Hello Worlds</h1>
> <pre>
> GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}
> MOD_PERL: $ENV{MOD_PERL}
> </pre>
> </body></html>
>
>
> ]);
>
> return Apache2::Const::OK;
> }
>
> 1;
>
> and here's the conf (these tests were all running through a light mod_proxy
> front end too)
>
>
> <Location /cgi/>
> <IfDefine FrontEnd>
> ProxyPass http://localhost:8080/cgi/
> ProxyPassReverse http://localhost:8080/cgi/
> </IfDefine>
> </Location>
> <IfDefine BackEnd>
> ScriptAlias /cgi/ /www/p/
> </IfDefine>
>
>
> Alias /perlrun/ /www/p/
> <Location /perlrun/>
> <IfDefine FrontEnd>
> ProxyPass http://localhost:8080/perlrun/
> ProxyPassReverse http://localhost:8080/perlrun/
> </IfDefine>
> <IfDefine BackEnd>
> SetHandler perl-script
> PerlHandler ModPerl::PerlRun
> Options +ExecCGI
> PerlSendHeader On
> </IfDefine>
> </Location>
>
> Alias /registry/ /www/p/
> <Location /registry/>
> <IfDefine FrontEnd>
> ProxyPass http://localhost:8080/registry/
> ProxyPassReverse http://localhost:8080/registry/
> </IfDefine>
> <IfDefine BackEnd>
> SetHandler perl-script
> PerlHandler ModPerl::Registry
> Options +ExecCGI
> PerlSendHeader On
> </IfDefine>
> </Location>
>
>
>
> <Location /perlscript/>
> <IfDefine FrontEnd>
> ProxyPass http://localhost:8080/perlscript/
> ProxyPassReverse http://localhost:8080/perlscript/
> </IfDefine>
> <IfDefine BackEnd>
> SetHandler perl-script
> PerlResponseHandler Kabob::HelloWorld
> </IfDefine>
> </Location>
>
> <Location /modperl/>
> <IfDefine FrontEnd>
> ProxyPass http://localhost:8080/modperl/
> ProxyPassReverse http://localhost:8080/modperl/
> </IfDefine>
> <IfDefine BackEnd>
> SetHandler modperl
> PerlResponseHandler Kabob::HelloWorld
> </IfDefine>
> </Location>
>
>
> and here's the results (which are no doubt flawed for a number of reasons)
>
> running: ab -n 10000 [url]
>
> CGI
> Requests per second: 217.80 [#/sec] (mean)
> Time per request: 4.591 [ms] (mean)
> Transfer rate: 53.17 [Kbytes/sec] received
>
> PerlRun
> Requests per second: 482.49 [#/sec] (mean)
> Time per request: 2.073 [ms] (mean)
> Transfer rate: 114.49 [Kbytes/sec] received
>
> Registry
> Requests per second: 693.33 [#/sec] (mean)
> Time per request: 1.442 [ms] (mean)
> Transfer rate: 164.53 [Kbytes/sec] received
>
> SetHandler perl-script
> Requests per second: 772.12 [#/sec] (mean)
> Time per request: 1.295 [ms] (mean)
> Transfer rate: 189.94 [Kbytes/sec] received
>
> SetHandler modperl
> Requests per second: 1048.66 [#/sec] (mean)
> Time per request: 0.954 [ms] (mean)
> Transfer rate: 250.84 [Kbytes/sec] received
>
> I'm not sure how well you can really compare the CGI emulation numbers to
> the PerlHandler numbers, but personally i think the 30%ish improvement from
> perl-script to modperl is pretty amazing. I wouldn't have imagined it would
> have been that high.
>
> Adam
>
>
>
>
>
>
>
>
>
--
Dodger
Re: some flawed benchmarks
Posted by Adam Prime <ad...@utoronto.ca>.
Andy Armstrong wrote:
> It would be interesting to see how FCGI compares to those numbers.
I don't know anything about fastcgi, but i suppose i could look at that
also this evening.
Adam
Re: some flawed benchmarks
Posted by Andy Armstrong <an...@hexten.net>.
On 10 Jul 2008, at 06:28, adam.prime@utoronto.ca wrote:
> I'm not sure how well you can really compare the CGI emulation
> numbers to the PerlHandler numbers, but personally i think the
> 30%ish improvement from perl-script to modperl is pretty amazing. I
> wouldn't have imagined it would have been that high.
It would be interesting to see how FCGI compares to those numbers.
--
Andy Armstrong, Hexten
Re: some flawed benchmarks
Posted by Adam Prime <ad...@utoronto.ca>.
I deliberately removed CGI from the script because i personally would
never use CGI in something written to be run as straight handlers, and
it obviously wouldn't make any sense to use CGI in the CGI emulations,
and then not use it in the Handler version.
Not using heredoc's shouldn't really have any effect on the (already
flawed) results, because they weren't used in any of the examples. But
yeah, it was late and i should have been sleeping but i was dorking
around with this instead.
I'm personally not really interested in how using CGI affects the
numbers, but if other people would like to see them i can do this later
tonight.
Adam
Dodger wrote:
> Oh. I would also recommend three variants, based on what people often
> do, what people sometimes do, and what people probably should do when
> using CGI.pm, which can make a difference (just for thoroughness):
>
> Usually done:
> #!/usr/bin/perl
> use CGI;
> print header;
>
> print <<"EOF";
> <html>
> <body>
> <h1>Environment dump:</h1>
> <dl>
> @{[map "<dt>$_</dt>\n<dd>$ENV{$_}</dd>\n", sort keys %ENV]}
> </dl>
> </body>
> </html>
> EOF
>
> Sometimes do:
> #!/usr/bin/perl
> use strict;
> use CGI;
> my $cgi = new CGI;
> print $cgi->header;
>
> print <<"EOF";
> <html>
> <body>
> <h1>Environment dump:</h1>
> <dl>
> @{[map "<dt>$_</dt>\n<dd>$ENV{$_}</dd>\n", sort keys %ENV]}
> </dl>
> </body>
> </html>
> EOF
>
> Might do occassionally, and probably should do all the time if using CGI:
>
> #!/usr/bin/perl
> use strict;
> use CGI(); # note the difference -- using CGI in OO mode, don't import
> *anything*
> my $cgi = new CGI;
> print $cgi->header;
>
> print <<"EOF";
> <html>
> <body>
> <h1>Environment dump:</h1>
> <dl>
> @{[map "<dt>$_</dt>\n<dd>$ENV{$_}</dd>\n", sort keys %ENV]}
> </dl>
> </body>
> </html>
> EOF
>
Re: some flawed benchmarks
Posted by Dodger <el...@gmail.com>.
Oh. I would also recommend three variants, based on what people often
do, what people sometimes do, and what people probably should do when
using CGI.pm, which can make a difference (just for thoroughness):
Usually done:
#!/usr/bin/perl
use CGI;
print header;
print <<"EOF";
<html>
<body>
<h1>Environment dump:</h1>
<dl>
@{[map "<dt>$_</dt>\n<dd>$ENV{$_}</dd>\n", sort keys %ENV]}
</dl>
</body>
</html>
EOF
Sometimes do:
#!/usr/bin/perl
use strict;
use CGI;
my $cgi = new CGI;
print $cgi->header;
print <<"EOF";
<html>
<body>
<h1>Environment dump:</h1>
<dl>
@{[map "<dt>$_</dt>\n<dd>$ENV{$_}</dd>\n", sort keys %ENV]}
</dl>
</body>
</html>
EOF
Might do occassionally, and probably should do all the time if using CGI:
#!/usr/bin/perl
use strict;
use CGI(); # note the difference -- using CGI in OO mode, don't import
*anything*
my $cgi = new CGI;
print $cgi->header;
print <<"EOF";
<html>
<body>
<h1>Environment dump:</h1>
<dl>
@{[map "<dt>$_</dt>\n<dd>$ENV{$_}</dd>\n", sort keys %ENV]}
</dl>
</body>
</html>
EOF
2008/7/9 <ad...@utoronto.ca>:
> A couple of months ago i was going through slides from gozers "From CGI to
> mod_perl 2.0, Fast!" talk, which has some benchmarks comparing CGI, perlrun
> and registry to each other. At which point i realized that i've never
> really known how much faster using straight handlers is than using one of
> the CGI emulation layers. I also didn't have any idea how much faster
> SetHandler modperl was vs SetHandler perl-script.
>
> So i decided to see what i could figure out. I took gozers CGI from the
> slides (slightly modified) and ran it through the paces on my laptop, then
> converted the script to run as a straight handler.
>
> here's the CGI version:
>
> #!/usr/bin/perl
>
>
> print qq[Content-Type: text/html\r\n\r\n];
>
> print(qq[
> <html><body>
> <h1>Hello Worlds</h1>
> <pre>
> GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}
> MOD_PERL: $ENV{MOD_PERL}
> </pre>
> </body></html>
>
> ]);
>
>
> Here's the Handler version
>
> package Kabob::HelloWorld;
>
> use strict;
> use warnings;
>
> use Apache2::RequestRec ();
>
> use Apache2::Const -compile =>qw(:common);
>
> sub handler {
> my $r = shift;
>
> $r->content_type('text/html');
> $r->print(qq[
> <html><body>
> <h1>Hello Worlds</h1>
> <pre>
> GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}
> MOD_PERL: $ENV{MOD_PERL}
> </pre>
> </body></html>
>
>
> ]);
>
> return Apache2::Const::OK;
> }
>
> 1;
>
> and here's the conf (these tests were all running through a light mod_proxy
> front end too)
>
>
> <Location /cgi/>
> <IfDefine FrontEnd>
> ProxyPass http://localhost:8080/cgi/
> ProxyPassReverse http://localhost:8080/cgi/
> </IfDefine>
> </Location>
> <IfDefine BackEnd>
> ScriptAlias /cgi/ /www/p/
> </IfDefine>
>
>
> Alias /perlrun/ /www/p/
> <Location /perlrun/>
> <IfDefine FrontEnd>
> ProxyPass http://localhost:8080/perlrun/
> ProxyPassReverse http://localhost:8080/perlrun/
> </IfDefine>
> <IfDefine BackEnd>
> SetHandler perl-script
> PerlHandler ModPerl::PerlRun
> Options +ExecCGI
> PerlSendHeader On
> </IfDefine>
> </Location>
>
> Alias /registry/ /www/p/
> <Location /registry/>
> <IfDefine FrontEnd>
> ProxyPass http://localhost:8080/registry/
> ProxyPassReverse http://localhost:8080/registry/
> </IfDefine>
> <IfDefine BackEnd>
> SetHandler perl-script
> PerlHandler ModPerl::Registry
> Options +ExecCGI
> PerlSendHeader On
> </IfDefine>
> </Location>
>
>
>
> <Location /perlscript/>
> <IfDefine FrontEnd>
> ProxyPass http://localhost:8080/perlscript/
> ProxyPassReverse http://localhost:8080/perlscript/
> </IfDefine>
> <IfDefine BackEnd>
> SetHandler perl-script
> PerlResponseHandler Kabob::HelloWorld
> </IfDefine>
> </Location>
>
> <Location /modperl/>
> <IfDefine FrontEnd>
> ProxyPass http://localhost:8080/modperl/
> ProxyPassReverse http://localhost:8080/modperl/
> </IfDefine>
> <IfDefine BackEnd>
> SetHandler modperl
> PerlResponseHandler Kabob::HelloWorld
> </IfDefine>
> </Location>
>
>
> and here's the results (which are no doubt flawed for a number of reasons)
>
> running: ab -n 10000 [url]
>
> CGI
> Requests per second: 217.80 [#/sec] (mean)
> Time per request: 4.591 [ms] (mean)
> Transfer rate: 53.17 [Kbytes/sec] received
>
> PerlRun
> Requests per second: 482.49 [#/sec] (mean)
> Time per request: 2.073 [ms] (mean)
> Transfer rate: 114.49 [Kbytes/sec] received
>
> Registry
> Requests per second: 693.33 [#/sec] (mean)
> Time per request: 1.442 [ms] (mean)
> Transfer rate: 164.53 [Kbytes/sec] received
>
> SetHandler perl-script
> Requests per second: 772.12 [#/sec] (mean)
> Time per request: 1.295 [ms] (mean)
> Transfer rate: 189.94 [Kbytes/sec] received
>
> SetHandler modperl
> Requests per second: 1048.66 [#/sec] (mean)
> Time per request: 0.954 [ms] (mean)
> Transfer rate: 250.84 [Kbytes/sec] received
>
> I'm not sure how well you can really compare the CGI emulation numbers to
> the PerlHandler numbers, but personally i think the 30%ish improvement from
> perl-script to modperl is pretty amazing. I wouldn't have imagined it would
> have been that high.
>
> Adam
>
>
>
>
>
>
>
>
>
--
Dodger