You are viewing a plain text version of this content. The canonical link for it is here.
Posted to asp@perl.apache.org by Joshua Chamas <jo...@chamas.com> on 2001/11/20 02:05:53 UTC
Linux Hello World Benchmarks - 11/19/2001
Hey, [[ NUMBERS ARE BELOW ]]
It has been a while, but here's a new set of Hello World benchmarks!
What took me so long in getting these out is that the java web environments
that I had set up would keep crashing during the tests in ways that would
not only render their benchmarks meaningless, but also skewed the results
for other tests, say when a java thread would spin out of control!
You can download the latest benchmark suite to run on your system at:
http://www.chamas.com/bench/hello.tar.gz
It seems to run on linux & solaris just fine, though the tests that
are run depend on what environments you have set up. Please see the
distribution README on how to run the suite for yourself.
--Josh
METHODOLOGY: These latest numbers come from my running the pubbench.sh script:
]# cat pubbench.sh
#!/bin/bash
./bench.pl -version --ram --init-exec="perl javainit.pl -restart" -time=180 --store=results
perl ./javainit.pl -stop
The javainit.pl will kill any of my existing java environments, and if the current
test needs java will start them up. The test run for 180 seconds total in 60 second
batches, and for each 60 second run the javainit.pl script will be executed.
The --ram flag attempts to calculate the RAM used by each test on my linux system
during each 60 second test with the results averaged across each, so for applications
with memory leaks, the RAM used only shows 60 seconds worth of them. The RAM
calculation looks at RAM used & SWAP used from the linux /proc/meminfo file... as
it looks at RAM before the httpd is fired up, but after the init-exec script
executes, the RAM does not take into account the java web server environments
being started, just memory usage that occurs after they start being used.
DISCLAIMER: These numbers are just that, numbers. Please don't let them
upset you. If you have a positive contribution that you would like to make,
you may download the source code and submit patches ( preferably as diff -u )
to the test suite. This benchmark suite has evolved over years of benchmarking,
and takes the point of view that web environments can only be well compared when
run on the exact same system configuration... this is why the suite is done
is such a way that you can run it on _your_ system giving you results relevant
to you!
NUMBERS: And without further adieu ... ( see NOTES below )
Test Name Test File Hits/sec # of Hits Time(sec) secs/Hit Bytes/Hit Mem(KB)
--------- --------- --------- --------- --------- --------- --------- ---------
Apache::ASP v2.29 2000 h2000.asp 216.8 39042 180.09 0.004613 28998 33632
Apache::Registry v2.01 2000 mod_per h2000.reg 341.3 61442 180.04 0.002930 28179 16081
HTML::Embperl v1.3.0 2000 h2000.epl 113.2 20391 180.12 0.008833 28841 20418
HTML::Mason v1.03 2000 h2000.mas 84.0 15115 180.03 0.011911 28799 112220
HTML::Template v2.4 2000 h2000.htm 99.1 17848 180.03 0.010087 29152 50292
mod_caucho JSP 2000 h2000.jsp 97.6 17617 180.54 0.010248 28965 14057
mod_jserv JSP 2000 h2000.jsp 148.7 26773 180.04 0.006725 29408 29613
mod_php PHP 2000 h2000.php 258.2 46477 180.02 0.003873 28866 10212
Template v2.04 Toolkit 2000 h2000.tt 53.3 9604 180.03 0.018745 28889 55629
Test Name Test File Hits/sec # of Hits Time(sec) secs/Hit Bytes/Hit Mem(KB)
--------- --------- --------- --------- --------- --------- --------- ---------
Apache::ASP v2.29 hello.asp 349.8 62985 180.05 0.002859 242 29601
Apache::Dispatch v0.09 handler hello/wor 585.8 105443 180.01 0.001707 197 9205
Apache::ePerl hello.epe 347.1 62508 180.08 0.002881 218 17477
Apache::Registry v2.01 CGI Raw hello.cgi 677.7 122003 180.03 0.001476 52 13061
Apache::Registry v2.01 CGI.pm hello.cgi 449.7 80967 180.05 0.002224 217 20717
Apache::SSI v2.16 hello.sht 546.7 98426 180.03 0.001829 200 13061
CGI::SpeedyCGI CGI Raw hello.scg 159.6 28722 180.00 0.006267 197 9748
CGI::SpeedyCGI CGI.pm hello.scg 145.1 26117 180.01 0.006892 217 10648
HTML static hello.htm 879.2 150000 170.61 0.001137 312 1982
HTML::Embperl v1.3.0 hello.epl 463.6 83471 180.05 0.002157 221 17333
HTML::Mason v1.03 hello.mas 368.5 66348 180.04 0.002714 198 29009
HTML::Template v2.4 hello.htm 539.1 97056 180.03 0.001855 199 14850
mod_caucho JSP hello.jsp 147.6 26596 180.20 0.006775 231 14442
mod_cgi CGI Raw hello.cgi 156.7 28208 180.02 0.006382 197 9506
mod_cgi CGI.pm hello.cgi 11.1 2040 183.53 0.089966 217 25893
mod_include SSI hello.sht 241.8 43605 180.36 0.004136 199 4297
mod_jserv JSP hello.jsp 73.1 13209 180.81 0.013688 358 10788
mod_perl handler hello.ben 813.8 146490 180.00 0.001229 197 5420
mod_php PHP hello.php 687.5 123753 180.00 0.001455 226 6008
Template v2.04 Toolkit hello.tt 469.2 84470 180.02 0.002131 199 42472
Test Name Test File Hits/sec # of Hits Time(sec) secs/Hit Bytes/Hit Mem(KB)
--------- --------- --------- --------- --------- --------- --------- ---------
Apache::ASP v2.29 XSLT Hello hxslt.xml 185.0 33292 180.00 0.005407 292 39892
AxKit v1.4 XSLT Hello hxslt.xml 298.2 53678 180.03 0.003354 411 30622
Cocoon 1.8 mod_jserv XSLT Hello hxslt.xml 56.1 10144 180.80 0.017823 507 15784
Test Name Test File Hits/sec # of Hits Time(sec) secs/Hit Bytes/Hit Mem(KB)
--------- --------- --------- --------- --------- --------- --------- ---------
Apache::ASP v2.29 XSLT Big hxsltbig. 138.6 24957 180.03 0.007214 21521 50572
AxKit v1.4 XSLT Big hxsltbig. 262.1 47206 180.08 0.003815 21859 31390
Cocoon 1.8 mod_jserv XSLT Big hxsltbig. 79.2 9525 120.25 0.012625 21921 29116
Apache Server Header Tokens
---------------------------
(Unix)
Apache/1.3.20
AxKit/1.4
OpenSSL/0.9.6a
PHP/4.0.3pl1
Resin/1.2.1
mod_perl/1.25
mod_ssl/2.8.4
tomcat/1.0
PERL Version: 5.00503
JAVA Version: java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0)
Classic VM (build 1.3.0, J2RE 1.3.0 IBM build cx130-20001124 (JIT enabled: jitc))
Operating System: Linux 2.2.14-9.0 (root@gate) (gcc egcs-2.91.66) #1 2CPU [gate.]
NOTES:
h2000.epl - Embperl 2.0 is much faster on this test, about the same speed as PHP,
but its not installable from CPAN which is why I don't run it. When the release
stablizes ( from alpha/beta release ), I will run 2.0 for this test.
h2000.mas - HTML::Mason seems to have a memory leak on this test where it does not
seem to have it on the hello.mas. If someone has a patch for the benchmark
to fix this memory leak which would be consistent with ordinary Mason programming
methods, please submit it. If there is a real memory leak under Mason
for this test, then perhaps this test will help the authors fix it?
h2000.jsp - mod_jserv beats mod_caucho on this test, but results flip flop for
hello.jsp. I am not sure why this is. The java environments tend to benchmark
unpredicatably, even for longer tests. This may be a standard result though
and would not surprise me.
HTML Static - Static HTML should generally be much faster than any other test, but is
only slightly faster than mod_perl on my system. This is likely because
my box is a dual processor 450 PIII running Linux 2.2.14 ( Redhat 6.2 ),
and this version of linux does not deal efficiently with many processors.
I imagine a benchmark on a 2.4.x linux kernel on this platform would show
more of a difference.
Cocoon 1.8 mod_jserv XSLT Big - This test only ran for 120 seconds, instead
of 180 that all the other tests ran for. This is because of the 3 60 seconds
test, the java process core dumped on one of them, so we only have 120 seconds
worth of data. Also, why this test performs better than the Cocoon hxslt.xml,
I have no idea.
AxKit & Apache::ASP XSLT tests - These tests used XML::Sablotron, though they
could have used XML::LibXSLT. For these tests it does not matter, because
the transformation ends up being cached to disk anyway.
Apache::ASP v2.29 XSLT Big - This test uses a lot of memory. This is likely
due to Apache::ASP treating XSLT XML data & XSL templates as ASP scripts,
which involves first compiling them to perl bytecode, prior to execution
and final XSLT rendering of their output. This will likely be fixed in
later releases by Apache::ASP detecting at runtime when files are
static & don't have <% %> ASP code blocks, allow it to bypass the
perl bytecode compilation process, which will help particularly for
static XSLT scenarios.
---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org
Re: Linux Hello World Benchmarks - 11/19/2001
Posted by Joshua Chamas <jo...@chamas.com>.
Perrin Harkins wrote:
>
> on 11/19/01 8:05 PM, Joshua Chamas at joshua@chamas.com wrote:
> > It has been a while, but here's a new set of Hello World benchmarks!
>
> There was a recent announcement of HTML::Template::JIT, and Template Toolkit
> has an XS option now. Any chance you could put those into the next round?
> - Perrin
OK, I upgraded Template Toolkit to version 2.06 with XS option on
for all templates, and performance is much improved!!
However, HTML::Template::JIT will likely not work on my system any
time soon, because it was written for perl 5.6, and I only run 5.00503
currently. Someone else can submit a test & config specifically for
HTML::Template::JIT, I'd suggest the h2000 type where speed matters, and
I'll add it to the test suite even it I can't run the benchmark myself.
[hello]# ./bench.pl -type=2000 -ram -time=60
Test Name Test File Hits/sec # of Hits Time(sec) secs/Hit Bytes/Hit Mem(KB)
--------- --------- --------- --------- --------- --------- --------- ---------
Apache::ASP 2000 h2000.asp 209.4 12566 60.02 0.004776 28998 33200
Apache::Registry 2000 mod_perl API h2000.reg 327.0 19622 60.02 0.003059 28179 15300
HTML::Embperl 2000 h2000.epl 107.5 6465 60.12 0.009299 28841 20048
HTML::Mason 2000 h2000.mas 80.8 4849 60.00 0.012374 28799 110292
HTML::Template 2000 h2000.htm 95.4 5724 60.01 0.010484 29152 49932
mod_php PHP 2000 h2000.php 247.6 14862 60.02 0.004038 28866 10384
Template Toolkit 2000 h2000.tt 125.5 7533 60.02 0.007967 28889 55748
-- Josh
_________________________________________________________________
Joshua Chamas Chamas Enterprises Inc.
NodeWorks Founder Huntington Beach, CA USA
http://www.nodeworks.com 1-714-625-4051
Re: Linux Hello World Benchmarks - 11/19/2001
Posted by Joshua Chamas <jo...@chamas.com>.
Perrin Harkins wrote:
>
> on 11/19/01 8:05 PM, Joshua Chamas at joshua@chamas.com wrote:
> > It has been a while, but here's a new set of Hello World benchmarks!
>
> There was a recent announcement of HTML::Template::JIT, and Template Toolkit
> has an XS option now. Any chance you could put those into the next round?
> - Perrin
OK, I upgraded Template Toolkit to version 2.06 with XS option on
for all templates, and performance is much improved!!
However, HTML::Template::JIT will likely not work on my system any
time soon, because it was written for perl 5.6, and I only run 5.00503
currently. Someone else can submit a test & config specifically for
HTML::Template::JIT, I'd suggest the h2000 type where speed matters, and
I'll add it to the test suite even it I can't run the benchmark myself.
[hello]# ./bench.pl -type=2000 -ram -time=60
Test Name Test File Hits/sec # of Hits Time(sec) secs/Hit Bytes/Hit Mem(KB)
--------- --------- --------- --------- --------- --------- --------- ---------
Apache::ASP 2000 h2000.asp 209.4 12566 60.02 0.004776 28998 33200
Apache::Registry 2000 mod_perl API h2000.reg 327.0 19622 60.02 0.003059 28179 15300
HTML::Embperl 2000 h2000.epl 107.5 6465 60.12 0.009299 28841 20048
HTML::Mason 2000 h2000.mas 80.8 4849 60.00 0.012374 28799 110292
HTML::Template 2000 h2000.htm 95.4 5724 60.01 0.010484 29152 49932
mod_php PHP 2000 h2000.php 247.6 14862 60.02 0.004038 28866 10384
Template Toolkit 2000 h2000.tt 125.5 7533 60.02 0.007967 28889 55748
-- Josh
_________________________________________________________________
Joshua Chamas Chamas Enterprises Inc.
NodeWorks Founder Huntington Beach, CA USA
http://www.nodeworks.com 1-714-625-4051
---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org
Re: Linux Hello World Benchmarks - 11/19/2001
Posted by Perrin Harkins <pe...@elem.com>.
on 11/19/01 8:05 PM, Joshua Chamas at joshua@chamas.com wrote:
> It has been a while, but here's a new set of Hello World benchmarks!
There was a recent announcement of HTML::Template::JIT, and Template Toolkit
has an XS option now. Any chance you could put those into the next round?
- Perrin
---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org
Re: Linux Hello World Benchmarks - 11/19/2001
Posted by Perrin Harkins <pe...@elem.com>.
on 11/19/01 8:05 PM, Joshua Chamas at joshua@chamas.com wrote:
> It has been a while, but here's a new set of Hello World benchmarks!
There was a recent announcement of HTML::Template::JIT, and Template Toolkit
has an XS option now. Any chance you could put those into the next round?
- Perrin