You are viewing a plain text version of this content. The canonical link for it is here.
Posted to test-dev@httpd.apache.org by Justin Erenkrantz <je...@ebuilt.com> on 2001/08/25 02:08:37 UTC

Re: Pools, possible replacement, WAS: RE: [Fwd: brianp patch Quantify results]

On Sat, Aug 25, 2001 at 12:23:26AM +0200, Sander Striker wrote:
> Justin will probably follow up later with test results.

Well, I did run the tests.  However, in all cases/combinations 
(new pool/old pool, threaded/prefork MPM), our limitation is the 
network bandwidth.  We're currently pulling about 70-75Mbps 
which seems to max out our Cisco Catalyst 2940 (100BaseT) 
switches.  This leads to about ~405rps for my test.

This is a *good* sign however - in the (recent) past with Apache 
2.0, I haven't been able to max out the network bandwidth.  Roy
has suggested using smaller files/requests - I'll try that
next week.  What I'll say is that Sander's new code doesn't
crash (it is cleaner code).  As far as performance goes, I can't
really judge that with the network we have - as we can't get 
much better without a network overhaul.  =)

If you have the backplane (my guess would be Gigabit Ethernet) to
support this, I can send anyone the flood configs I was using.
They are similar to the URLs I posted earlier as a suggestion 
for an attempt at standardizing benchmarks for common httpd-2.0 
code.

For the curious, the machines we were using were both dual 
PIII/550s with 1GB of RAM running Solaris (client is Sol7, 
server is Sol8).  They were yawning throughout the test (about 
50% idle on the server).  -- justin


Re: Pools, possible replacement, WAS: RE: [Fwd: brianp patch Quantify results]

Posted by Justin Erenkrantz <je...@ebuilt.com>.
> The t/htdocs/modules/include/*.shtml stuff is probably the best at the
> moment... if you can run those many, many times with a lot of concurrency,
> that'd probably get you somewhere.

Thanks.  I didn't even think of using httpd-test's perl-framework (I'm 
not really familiar with it).  I snarfed in that directory and pointed
flood at big.  Now, I think we're getting somewhere - the CPU is maxed
out.  Cool.

> Throw mod_ssl into the equation and you might get even closer to something
> testable, though it'd arguably be better if the load were pool-use-heavy.

Well, I think with all of the benchmarks flying around, it'd be nice to
agree that "URL X, followed by URL Y, followed by URL Z" is at least
a decent coverage of httpd-2.0 code.  At least for determining relative
merits of code A vs. code B that affects the entire server.  I dunno,
it's a thought.  Right now, everybody just uses ab and points at /
or some small static file.  I think it'd be nice to standardize this
somewhat.

FWIW, the default httpd.conf is awful.  mod_include isn't allowed by
default anywhere (even in the manual).  That needs to be fixed.  It
took me 30 minutes to get mod_include to serve a shtml file.  Drats.
-- justin


Re: Pools, possible replacement, WAS: RE: [Fwd: brianp patch Quantify results]

Posted by Cliff Woolley <cl...@yahoo.com>.
On Fri, 24 Aug 2001, Justin Erenkrantz wrote:

> On Fri, Aug 24, 2001 at 05:44:40PM -0700, Aaron Bannert wrote:
> > mod_include, PHP -- something dynamic? You just want to CPU-strap
> > some typical system usage, and then see what the maximum req/sec
> > throughput is you can get, right?
>
> Right, but I see no mod_include pages that are in the current CVS
> tree (FAQ.html looks likes it should be).  I'm trying to do
> something that is reproducible by other people.  =-)  -- justin

The t/htdocs/modules/include/*.shtml stuff is probably the best at the
moment... if you can run those many, many times with a lot of concurrency,
that'd probably get you somewhere.

Better yet, take all of those, make a bigass.shtml out of them with loads
of recursion and multiple includes of the same file over and over, etc.

Throw mod_ssl into the equation and you might get even closer to something
testable, though it'd arguably be better if the load were pool-use-heavy.

--Cliff

--------------------------------------------------------------
   Cliff Woolley
   cliffwoolley@yahoo.com
   Charlottesville, VA



Re: Pools, possible replacement, WAS: RE: [Fwd: brianp patch Quantify results]

Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Fri, Aug 24, 2001 at 05:44:40PM -0700, Aaron Bannert wrote:
> mod_include, PHP -- something dynamic? You just want to CPU-strap
> some typical system usage, and then see what the maximum req/sec
> throughput is you can get, right?

Right, but I see no mod_include pages that are in the current CVS
tree (FAQ.html looks likes it should be).  I'm trying to do 
something that is reproducible by other people.  =-)  -- justin


Re: Pools, possible replacement, WAS: RE: [Fwd: brianp patch Quantify results]

Posted by Aaron Bannert <aa...@clove.org>.
On Fri, Aug 24, 2001 at 05:40:59PM -0700, Justin Erenkrantz wrote:
> On Fri, Aug 24, 2001 at 05:21:11PM -0700, Marc Slemko wrote:
> > On Fri, 24 Aug 2001, Justin Erenkrantz wrote:
> > 
> > > If you have the backplane (my guess would be Gigabit Ethernet) to
> > 
> > Well, or just a less powerful server.  The goal isn't the best raw
> > numbers, the goal is comparison.  Granted, that comparison can be
> > different on various sized machines (eg. SMP vs. not, etc.).
> 
> Well, part of the problem is that Sander's pool optimization is 
> intended to benefit MP boxes.  It adds thread-local free-lists - so
> the benefit is probably only noticable on MP boxes under load.
> 
> The problem is that I can't get enough out of our network (with
> the URLs I hit) to see if it improves the performance.  I will
> try to adjust the URLs I hit to see if I can move the bottleneck
> back to the CPU.  
> 
> Any suggestions here?  -- justin

mod_include, PHP -- something dynamic? You just want to CPU-strap
some typical system usage, and then see what the maximum req/sec
throughput is you can get, right?

-aaron

Re: Pools, possible replacement, WAS: RE: [Fwd: brianp patch Quantify results]

Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Fri, Aug 24, 2001 at 10:36:11PM -0700, Ian Holsman wrote:
> BTW can flood simulate a steady # of requests being attempted per
> second?  or does it just fire up N threads/processes and whack away.
> ( check out http://www.cs.rice.edu/CS/Systems/Web-measurement/ for more
> info)

Yeah, I read that paper last night and I believe flood can simulate
the S-Client under the following syntax:

<usefarmer count="10000" startdelay="1" startcount="2">Joe</usefarmer>

Flood will start two threads every second until there are 10000 active.
I suppose we could make it open-ended, but that complicates things.  
We could also make the resolution of the XML more precise (i.e. .5 
seconds), but Aaron and I disagreed on how to do this.  =)

In order to do the S-Client as described, you must have two processes 
with a domain socket and use event-based socket operations.  I don't 
feel that is very portable or scalable.  However, their idea of
continuing to ramp up the requests regardless of the server's response
time does make sense.  -- justin


Re: Pools, possible replacement, WAS: RE: [Fwd: brianp patch Quantify results]

Posted by "Roy T. Fielding" <fi...@ebuilt.com>.
> BTW can flood simulate a steady # of requests being attempted per
> second? 
> or does it just fire up N threads/processes and whack away.
> ( check out http://www.cs.rice.edu/CS/Systems/Web-measurement/ for more
> info)

Heh, nice coincidence -- I printed that paper out earlier today and
gave Justin a copy.

Flood has settable delays between requests with (I think) pseudo-random
adjustments.  It can also blast away, if that's what is needed for a
given test.

....Roy


Re: Pools, possible replacement, WAS: RE: [Fwd: brianp patch Quantify results]

Posted by "Roy T. Fielding" <fi...@ebuilt.com>.
> BTW can flood simulate a steady # of requests being attempted per
> second? 
> or does it just fire up N threads/processes and whack away.
> ( check out http://www.cs.rice.edu/CS/Systems/Web-measurement/ for more
> info)

Heh, nice coincidence -- I printed that paper out earlier today and
gave Justin a copy.

Flood has settable delays between requests with (I think) pseudo-random
adjustments.  It can also blast away, if that's what is needed for a
given test.

....Roy


Re: Pools, possible replacement, WAS: RE: [Fwd: brianp patch Quantify results]

Posted by Ian Holsman <Ia...@cnet.com>.
On Fri, 2001-08-24 at 17:40, Justin Erenkrantz wrote:
> On Fri, Aug 24, 2001 at 05:21:11PM -0700, Marc Slemko wrote:
> > On Fri, 24 Aug 2001, Justin Erenkrantz wrote:
> > 
instead of looking at raw network thoughput look at heavy users of
pools, CPU utilization, and truss/prof output.
and compare these results with standard pool, and with brians patch.

I would concentrate more on getting 100 rps going through in the
following conditions:

* standard file no parsing (ie .. sendfile)
* mod include with 2 includes (header/footer) (as this REALLY uses
pools heavily)
* Keep-alives

I'll fire up the 8-way box on monday and compare sanders and brians
pool code against the 'stock' pool and see what I can see.

There was also the pool replay code I wrote a while back which could be
used to compare pool allocaters, as well as dreieds memory tests for the
SMS.

BTW can flood simulate a steady # of requests being attempted per
second? 
or does it just fire up N threads/processes and whack away.
( check out http://www.cs.rice.edu/CS/Systems/Web-measurement/ for more
info)
> > > If you have the backplane (my guess would be Gigabit Ethernet) to
> > 
> > Well, or just a less powerful server.  The goal isn't the best raw
> > numbers, the goal is comparison.  Granted, that comparison can be
> > different on various sized machines (eg. SMP vs. not, etc.).
> 
> Well, part of the problem is that Sander's pool optimization is 
> intended to benefit MP boxes.  It adds thread-local free-lists - so
> the benefit is probably only noticable on MP boxes under load.
> 
> The problem is that I can't get enough out of our network (with
> the URLs I hit) to see if it improves the performance.  I will
> try to adjust the URLs I hit to see if I can move the bottleneck
> back to the CPU.  
> 
> Any suggestions here?  -- justin
-- 
Ian Holsman
Performance Measurement & Analysis
CNET Networks    -    415 364-8608

Re: Pools, possible replacement, WAS: RE: [Fwd: brianp patch Quantify results]

Posted by Ian Holsman <Ia...@cnet.com>.
On Fri, 2001-08-24 at 17:40, Justin Erenkrantz wrote:
> On Fri, Aug 24, 2001 at 05:21:11PM -0700, Marc Slemko wrote:
> > On Fri, 24 Aug 2001, Justin Erenkrantz wrote:
> > 
instead of looking at raw network thoughput look at heavy users of
pools, CPU utilization, and truss/prof output.
and compare these results with standard pool, and with brians patch.

I would concentrate more on getting 100 rps going through in the
following conditions:

* standard file no parsing (ie .. sendfile)
* mod include with 2 includes (header/footer) (as this REALLY uses
pools heavily)
* Keep-alives

I'll fire up the 8-way box on monday and compare sanders and brians
pool code against the 'stock' pool and see what I can see.

There was also the pool replay code I wrote a while back which could be
used to compare pool allocaters, as well as dreieds memory tests for the
SMS.

BTW can flood simulate a steady # of requests being attempted per
second? 
or does it just fire up N threads/processes and whack away.
( check out http://www.cs.rice.edu/CS/Systems/Web-measurement/ for more
info)
> > > If you have the backplane (my guess would be Gigabit Ethernet) to
> > 
> > Well, or just a less powerful server.  The goal isn't the best raw
> > numbers, the goal is comparison.  Granted, that comparison can be
> > different on various sized machines (eg. SMP vs. not, etc.).
> 
> Well, part of the problem is that Sander's pool optimization is 
> intended to benefit MP boxes.  It adds thread-local free-lists - so
> the benefit is probably only noticable on MP boxes under load.
> 
> The problem is that I can't get enough out of our network (with
> the URLs I hit) to see if it improves the performance.  I will
> try to adjust the URLs I hit to see if I can move the bottleneck
> back to the CPU.  
> 
> Any suggestions here?  -- justin
-- 
Ian Holsman
Performance Measurement & Analysis
CNET Networks    -    415 364-8608

Re: Pools, possible replacement, WAS: RE: [Fwd: brianp patch Quantify results]

Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Fri, Aug 24, 2001 at 05:21:11PM -0700, Marc Slemko wrote:
> On Fri, 24 Aug 2001, Justin Erenkrantz wrote:
> 
> > If you have the backplane (my guess would be Gigabit Ethernet) to
> 
> Well, or just a less powerful server.  The goal isn't the best raw
> numbers, the goal is comparison.  Granted, that comparison can be
> different on various sized machines (eg. SMP vs. not, etc.).

Well, part of the problem is that Sander's pool optimization is 
intended to benefit MP boxes.  It adds thread-local free-lists - so
the benefit is probably only noticable on MP boxes under load.

The problem is that I can't get enough out of our network (with
the URLs I hit) to see if it improves the performance.  I will
try to adjust the URLs I hit to see if I can move the bottleneck
back to the CPU.  

Any suggestions here?  -- justin


Re: Pools, possible replacement, WAS: RE: [Fwd: brianp patch Quantify results]

Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Fri, Aug 24, 2001 at 05:21:11PM -0700, Marc Slemko wrote:
> On Fri, 24 Aug 2001, Justin Erenkrantz wrote:
> 
> > If you have the backplane (my guess would be Gigabit Ethernet) to
> 
> Well, or just a less powerful server.  The goal isn't the best raw
> numbers, the goal is comparison.  Granted, that comparison can be
> different on various sized machines (eg. SMP vs. not, etc.).

Well, part of the problem is that Sander's pool optimization is 
intended to benefit MP boxes.  It adds thread-local free-lists - so
the benefit is probably only noticable on MP boxes under load.

The problem is that I can't get enough out of our network (with
the URLs I hit) to see if it improves the performance.  I will
try to adjust the URLs I hit to see if I can move the bottleneck
back to the CPU.  

Any suggestions here?  -- justin


Re: Pools, possible replacement, WAS: RE: [Fwd: brianp patch Quantify results]

Posted by Marc Slemko <ma...@znep.com>.
On Fri, 24 Aug 2001, Justin Erenkrantz wrote:

> If you have the backplane (my guess would be Gigabit Ethernet) to

Well, or just a less powerful server.  The goal isn't the best raw
numbers, the goal is comparison.  Granted, that comparison can be
different on various sized machines (eg. SMP vs. not, etc.).

More important than just the performance of a given workload is an
understanding of what different types of things are slower or faster, and
what types of workloads are impacted the most by the changes.  And, of
course, relating those workloads to reality.

So I guess I'm just trying to remind people not to focus too much on
numbers generated by any particular benchmark since they can be misleading
at the best, and can cause "optimizations" that are actually more
expensive in the "real world"...


Re: Pools, possible replacement, WAS: RE: [Fwd: brianp patch Quantify results]

Posted by Marc Slemko <ma...@znep.com>.
On Fri, 24 Aug 2001, Justin Erenkrantz wrote:

> If you have the backplane (my guess would be Gigabit Ethernet) to

Well, or just a less powerful server.  The goal isn't the best raw
numbers, the goal is comparison.  Granted, that comparison can be
different on various sized machines (eg. SMP vs. not, etc.).

More important than just the performance of a given workload is an
understanding of what different types of things are slower or faster, and
what types of workloads are impacted the most by the changes.  And, of
course, relating those workloads to reality.

So I guess I'm just trying to remind people not to focus too much on
numbers generated by any particular benchmark since they can be misleading
at the best, and can cause "optimizations" that are actually more
expensive in the "real world"...