old ZD benchmark comments

[Marc Slemko <ma...@znep.com>]

I haven't actually looked at ZD's previous benchmarks before in
detail because I never saw real graphs.  

Now that I look again, they have better detail online.

http://www.zdnet.com/pcmag/features/webserver/_intro.htm is their 
latest test, from... erm... July 97?  Not sure of the date.  That's
the last-modified anyway.  Oh, this tested 1.1.3.

A reasonable quality PDF version of their results is at:

    ftp://ftp.zdnet.com/pcmag/1997/0527/labsdata/in1610ba.pdf

This shows a few things.

First, Apache had decent latency throughout their tests.  

The two servers they tested on Solaris had almost identical latencies
just below the 200ms mark, and they were identical right from 1 to 60 
clients.  Whenever you see 200ms constants around TCP you should be 
suspicious; that is just obvious.  This is, if I remember, the test that
led to the documentation of the slow-start "disagreement" between Solaris
and NT.  The fact that ZD did no more than say "hmm, that looks odd"
and ask Sun doesn't give me confidence.  These are the things you need
to look at, at the protocol level if necessary, to ensure that you are
not intorducing bogus factors into your tests.  Blackbox doesn't work.

For single processor static content throughput, Apache was middle
of the pack.  About the same as Netscape Enterprise and Fasttrack
on IRIX, but below a bunch of others.  It was around 5000kbytes/sec.
IIS hit the best, hitting up to ~13000kbytes.  Everything else
(eg. FastTrack and Enterprise on NT, DGUX) topped out at ~9000kbytes.

Apache was right in there growing at a linear rate with the number
of clients, like most other servers, until it hit 12 clients, then it 
hit a cap and stopped increasing.

In their NSAPI test, Enterprise on Solairs showed almost the same
rate for both NSAPI and CGI, while on all other platforms there
was a drastic difference.  If that doesn't set off red lights...

Oh yea, I like this:

    Some products, such as O'Reilly's WebSite, have higher latency
    numbers than others because of the way the Web server handles
    incoming requests. This product goes to disk to read a fresh version
    of the requested HTML page every time a request comes in. Going to
    disk every time a page is requested results in a much slower response
    time than caching the page right away (but possibly still acceptable).

Gee, Apache had pretty good latency even in your tests but you didn't
complain about it always going to disk now did you...

Apache showed almost zippo (but did have a little bit) increase adding
another processor.  IIS showed almost zippo increase.  Enterprise on NT,
OTOH, showed a good increase to bring it right up to IIS.

IIS's bouncy results that can't really be fit to a curve concern bit a bit
about IIS.  Enterprise under NT, for example, shows a very well fit
increase.

Re: old ZD benchmark comments

[Brian Behlendorf <br...@organic.com>]

At 10:50 PM 1/21/98 -0700, you wrote:
>Oh yea, and for anyone interested their table is at 
>
>ftp://ftp.zdnet.com/pcmag/1997/0527/labsdata/in1610fa.pdf
>
>Some of the more... "interesting" points:
>
>Apache does not log accesses or page hits, but it does have
>customizable log files.

Prolly not worth correcting them at this point - is there a current table
of server features maintained by ZD somewhere?  

>It can't control access to "parts of documents".  WTF is that?

I can't remember which, but some packages out there let you specify a part
of a file as available only to certain users.  E.g.

  <UL>
  <LI> some stuff
  <LI> other stuff
  <!--#if authenticated_user -->
  <LI> secret plans to overthrow government
  <!--#endif -->
  </UL>

Depending on what "authenticated_user" is I'm sure you could do this with
Apache.

>It can't create private certificates, yet it has scripts or wizard for
>creating certificates.
>
>It supports "no" other APIs, and the Apache API doesn't count I guess.
>
>Oh, and access control isn't handled by the OS.  Erm... sure it is...
>Unix has permissions.  Apache respects them, not that it has a
>choice.  I fail to see why magazines goes on and on about how great it
>is that IIS makes you use NT logins for your web server when in the
>real world it sucks.
>
>Oh yea, and they incorrectly call IIS free.  They label Novell's
>server as "included with intranetware" but for some reason they
>think IIS is different.
>
>Hehehehehahahahahahahehehehe.  You would think they would do something
>odd like trying to verify their facts.
>
>(yes, this has all been gone through before and is old stuff, but...
>I haven't looked at it before...)

Oh, if only we had a marketing department which could spend some time
correcting these things :)

If incorrect reporting was really a problem, we could add the following
somewhere, like to README or LICENSE or something:

"Any publication of information about Apache referring to its capabilities
or performance is unauthorized without contacting the Apache Group at
apache@apache.org".

Like Microsoft does.

	Brian


--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
specialization is for insects				  brian@organic.com

Re: old ZD benchmark comments

[Marc Slemko <ma...@worldgate.com>]

On Wed, 21 Jan 1998, Marc Slemko wrote:

> I haven't actually looked at ZD's previous benchmarks before in
> detail because I never saw real graphs.  

Oh yea, and for anyone interested their table is at 

ftp://ftp.zdnet.com/pcmag/1997/0527/labsdata/in1610fa.pdf

Some of the more... "interesting" points:

Apache does not log accesses or page hits, but it does have
customizable log files.

It can't control access to "parts of documents".  WTF is that?

It can't create private certificates, yet it has scripts or wizard for
creating certificates.

It supports "no" other APIs, and the Apache API doesn't count I guess.

Oh, and access control isn't handled by the OS.  Erm... sure it is...
Unix has permissions.  Apache respects them, not that it has a
choice.  I fail to see why magazines goes on and on about how great it
is that IIS makes you use NT logins for your web server when in the
real world it sucks.

Oh yea, and they incorrectly call IIS free.  They label Novell's
server as "included with intranetware" but for some reason they
think IIS is different.

Hehehehehahahahahahahehehehe.  You would think they would do something
odd like trying to verify their facts.

(yes, this has all been gone through before and is old stuff, but...
I haven't looked at it before...)