You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by "Daniel J. Cody" <dc...@oracular.com> on 1999/04/14 05:00:37 UTC
Holy FUD Batman!
Was just checking out a link from slashdot. This site makes comparisons
between NT/Linux and IIS/Apache including,
"The Linux/Apache Web server performance collapsed with a
WebBench load above 160 client test threads, while Windows NT Server/IIS
continued
to increase performance up through 288 client test threads without
reaching their peak
performance,"
http://www.newsalert.com/bin/story?StoryId=CnXlbqbWbu0zuvta3o&FQ=linux&SymHdl=1&Nav=na-search-&StoryTitle=linux
(wrap)
Thoughts?
--
Daniel J. Cody
-Senior Web Developer-
Oracular Systems & Software, Inc.
dcody@oracular.com
Re: Holy FUD Batman!
Posted by Marc Slemko <ma...@znep.com>.
On Tue, 13 Apr 1999, Dean Gaudet wrote:
> Oh this is on NT.
>
> Read the 4th paragraph at that URL.
No, the Apache is on Linux.
>
> Dean
>
> On Tue, 13 Apr 1999, Dean Gaudet wrote:
>
> > MaxClients 150.
> >
> > Read the first paragraph at
> > http://www.apache.org/docs/misc/perf-tuning.html.
They set MaxClients to 290 after recompiling Apache.
>From their site:
number of threads number of reqs/sec
===================== =====================
2 12
32 199
64 403
96 601
128 802
160 1,000
192 287
224 68
256 78
288 87
That just doesn't make sense to me.
It appears that x threads are running on x/2 systems, with each running
two threads.
I can't imagine why, on a machine using 960 megs with 290 MaxClients,
you would hit that drop at 160. Nothing about the Apache architecture
imposes any such limits. Unless it just has to do with the test
set and one machine being able to cache the whole thing while the
other couldn't. However, without having more details about the
behaviour of the machine under such situations (eg. swapping, out
of CPU, etc.), it is impossible to tell.
Unless it is like SMP issues with 4 processors or something.
Re: Holy FUD Batman!
Posted by Dean Gaudet <dg...@arctic.org>.
Oh this is on NT.
Read the 4th paragraph at that URL.
Dean
On Tue, 13 Apr 1999, Dean Gaudet wrote:
> MaxClients 150.
>
> Read the first paragraph at
> http://www.apache.org/docs/misc/perf-tuning.html.
>
> Dean
>
> On Tue, 13 Apr 1999, Daniel J. Cody wrote:
>
> > Was just checking out a link from slashdot. This site makes comparisons
> > between NT/Linux and IIS/Apache including,
> >
> > "The Linux/Apache Web server performance collapsed with a
> > WebBench load above 160 client test threads, while Windows NT Server/IIS
> > continued
> > to increase performance up through 288 client test threads without
> > reaching their peak
> > performance,"
> >
> > http://www.newsalert.com/bin/story?StoryId=CnXlbqbWbu0zuvta3o&FQ=linux&SymHdl=1&Nav=na-search-&StoryTitle=linux
> > (wrap)
> >
> > Thoughts?
> >
> > --
> > Daniel J. Cody
> > -Senior Web Developer-
> > Oracular Systems & Software, Inc.
> > dcody@oracular.com
> >
>
>
Re: Holy FUD Batman!
Posted by Dean Gaudet <dg...@arctic.org>.
MaxClients 150.
Read the first paragraph at
http://www.apache.org/docs/misc/perf-tuning.html.
Dean
On Tue, 13 Apr 1999, Daniel J. Cody wrote:
> Was just checking out a link from slashdot. This site makes comparisons
> between NT/Linux and IIS/Apache including,
>
> "The Linux/Apache Web server performance collapsed with a
> WebBench load above 160 client test threads, while Windows NT Server/IIS
> continued
> to increase performance up through 288 client test threads without
> reaching their peak
> performance,"
>
> http://www.newsalert.com/bin/story?StoryId=CnXlbqbWbu0zuvta3o&FQ=linux&SymHdl=1&Nav=na-search-&StoryTitle=linux
> (wrap)
>
> Thoughts?
>
> --
> Daniel J. Cody
> -Senior Web Developer-
> Oracular Systems & Software, Inc.
> dcody@oracular.com
>
Re: Holy FUD Batman!
Posted by Marc Slemko <ma...@znep.com>.
Yea, whatever.
Wonder who paid for that survey.
Hmm:
Performance collapses with a WebBench load above 160 threads. We
verified that the problem was with Apache, not Linux, by restarting
Apache at the 256 threads data point during a WebBench test run.
After the restart, Apache performance climbed back to within 30%
of its peak from a low of about 6% of the peak performance.
that makes so little sense that I can't even figure out what they are
trying to say.
As I have always said, you can't do any sort of useful benchmark
(even if zd's static content benchmark had any meaning) without
understanding exactly what is going on on the system and why it is. This
isn't particular to Unix, or NT, or anything, but a simple reflection
of the fact that current benchmarks are so far from any sort of real
life situation and that there are so many little things that significantly
impact results either way and give a skewed picture.
Their numbers are not consistent with anything I have seen in the past,
and it seems very... odd that their claimed peak was exactly 1000
connections/sec.
On Tue, 13 Apr 1999, Daniel J. Cody wrote:
> Was just checking out a link from slashdot. This site makes comparisons
> between NT/Linux and IIS/Apache including,
>
> "The Linux/Apache Web server performance collapsed with a
> WebBench load above 160 client test threads, while Windows NT Server/IIS
> continued
> to increase performance up through 288 client test threads without
> reaching their peak
> performance,"
>
> http://www.newsalert.com/bin/story?StoryId=CnXlbqbWbu0zuvta3o&FQ=linux&SymHdl=1&Nav=na-search-&StoryTitle=linux
> (wrap)
>
> Thoughts?
>
> --
> Daniel J. Cody
> -Senior Web Developer-
> Oracular Systems & Software, Inc.
> dcody@oracular.com
>
Re: Holy FUD Batman!
Posted by Dirk-Willem van Gulik <di...@jrc.it>.
Marc Slemko wrote:
> >The problem: the server performs well, delivering in excess of 1300 HTTP
> >GET requests per second. But then performance drops WAAAY off, like down
> >to 70 connections per second. We're not swapping, the network isn't
> >saturated (4 x 100Mbit nets), disks are hardly used, but the system is
> >just crawling. If it were saturated then performance should level off,
> >not drop like this. Neither vmstat nor top show anything unusual. No
> >error messages in the web server. Its puzzling.
Actually I have seen the same, specifically with RedHat 5.1. We kind of did not
had the time to investigate; so we installed FreeBSD, got rid of the problem,
and kind of schrugged (assuming it we should have spend a patch or something,
but we needed a <30minute solution :-( ). No swapping, no unusual behaviour
except for two things; a lot of kernel time was spend in (ide) interup handling
(even though the disk which counted are all scsi, i.e. swap and content; only
the boot disk was IDE) and the kernel buffers seemed allocated weirdly between
tcp and exfs. The other thing which was confirming was the 'sharpness' of the cut-off
moment; which coincieded with the interupt going up; followed by the free kernel
buffer ditching. But then, I am no RH expert, so we might have done something
way wrong.
Dw
Re: Holy FUD Batman!
Posted by un...@riverstyx.net.
Actually, they're all documented in the Documentation directory of the
kernel source. If they weren't Linux experts running the tests, well..
they were obviously NT experts. I hardly think that makes a fair
comparison. I've seen this happen with nothing in the Apache error logs,
but dmesg will usually report something. I'm assuming they were at
console, but if they were in X or something they might not have seen the
messages...
---
tani hosokawa
river styx internet
On Wed, 14 Apr 1999, Marc Slemko wrote:
> On Wed, 14 Apr 1999 unknown@riverstyx.net wrote:
>
> > Something just occurred to me reading that dejanews post... On Linux, last
> > I checked, NR_TASKS=512 means MAX_TASKS_PER_USER is only 256 (NR_TASKS/2).
> > So, with MaxClients=460, your server will pretty much just "pop". Ends up
> > with a bunch of screwed up processes that sometimes don't respond to
> > signals (even KILL).
>
> There are many such possibilities. Unfortunately, Linux has some rather
> low and poorly documented limits in this area.
>
> Such problems, as long as Linux handles them properly by having
> fork() fail or something, should have been reported in the error logs
> by Apache. They claim there was nothing there.
>
Re: Holy FUD Batman!
Posted by Marc Slemko <ma...@znep.com>.
On Wed, 14 Apr 1999 unknown@riverstyx.net wrote:
> Something just occurred to me reading that dejanews post... On Linux, last
> I checked, NR_TASKS=512 means MAX_TASKS_PER_USER is only 256 (NR_TASKS/2).
> So, with MaxClients=460, your server will pretty much just "pop". Ends up
> with a bunch of screwed up processes that sometimes don't respond to
> signals (even KILL).
There are many such possibilities. Unfortunately, Linux has some rather
low and poorly documented limits in this area.
Such problems, as long as Linux handles them properly by having
fork() fail or something, should have been reported in the error logs
by Apache. They claim there was nothing there.
Re: Holy FUD Batman!
Posted by un...@riverstyx.net.
Something just occurred to me reading that dejanews post... On Linux, last
I checked, NR_TASKS=512 means MAX_TASKS_PER_USER is only 256 (NR_TASKS/2).
So, with MaxClients=460, your server will pretty much just "pop". Ends up
with a bunch of screwed up processes that sometimes don't respond to
signals (even KILL).
---
tani hosokawa
river styx internet
On Tue, 13 Apr 1999, Marc Slemko wrote:
> It figures.
>
> See http://www.dejanews.com/[ST_rn=ps]/getdoc.xp?AN=453662156 for what
> someone claims to be their Usenet post describing their problems:
>
> >The problem: the server performs well, delivering in excess of 1300 HTTP
> >GET requests per second. But then performance drops WAAAY off, like down
> >to 70 connections per second. We're not swapping, the network isn't
> >saturated (4 x 100Mbit nets), disks are hardly used, but the system is
> >just crawling. If it were saturated then performance should level off,
> >not drop like this. Neither vmstat nor top show anything unusual. No
> >error messages in the web server. Its puzzling.
>
> Interesting that the above post was posted from:
>
> tide71.microsoft.com
>
> It is pretty silly to have such an obvious unexplained oddity in your
> configuration and not think it is probably a configuration problem.
>
> Three thumbs down. Anyone have any contacts at Mindcraft?
>
Re: Holy FUD Batman!
Posted by Marc Slemko <ma...@znep.com>.
It figures.
See http://www.dejanews.com/[ST_rn=ps]/getdoc.xp?AN=453662156 for what
someone claims to be their Usenet post describing their problems:
>The problem: the server performs well, delivering in excess of 1300 HTTP
>GET requests per second. But then performance drops WAAAY off, like down
>to 70 connections per second. We're not swapping, the network isn't
>saturated (4 x 100Mbit nets), disks are hardly used, but the system is
>just crawling. If it were saturated then performance should level off,
>not drop like this. Neither vmstat nor top show anything unusual. No
>error messages in the web server. Its puzzling.
Interesting that the above post was posted from:
tide71.microsoft.com
It is pretty silly to have such an obvious unexplained oddity in your
configuration and not think it is probably a configuration problem.
Three thumbs down. Anyone have any contacts at Mindcraft?
Re: Holy FUD Batman!
Posted by Rasmus Lerdorf <ra...@lerdorf.on.ca>.
> We don't have a 1 fork per second rate, so the answer would be... no...
> erm... yes... erm... you know.
Uh.. right, I'm thinking pre-1.3 code here.
-Rasmus
Re: Holy FUD Batman!
Posted by Marc Slemko <ma...@znep.com>.
We don't have a 1 fork per second rate, so the answer would be... no...
erm... yes... erm... you know.
On Tue, 13 Apr 1999, Rasmus Lerdorf wrote:
> > StartServers shouldn't have impacted this test, both because of the length
> > of it and because there is a ramp up period at the start during which
> > results aren't recorded. That period is when stuff like this, caching the
> > whole thing in memory, etc. gets sorted out.
>
> But is the ramp-up period long enough to compensate for our 1 fork per
> second rate?
>
> -Rasmus
>
Re: Holy FUD Batman!
Posted by Marc Slemko <ma...@znep.com>.
On Wed, 14 Apr 1999, Shane Owenby wrote:
> On Wed, Apr 14, 1999 at 03:11:09AM -0700, Marc Slemko wrote:
> >
> > That doesn't mean they aren't useful or that they aren't fun to optimize
> > for.
>
> Marc since it seems that you have figured out what the problems are with
> this test are it would be nice if you were to share the
> finding from your research @
>
> http://lwn.net/1999/features/MindCraft.phtml
> [lwn = Linux Weekly News]
>
> The SAMBA guys have already put up some issues there, and
> some Linux kernel guys have also raised some issues about
> the OS config, and how the SMP kernel was configured.
> I suppose that just leaves the Apache config problems.
The thing is, I don't know that there are any real Apache config problems.
Sure, there are lots of things that may result in a 20% difference across
the board, etc. But those aren't the things that matter. The thing that
matters is that Apache fell off at such a low number of connections.
At this piont, based on the information that has been given, I am going to
say that I am not aware of any Apache configuration problems that would
cause this and it appears to be either an OS issue particular to their
setup (eg. running out of some sort of resource) or a kernel issue (eg.
someone somewhere raised some 2.2.2 issues; eg.
http://www.citi.umich.edu/projects/citi-netscape/reports/bforget.html).
Most of the issues raised on places like slashdot, however, are completely
and totally red herrings. One person says MaxRequestsPerChild is way too
low, one says it is way too high, etc. I have not seen any legit problems
with their Apache config that would cause the sort of degredation at the
sorts of loads they were seeing. Things that could be a 30% difference,
sure, but that isn't what matters. IIS can "legitimately" beat Apache in
this sort of benchmark. The problem here is where it dropped off and the
implication that Apache doesn't scale; made worse by the fact that they
claimed they verified it was the web server and not the OS through some BS
method.
Re: Holy FUD Batman!
Posted by Shane Owenby <sh...@raleigh.ibm.com>.
On Wed, Apr 14, 1999 at 03:11:09AM -0700, Marc Slemko wrote:
>
> That doesn't mean they aren't useful or that they aren't fun to optimize
> for.
Marc since it seems that you have figured out what the problems are with
this test are it would be nice if you were to share the
finding from your research @
http://lwn.net/1999/features/MindCraft.phtml
[lwn = Linux Weekly News]
The SAMBA guys have already put up some issues there, and
some Linux kernel guys have also raised some issues about
the OS config, and how the SMP kernel was configured.
I suppose that just leaves the Apache config problems.
I am going to try to simulate the test on my RH5.2/2.2.1 system
a Dual PII 400Mhz with 384Ram. Also I am going to look into
the numbers our test team is using during the 'stress tests'
of Apache on Linux. If I find something interesting I will
post it.
Shane Owenby
IBM-Apache Development
Re: Holy FUD Batman!
Posted by Marc Slemko <ma...@znep.com>.
On Wed, 14 Apr 1999, John Bley wrote:
> On Wed, 14 Apr 1999 unknown@riverstyx.net wrote:
>
> > It was with 1.3.4 which has the exponential fork rate... so it should only
> > take about 11 seconds to get up to 256 processes... The test really didn't
> > seem to simulate anything like a normal webserver that I've ever seen
> > though.
> But they also used:
> MinSpareServers 1
> (whereas the default is 5)
>
> Unless I'm very wrong, the documentation explains that the "exponential"
> rate continues until the MinSpareServers level of spare servers is
> achieved, and since MinSpareServers is 1, this feels rather suboptimal.
Not really; if you have 200 connections being made, then exponential
spawning will continue until 201 servers are running.
> You're absolutely right about the test not seeming to simulate a normal
> webserver, even where "normal" is defined as "benchmark environment."
Facts are that web benchmark numbers today aren't about useful things,
they aren't about trying to simulate real users, they aren't about trying
to simulate real server content, and they aren't about giving people
valuable information that will actually help them in doing planning for
high-traffic sites.
That doesn't mean they aren't useful or that they aren't fun to optimize
for.
Re: Holy FUD Batman!
Posted by un...@riverstyx.net.
I just checked -- even with MinSpareServers 1 and StartServers 1 it works
fine. Ramped up to 242 processes in just a few seconds. So that should
jive fine with the 30 second ramp up time they allowed for. My
understanding was that the exponential increase applied until there was no
more need (or until it tops out at 32 children/sec).
---
tani hosokawa
river styx internet
On Wed, 14 Apr 1999, John Bley wrote:
> On Wed, 14 Apr 1999 unknown@riverstyx.net wrote:
>
> > It was with 1.3.4 which has the exponential fork rate... so it should only
> > take about 11 seconds to get up to 256 processes... The test really didn't
> > seem to simulate anything like a normal webserver that I've ever seen
> > though.
> But they also used:
> MinSpareServers 1
> (whereas the default is 5)
>
> Unless I'm very wrong, the documentation explains that the "exponential"
> rate continues until the MinSpareServers level of spare servers is
> achieved, and since MinSpareServers is 1, this feels rather suboptimal.
> You're absolutely right about the test not seeming to simulate a normal
> webserver, even where "normal" is defined as "benchmark environment."
>
> --
> John Bley - jbb6@acpub.duke.edu
> Duke '99 - English/Computer Science
> Since English is a mess, it maps well onto the problem space,
> which is also a mess, which we call reality. - Larry Wall
>
Re: Holy FUD Batman!
Posted by John Bley <jb...@acpub.duke.edu>.
On Wed, 14 Apr 1999 unknown@riverstyx.net wrote:
> It was with 1.3.4 which has the exponential fork rate... so it should only
> take about 11 seconds to get up to 256 processes... The test really didn't
> seem to simulate anything like a normal webserver that I've ever seen
> though.
But they also used:
MinSpareServers 1
(whereas the default is 5)
Unless I'm very wrong, the documentation explains that the "exponential"
rate continues until the MinSpareServers level of spare servers is
achieved, and since MinSpareServers is 1, this feels rather suboptimal.
You're absolutely right about the test not seeming to simulate a normal
webserver, even where "normal" is defined as "benchmark environment."
--
John Bley - jbb6@acpub.duke.edu
Duke '99 - English/Computer Science
Since English is a mess, it maps well onto the problem space,
which is also a mess, which we call reality. - Larry Wall
Re: Holy FUD Batman!
Posted by un...@riverstyx.net.
It was with 1.3.4 which has the exponential fork rate... so it should only
take about 11 seconds to get up to 256 processes... The test really didn't
seem to simulate anything like a normal webserver that I've ever seen
though.
---
tani hosokawa
river styx internet
On Tue, 13 Apr 1999, Rasmus Lerdorf wrote:
> > StartServers shouldn't have impacted this test, both because of the length
> > of it and because there is a ramp up period at the start during which
> > results aren't recorded. That period is when stuff like this, caching the
> > whole thing in memory, etc. gets sorted out.
>
> But is the ramp-up period long enough to compensate for our 1 fork per
> second rate?
>
> -Rasmus
>
Re: Holy FUD Batman!
Posted by Rasmus Lerdorf <ra...@lerdorf.on.ca>.
> StartServers shouldn't have impacted this test, both because of the length
> of it and because there is a ramp up period at the start during which
> results aren't recorded. That period is when stuff like this, caching the
> whole thing in memory, etc. gets sorted out.
But is the ramp-up period long enough to compensate for our 1 fork per
second rate?
-Rasmus
Re: Holy FUD Batman!
Posted by Marc Slemko <ma...@znep.com>.
On Tue, 13 Apr 1999, Rasmus Lerdorf wrote:
> - Their NT server had 1024M while their Linux box had 960M (why did they
> put 4G in the box if neither OS used it? Or am I confused?)
Doesn't make much sense to me.
> - tcp windowsize was set to 65535 on NT which is nice on a perfect
> network, but not really what you would use in the real world. They
> did not make the corresponding change on the Linux side
> - They only tested static html and I think only the same static
> file over and over again which plays right into IIS's cacheing.
No, it is actually a reasonably (but still far too small) set of static
files. Last I checked, it was still of an order to easily fit into RAM.
> - IIS log entries were written to a RAID disk once every time the
> accumulated log data hit 100 megs. With Apache logs were written
> on each hit to a non-Raid disk
> - Apache config: StartServers 10 ?? They complained that they had
> sent messages to newsgroups and scoured the Net for information, yet
> they set StartsServers to 10 which is the most basic benchmark screwup
> you can make and it is clearly documented why this is so on the Apache
> site.
StartServers shouldn't have impacted this test, both because of the length
of it and because there is a ramp up period at the start during which
results aren't recorded. That period is when stuff like this, caching the
whole thing in memory, etc. gets sorted out.
They did actually address a lot of issues "properly" that are often
ignored or details often ommitted from reports.
Re: Holy FUD Batman!
Posted by Rasmus Lerdorf <ra...@lerdorf.on.ca>.
> Thoughts?
I read through the Mindcraft test quickly. Some things that stood out for
me. The URL is: http://www.mindcraft.com/whitepapers/nts4rhlinux.html
- Their NT server had 1024M while their Linux box had 960M (why did they
put 4G in the box if neither OS used it? Or am I confused?)
- tcp windowsize was set to 65535 on NT which is nice on a perfect
network, but not really what you would use in the real world. They
did not make the corresponding change on the Linux side
- They only tested static html and I think only the same static
file over and over again which plays right into IIS's cacheing.
- IIS log entries were written to a RAID disk once every time the
accumulated log data hit 100 megs. With Apache logs were written
on each hit to a non-Raid disk
- Apache config: StartServers 10 ?? They complained that they had
sent messages to newsgroups and scoured the Net for information, yet
they set StartsServers to 10 which is the most basic benchmark screwup
you can make and it is clearly documented why this is so on the Apache
site.
And I am sure there are a bunch of other things they did to skew the
results. After all, if you read their whitepaper closely you will see:
Mindcraft, Inc. conducted the performance tests described in this report
between March 10 and March 13, 1999. Microsoft Corporation sponsored the
testing reported herein.
They also seemed to know a heck more about NT than Linux. Some of the NT
tweaks they describe are way beyond me (which doesn't mean much given the
little I know about NT) while their Linux tweaks, if you can call them
that, are rather odd. Why go out of their way to compile Apache with -O4
-m486 for example? And why turn on NFS filesystem support? And why did
they have kerneld running? And they saw a sharp dropoff at 160 concurrent
clients which to me would indicate some sort of OS-level resource being
exhausted at this point. A little bit of investigation should have
cleared that up.
-Rasmus
Re: Holy FUD Batman!
Posted by un...@riverstyx.net.
I don't see how that's possible. I've got servers doing way over 5.9
Mbps, and they're only PII-400 single processor machines, 384 megs RAM.
And they're doing a bunch of CGI on top of it. Those look like the kind
of numbers the FreeBSD people were spewing on the freebsd-chat list.
They'd somehow screwed up getting Linux into SMP mode... I mean, really.
4 Xeon-400 processors?
"Windows NT Server and Internet Information Server (IIS) 4.0 reach a peak
of 3,771 requests/second and 22.4 Megabits/second(Mbps) of throughput. The
report goes on to show that Linux and the Apache 1.3.4 Web server reach a
peak of 1,000 request/second and 5.9 Mbps of throughput."
Looks like a factor of about 4 there to me. I've used IIS server before,
and the performance was NOWHERE near that of Apache. How odd. I wonder
if they're intentionally screwing up...
---
tani hosokawa
river styx internet
On Tue, 13 Apr 1999, Daniel J. Cody wrote:
> Was just checking out a link from slashdot. This site makes comparisons
> between NT/Linux and IIS/Apache including,
>
> "The Linux/Apache Web server performance collapsed with a
> WebBench load above 160 client test threads, while Windows NT Server/IIS
> continued
> to increase performance up through 288 client test threads without
> reaching their peak
> performance,"
>
> http://www.newsalert.com/bin/story?StoryId=CnXlbqbWbu0zuvta3o&FQ=linux&SymHdl=1&Nav=na-search-&StoryTitle=linux
> (wrap)
>
> Thoughts?
>
> --
> Daniel J. Cody
> -Senior Web Developer-
> Oracular Systems & Software, Inc.
> dcody@oracular.com
>