You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Juergen Schmidt <ju...@ct.heise.de> on 1999/05/17 12:52:08 UTC
Bad apache performance with linux-SMP
Hello,
first of all, please excuse this off-topic posting. I'm doing a test
with apache on a 4CPU-Box (Siemens Primergy 870) with Linux/apache and
NT/IIS. Now I got some *very* strange results and want to make sure,
that you -- the developers -- have the occasion to comment on them,
*before* I publish them.
If there's a better way to do this, please feel free to tell me.
So now to the facts:
I've got a Siemens Primergy 870 with:
4 CPUs PII Xeon 450 MHz
2 GByte RAM
Mylex RAID Controler DAC 960 (64MB RAM), RAID5
2 x Intel EEpro 100 (only one is used, the other is not ifconfig'ed but
detected)
Linux 2.2.8 (SuSE 6.1., glibc based)
Apache 1.3.6
(NT 4.0 SP4, IIS 4.0)
Network: switched 100 MBit/s, half duplex
Neither system is intensivly tuned -- apart from the obvious stuff, like
disabling not needed modules, turning off hostname lookups, ...
I'm measuring plain HTTP-GET on a static html-file with 8
(Linux-)clients each running up to 64 processes (for a total of 512)
doing HTTP-GET-requests in a tight loop. All files come from a partition
on the RAID-array (RAID 5) which is used for logging, too. File size is
4KByte (I measured 1k and 8k with similar results).
Results:
-- with one CPU both NT/IIS and Linux/apache deliver about the same
perfomance (+- 10%). Even Linux-SMP and Non-SMP only differ by < 10%.
-- with 4 CPUs NT/IIS gives sligthly more Reqeusts/sec (< 10%): that's
mainly because my setup doesn't support real heavy load by doing plain
HTTP-GETs. One CPU obviously is sufficient for this task here (as in
most real life setups with plain HTML serving)
-- Linux with 4 CPUs is disastrous: It delivers significantly less RPS
than the single-CPU version -- about a factor of 4 ! Only at high loads
(256 procs und up) it catches up.
One other strange thing (but I still have to doublecheck this):
- Linux 4 CPUs: I get slightly better perfomance if the processes fetch
a random page (out of 10000 files, so that they still fit in the buffer
cache). All the other combinations (Linux 1 CPU, NT) are a bit slower in
comparison to fetching only one page.
For clarity I included a picture (it says more than a lot of words.
The red line is the perfomance of linux with 4 CPUs, blue with random
files, green Linux one CPU, pink is NT 1 CPU (4 CPUS are slightly above
the linux curve). The drop at 512 processes is due to connect()-failures
-- I'm currently investigating this too.
Do you have any ideas, what's happening there?
Or even better, how to fix this?
It seems to me, that this might exactly be the factor 4 those mindcraft
people have measured. Do you think, this is possible?
bye, juergen
BTW: In another test with cgi-scripts I set NR_OPEN in the linux kernel
from 1024 to 2048 (in include/linux/limits.h, fs.h and posix_types.h)
and recompiled apache.
So I got rid of the open() errors that occured under heavy load. But
therefore I get only about half the rps with <=128 processes. Did I
forget something?
PS: Please CC me per mail, as I have not subscribed to the list.
Juergen Schmidt Redakteur/editor c't magazin PGP-Key available
Verlag Heinz Heise GmbH & Co KG, Helstorferstr. 7, D-30625 Hannover
EMail: ju@ct.heise.de - Tel.: +49 511 5352 300 - FAX: +49 511 5352 417
Re: Bad apache performance with linux-SMP
Posted by Juergen Schmidt <ju...@ct.heise.de>.
On Wed, 19 May 1999, Dan Kegel wrote:
> Please keep us informed about your benchmark efforts. I'm very
> interested.
The most important news is, that a patch suggested on the linux kernel
mailinglist seems to solve the problem. It removes serialization in the
copy-and-checksum-Routine in tcp.c by calling unlock_kernel() before.
> p.s.
> Are all clients fetching the same file? You didn't seem to specify.
In this test yes -- r3 is a graph with random files out of a set of 10000
> Also, your graph needs better legends, IMHO. I saw it before on the
> web and didn't really grok it.
Those graphs are not ment for publication -- just as a quick explanation
of what I was posting. Please tell this person who put it on the web, to
remove them. Those are *not* final results, ment to be published.
bye, juergen
Juergen Schmidt Redakteur/editor c't magazin PGP-Key available
Verlag Heinz Heise GmbH & Co KG, Helstorferstr. 7, D-30625 Hannover
EMail: ju@ct.heise.de - Tel.: +49 511 5352 300 - FAX: +49 511 5352 417
Re: Bad apache performance with linux-SMP
Posted by Dan Kegel <da...@alumni.caltech.edu>.
Juergen Schmidt wrote:
> 4 CPUs PII Xeon 450 MHz, 2 GByte RAM
> ... 2 x Intel EEpro 100 (only one is used...
> Linux 2.2.8 (SuSE 6.1., glibc based), Apache 1.3.6
> Network: switched 100 MBit/s, half duplex
> [unneeded modules disabled, hostname lookups turned off...]
>
> I'm measuring plain HTTP-GET on a static html-file with 8
> (Linux-)clients each running up to 64 processes (for a total of 512)
> doing HTTP-GET-requests in a tight loop. ... File size is 4KByte ...
>
> ... -- Linux with 4 CPUs is disastrous: It delivers significantly less RPS
> than the single-CPU version -- about a factor of 4 ! Only at high loads
> (256 procs und up) it catches up.
Dean Guadet replied:
> Upgrade to 2.2.9 or later, and do this when configuring apache:
> env CFLAGS='-DSINGLE_LISTEN_UNSERIALIZED_ACCEPT' ./configure
> and ensure you only listen on one socket.
> You shouldn't need to change NR_OPEN.
> I think 2.2.9 has the scheduler fixes. Maybe not.
I don't think 2.2.9 has the wake-one fix, which might be very important.
One wake-one patch is at
ftp://e-mind.com/pub/andrea/kernel/2.2.8_andrea1.bz,
and another is in 2.3.3.
The wake-one patches only help accept(), so it's important to do as Dean
suggests, and make sure Apache sleeps in accept() rather than flock()
or something like that.
See http://www.kegel.com/mindcraft_redux.html for a list of the
issues found and addressed since Mindcraft's tests.
Please keep us informed about your benchmark efforts. I'm very
interested.
MfG
Dan
p.s.
Are all clients fetching the same file? You didn't seem to specify.
Also, your graph needs better legends, IMHO. I saw it before on the
web and didn't really grok it.
Re: Bad apache performance with linux-SMP
Posted by un...@riverstyx.net.
Isn't that OPEN_MAX and not NR_OPEN? But if that's the case, then yeah,
NR_OPEN probably doesn't ever need to be changed :)
---
tani hosokawa
river styx internet
On Wed, 19 May 1999, Dean Gaudet wrote:
>
>
> On Wed, 19 May 1999 unknown@riverstyx.net wrote:
>
> > From what I understand on the kernel list, 2.2.9 does indeed have the
> > scheduler problems fixed. It's easier to change NR_OPEN in
> > /proc/sys/fs/file-max and inode-max.
>
> Those aren't the same:
>
> NR_OPEN is the maximum number of fds a single process can have open at any
> one time.
>
> file-max, and inode-max are settings for the system as a whole
>
> Dean
>
Re: Bad apache performance with linux-SMP
Posted by Dean Gaudet <dg...@arctic.org>.
On Wed, 19 May 1999 unknown@riverstyx.net wrote:
> From what I understand on the kernel list, 2.2.9 does indeed have the
> scheduler problems fixed. It's easier to change NR_OPEN in
> /proc/sys/fs/file-max and inode-max.
Those aren't the same:
NR_OPEN is the maximum number of fds a single process can have open at any
one time.
file-max, and inode-max are settings for the system as a whole
Dean
Re: Bad apache performance with linux-SMP
Posted by un...@riverstyx.net.
>From what I understand on the kernel list, 2.2.9 does indeed have the
scheduler problems fixed. It's easier to change NR_OPEN in
/proc/sys/fs/file-max and inode-max.
---
tani hosokawa
river styx internet
On Wed, 19 May 1999, Dean Gaudet wrote:
> Upgrade to 2.2.9 or later, and do this when configuring apache:
>
> env CFLAGS='-DSINGLE_LISTEN_UNSERIALIZED_ACCEPT' ./configure
>
> and ensure you only listen on one socket.
>
> You shouldn't need to change NR_OPEN.
>
> I think 2.2.9 has the scheduler fixes. Maybe not.
>
> Dean
>
> On Mon, 17 May 1999, Juergen Schmidt wrote:
>
> > Hello,
> >
> > first of all, please excuse this off-topic posting. I'm doing a test
> > with apache on a 4CPU-Box (Siemens Primergy 870) with Linux/apache and
> > NT/IIS. Now I got some *very* strange results and want to make sure,
> > that you -- the developers -- have the occasion to comment on them,
> > *before* I publish them.
> > If there's a better way to do this, please feel free to tell me.
> >
> > So now to the facts:
> >
> > I've got a Siemens Primergy 870 with:
> >
> > 4 CPUs PII Xeon 450 MHz
> > 2 GByte RAM
> > Mylex RAID Controler DAC 960 (64MB RAM), RAID5
> > 2 x Intel EEpro 100 (only one is used, the other is not ifconfig'ed but
> > detected)
> >
> > Linux 2.2.8 (SuSE 6.1., glibc based)
> > Apache 1.3.6
> > (NT 4.0 SP4, IIS 4.0)
> >
> > Network: switched 100 MBit/s, half duplex
> >
> > Neither system is intensivly tuned -- apart from the obvious stuff, like
> > disabling not needed modules, turning off hostname lookups, ...
> >
> > I'm measuring plain HTTP-GET on a static html-file with 8
> > (Linux-)clients each running up to 64 processes (for a total of 512)
> > doing HTTP-GET-requests in a tight loop. All files come from a partition
> > on the RAID-array (RAID 5) which is used for logging, too. File size is
> > 4KByte (I measured 1k and 8k with similar results).
> >
> > Results:
> >
> > -- with one CPU both NT/IIS and Linux/apache deliver about the same
> > perfomance (+- 10%). Even Linux-SMP and Non-SMP only differ by < 10%.
> >
> > -- with 4 CPUs NT/IIS gives sligthly more Reqeusts/sec (< 10%): that's
> > mainly because my setup doesn't support real heavy load by doing plain
> > HTTP-GETs. One CPU obviously is sufficient for this task here (as in
> > most real life setups with plain HTML serving)
> >
> > -- Linux with 4 CPUs is disastrous: It delivers significantly less RPS
> > than the single-CPU version -- about a factor of 4 ! Only at high loads
> > (256 procs und up) it catches up.
> >
> > One other strange thing (but I still have to doublecheck this):
> >
> > - Linux 4 CPUs: I get slightly better perfomance if the processes fetch
> > a random page (out of 10000 files, so that they still fit in the buffer
> > cache). All the other combinations (Linux 1 CPU, NT) are a bit slower in
> > comparison to fetching only one page.
> >
> > For clarity I included a picture (it says more than a lot of words.
> >
> > The red line is the perfomance of linux with 4 CPUs, blue with random
> > files, green Linux one CPU, pink is NT 1 CPU (4 CPUS are slightly above
> > the linux curve). The drop at 512 processes is due to connect()-failures
> > -- I'm currently investigating this too.
> >
> > Do you have any ideas, what's happening there?
> > Or even better, how to fix this?
> >
> > It seems to me, that this might exactly be the factor 4 those mindcraft
> > people have measured. Do you think, this is possible?
> >
> > bye, juergen
> >
> > BTW: In another test with cgi-scripts I set NR_OPEN in the linux kernel
> > from 1024 to 2048 (in include/linux/limits.h, fs.h and posix_types.h)
> > and recompiled apache.
> > So I got rid of the open() errors that occured under heavy load. But
> > therefore I get only about half the rps with <=128 processes. Did I
> > forget something?
> >
> > PS: Please CC me per mail, as I have not subscribed to the list.
> >
> > Juergen Schmidt Redakteur/editor c't magazin PGP-Key available
> > Verlag Heinz Heise GmbH & Co KG, Helstorferstr. 7, D-30625 Hannover
> > EMail: ju@ct.heise.de - Tel.: +49 511 5352 300 - FAX: +49 511 5352 417
>
Re: Bad apache performance with linux-SMP
Posted by Dan Kegel <da...@alumni.caltech.edu>.
Juergen Schmidt wrote:
> FYI: I received a patch for tcp.c in the linux kernel which changes the
> results dramatically. ...
> Have a look at the thread in the linux-kernel list.
Thanks for keeping us up to date. I've added your experience
to http://www.kegel.com/mindcraft_redux.html#smp
- Dan
Re: Bad apache performance with linux-SMP
Posted by Juergen Schmidt <ju...@ct.heise.de>.
On Wed, 19 May 1999, Dean Gaudet wrote:
> Upgrade to 2.2.9 or later, and do this when configuring apache:
Thanks for your comments.
FYI: I received a patch for tcp.c in the linux kernel which changes the
results dramatically. It does an unlock_kernel() before and a
lock_kernel() after the copy_and_checksum-Routine. This avoids the
serialization at this point.
As a result I got almost the same performance with 4 CPUs as with one
(which is reasonable, because the load is not enough for more than one CPU
-- the NT/IIS-results are similar).
Have a look at the thread in the linux-kernel list.
bye, juergen
Juergen Schmidt Redakteur/editor c't magazin PGP-Key available
Verlag Heinz Heise GmbH & Co KG, Helstorferstr. 7, D-30625 Hannover
EMail: ju@ct.heise.de - Tel.: +49 511 5352 300 - FAX: +49 511 5352 417
Re: Bad apache performance with linux-SMP
Posted by Dean Gaudet <dg...@arctic.org>.
Upgrade to 2.2.9 or later, and do this when configuring apache:
env CFLAGS='-DSINGLE_LISTEN_UNSERIALIZED_ACCEPT' ./configure
and ensure you only listen on one socket.
You shouldn't need to change NR_OPEN.
I think 2.2.9 has the scheduler fixes. Maybe not.
Dean
On Mon, 17 May 1999, Juergen Schmidt wrote:
> Hello,
>
> first of all, please excuse this off-topic posting. I'm doing a test
> with apache on a 4CPU-Box (Siemens Primergy 870) with Linux/apache and
> NT/IIS. Now I got some *very* strange results and want to make sure,
> that you -- the developers -- have the occasion to comment on them,
> *before* I publish them.
> If there's a better way to do this, please feel free to tell me.
>
> So now to the facts:
>
> I've got a Siemens Primergy 870 with:
>
> 4 CPUs PII Xeon 450 MHz
> 2 GByte RAM
> Mylex RAID Controler DAC 960 (64MB RAM), RAID5
> 2 x Intel EEpro 100 (only one is used, the other is not ifconfig'ed but
> detected)
>
> Linux 2.2.8 (SuSE 6.1., glibc based)
> Apache 1.3.6
> (NT 4.0 SP4, IIS 4.0)
>
> Network: switched 100 MBit/s, half duplex
>
> Neither system is intensivly tuned -- apart from the obvious stuff, like
> disabling not needed modules, turning off hostname lookups, ...
>
> I'm measuring plain HTTP-GET on a static html-file with 8
> (Linux-)clients each running up to 64 processes (for a total of 512)
> doing HTTP-GET-requests in a tight loop. All files come from a partition
> on the RAID-array (RAID 5) which is used for logging, too. File size is
> 4KByte (I measured 1k and 8k with similar results).
>
> Results:
>
> -- with one CPU both NT/IIS and Linux/apache deliver about the same
> perfomance (+- 10%). Even Linux-SMP and Non-SMP only differ by < 10%.
>
> -- with 4 CPUs NT/IIS gives sligthly more Reqeusts/sec (< 10%): that's
> mainly because my setup doesn't support real heavy load by doing plain
> HTTP-GETs. One CPU obviously is sufficient for this task here (as in
> most real life setups with plain HTML serving)
>
> -- Linux with 4 CPUs is disastrous: It delivers significantly less RPS
> than the single-CPU version -- about a factor of 4 ! Only at high loads
> (256 procs und up) it catches up.
>
> One other strange thing (but I still have to doublecheck this):
>
> - Linux 4 CPUs: I get slightly better perfomance if the processes fetch
> a random page (out of 10000 files, so that they still fit in the buffer
> cache). All the other combinations (Linux 1 CPU, NT) are a bit slower in
> comparison to fetching only one page.
>
> For clarity I included a picture (it says more than a lot of words.
>
> The red line is the perfomance of linux with 4 CPUs, blue with random
> files, green Linux one CPU, pink is NT 1 CPU (4 CPUS are slightly above
> the linux curve). The drop at 512 processes is due to connect()-failures
> -- I'm currently investigating this too.
>
> Do you have any ideas, what's happening there?
> Or even better, how to fix this?
>
> It seems to me, that this might exactly be the factor 4 those mindcraft
> people have measured. Do you think, this is possible?
>
> bye, juergen
>
> BTW: In another test with cgi-scripts I set NR_OPEN in the linux kernel
> from 1024 to 2048 (in include/linux/limits.h, fs.h and posix_types.h)
> and recompiled apache.
> So I got rid of the open() errors that occured under heavy load. But
> therefore I get only about half the rps with <=128 processes. Did I
> forget something?
>
> PS: Please CC me per mail, as I have not subscribed to the list.
>
> Juergen Schmidt Redakteur/editor c't magazin PGP-Key available
> Verlag Heinz Heise GmbH & Co KG, Helstorferstr. 7, D-30625 Hannover
> EMail: ju@ct.heise.de - Tel.: +49 511 5352 300 - FAX: +49 511 5352 417