apache tuning for mirrors ?

[jason andrade <ja...@dstc.edu.au>]

Hi,

does anyone have ideas on how to tune apache for very large directory
listings ?  this is perhaps out of the direct context of apache
mirroring, but i'm hoping there is a (very thin) signal there :-)

i'm using apache to provide a http interface into our ftp archives
(including the apache mirror) and on some directories (considering
that it's quite a heavily loaded/busy server) it can take several
minutes (5 minutes+) to return a directory listing which has
5,000+ entries in it.

using ftp, it starts streaming almost immediately and the listing
is about 1.5-2M long.. which takes around 30 seconds to complete.

with http, it often times out after a few minutes :-/

regards,

-jason

Re: apache tuning for mirrors ?

[jason andrade <ja...@dstc.edu.au>]

On Wed, 7 Nov 2001, James R Grinter wrote:

> yeah - if you watch to see what the process is actually doing it spends
> a lot of time looking for <foo>.meta files, <foo>/meta/, and all of
> those things. I think (from vague memory) that each file results in 5
> or so stats() of the directory (each of which will take a long time for
> your classic UFS based directory with lots of files in it.)
> 
> You basically need to track down the features that check all that stuff
> and turn them off for your mirror directories.
> 
> (Can't give any more specific pointers than that, cos I never got
> around to looking further at the problem.)

thanks.  i've had a bit of a look and i'm no closer either to finding
out what these values are that i should try and turn off :-/

what you said does make sense.. hmm.  maybe i'll also try to run up an
apache 2.0.18 server on our test box and see if there are any speed
improvements there.

regards,

-jason

Re: apache tuning for mirrors ?

[James R Grinter <jr...@gbnet.net>]

On Wed, Nov 07, 2001 at 03:53:33PM +1000, jason andrade wrote:
> (including the apache mirror) and on some directories (considering
> that it's quite a heavily loaded/busy server) it can take several
> minutes (5 minutes+) to return a directory listing which has
> 5,000+ entries in it.

yeah - if you watch to see what the process is actually doing it spends
a lot of time looking for <foo>.meta files, <foo>/meta/, and all of
those things. I think (from vague memory) that each file results in 5
or so stats() of the directory (each of which will take a long time for
your classic UFS based directory with lots of files in it.)

You basically need to track down the features that check all that stuff
and turn them off for your mirror directories.

(Can't give any more specific pointers than that, cos I never got
around to looking further at the problem.)

James.