You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Dave O'Neill <dm...@ccs.carleton.ca> on 1998/07/29 22:29:24 UTC
Patch/kludge to mod_userdir
Hello all... another lurker decloaking here.
Anyway, I've hacked mod_userdir to cache a runtime-configurable number of
getpwnam entries internally. This was done because our NIS server was
handling a huge number of requests from one of our web servers, and we
needed some quick and dirty way to reduce this. The change seems to
improve performance, though I haven't had time to benchmark it at all.
The question is, would it be worthwhile for me to clean it up (it's not
too pretty right now) and submit it for inclusion? Any
suggestions/comments are welcome.
Dave
--
Dave O'Neill dmo@ccs.carleton.ca
Computing and Communications Services, Carleton University
"You, I'm sorry to say, are in need of a geek." - Rob Russell
Re: Patch/kludge to mod_userdir
Posted by "James H. Cloos Jr." <cl...@jhcloos.com>.
>>>>> "Dave" == Dave O'Neill <dm...@ccs.carleton.ca> writes:
Dave> Anyway, I've hacked mod_userdir to cache a runtime-configurable
Dave> number of getpwnam entries internally. This was done because
Dave> our NIS server was handling a huge number of requests from one
Dave> of our web servers, and we needed some quick and dirty way to
Dave> reduce this. The change seems to improve performance, though I
Dave> haven't had time to benchmark it at all.
What OS are you running? The better NIS implementations have a daemon
running on each client that does this globably for all processes on
that client. Its name, though, escapes me at the moment. [SIGH]
SunOS 5 definately has it, as does (will) glibc 2.1.
-JimC
--
James H. Cloos, Jr.
<cl...@jhcloos.com>
Re: Patch/kludge to mod_userdir
Posted by Marc Slemko <ma...@worldgate.com>.
No, but if you have a limited number of filesxx and machinename and yy,
you can use a huge list of entries in your UserDir directive to make it
try each one. That is quite inefficient though.
On Thu, 30 Jul 1998, Dan Jacobowitz wrote:
> On Thu, Jul 30, 1998 at 04:06:54PM -0400, Dave O'Neill wrote:
> > It would, if we could do that... on one of our main servers (15000
> > student accounts), the home directories are located in a rather bizzare
> > directory structure (/filesXX/<machinename>/YY/<username>, where XX and YY
> > are two different numbers) that's completely unfindable without a
> > getpwent().
>
> I'm assuming /files??/*/??/*/public_html wouldn't do the trick?
>
> Dan
>
Re: Patch/kludge to mod_userdir
Posted by Dan Jacobowitz <dr...@false.org>.
On Thu, Jul 30, 1998 at 04:06:54PM -0400, Dave O'Neill wrote:
> It would, if we could do that... on one of our main servers (15000
> student accounts), the home directories are located in a rather bizzare
> directory structure (/filesXX/<machinename>/YY/<username>, where XX and YY
> are two different numbers) that's completely unfindable without a
> getpwent().
I'm assuming /files??/*/??/*/public_html wouldn't do the trick?
Dan
Re: Patch/kludge to mod_userdir
Posted by Dean Gaudet <dg...@arctic.org>.
On Thu, 30 Jul 1998, Dave O'Neill wrote:
>
> On Wed, 29 Jul 1998, Dean Gaudet wrote:
>
> > Don't you see a large blowup in RAM usage under heavy load? each child
> > has to cache the names individually...
>
> Not really... the cache is a linked list, with each node containing a 9
> character string (the username), pointers to previous and next nodes, and
> a pointer to a passwd struct... it adds up to about 61 bytes per node.
> Even with 200 children running, and a 50-node cache for each, it's only
> ~600K of total cache.
> Like I said before, though, there may be better ways to do it.
That would be larger than the L2 ... with lots of duplicated data... it'd
really affect the machine's performance.
Another option is to download the entire passwd table at restart time, and
then send SIGUSR1 to the server once an hour or so. (Of course if you've
got multiple servers you want to stagger their restarts so you don't melt
the NIS server on the hour :)
> > Under high loads I'd think something like "UserDir /home/*/public_html"
> > would be far more effective. Then the cache is in the filesystem and
> > shared between all processes.
>
> It would, if we could do that... on one of our main servers (15000
> student accounts), the home directories are located in a rather bizzare
> directory structure (/filesXX/<machinename>/YY/<username>, where XX and YY
> are two different numbers) that's completely unfindable without a
> getpwent().
Right yeah I figured something like this. I still tend to prefer symlink
trees to deal with it... /home/a/b/abner or somesuch. mod_rewrite handles
multilevel hashes like that pretty well.
Dean
Re: Patch/kludge to mod_userdir
Posted by Marc Slemko <ma...@worldgate.com>.
On Thu, 30 Jul 1998, Dave O'Neill wrote:
> > Under high loads I'd think something like "UserDir /home/*/public_html"
> > would be far more effective. Then the cache is in the filesystem and
> > shared between all processes.
>
> It would, if we could do that... on one of our main servers (15000
> student accounts), the home directories are located in a rather bizzare
> directory structure (/filesXX/<machinename>/YY/<username>, where XX and YY
> are two different numbers) that's completely unfindable without a
> getpwent().
Automatic generation of a mapping of usernames to homedirs every x hours,
then use mod_rewrite or make symlinks and use normal userdir stuff?
Re: Patch/kludge to mod_userdir
Posted by Dave O'Neill <dm...@ccs.carleton.ca>.
On Wed, 29 Jul 1998, Dean Gaudet wrote:
> Don't you see a large blowup in RAM usage under heavy load? each child
> has to cache the names individually...
Not really... the cache is a linked list, with each node containing a 9
character string (the username), pointers to previous and next nodes, and
a pointer to a passwd struct... it adds up to about 61 bytes per node.
Even with 200 children running, and a 50-node cache for each, it's only
~600K of total cache.
Like I said before, though, there may be better ways to do it.
> Under high loads I'd think something like "UserDir /home/*/public_html"
> would be far more effective. Then the cache is in the filesystem and
> shared between all processes.
It would, if we could do that... on one of our main servers (15000
student accounts), the home directories are located in a rather bizzare
directory structure (/filesXX/<machinename>/YY/<username>, where XX and YY
are two different numbers) that's completely unfindable without a
getpwent().
Dave
--
Dave O'Neill dmo@ccs.carleton.ca
Computing and Communications Services, Carleton University
"You, I'm sorry to say, are in need of a geek." - Rob Russell
Re: Patch/kludge to mod_userdir
Posted by Dean Gaudet <dg...@arctic.org>.
Don't you see a large blowup in RAM usage under heavy load? each child
has to cache the names individually...
Under high loads I'd think something like "UserDir /home/*/public_html"
would be far more effective. Then the cache is in the filesystem and
shared between all processes.
Dean
On Wed, 29 Jul 1998, Dave O'Neill wrote:
>
> Hello all... another lurker decloaking here.
>
> Anyway, I've hacked mod_userdir to cache a runtime-configurable number of
> getpwnam entries internally. This was done because our NIS server was
> handling a huge number of requests from one of our web servers, and we
> needed some quick and dirty way to reduce this. The change seems to
> improve performance, though I haven't had time to benchmark it at all.
>
> The question is, would it be worthwhile for me to clean it up (it's not
> too pretty right now) and submit it for inclusion? Any
> suggestions/comments are welcome.
>
> Dave
> --
> Dave O'Neill dmo@ccs.carleton.ca
> Computing and Communications Services, Carleton University
> "You, I'm sorry to say, are in need of a geek." - Rob Russell
>
>
>
>