more on mod_dld_elf.c

[Andrew <an...@aaaaaaaa.demon.co.uk>]

Spoke to David about mod_dld_elf, he has some weird ideas for
modifying the core/API of the server...

Ay.

Forwarded message:
> From dhankins@hcf.aloha.net Sat Mar  9 13:36:55 1996
> Date: Sat, 9 Mar 1996 13:36:47 GMT
> From: dhankins@hcf.aloha.net
> Message-ID: <82...@hcf.aloha.net>
> Apparently-To: <an...@aaaaaaaa.demon.co.uk>
> 
> 	id m0tvOXO-000rbcC; Sat, 9 Mar 96 03:18 HST 
> 	   |              ( ( (             ...           ___      
> 	   |.===.       '. ___ .'      o,*,(o o)         _|_|_  Hi Julian! 
> 	   {}o o{}     '  (> <) '     8(o o)(_)Ooo       (o o)     
> 	ooO--(_)--Ooo-ooO--(_)--Ooo-ooO-(_)---Ooo----ooO--(_)--Ooo-
> Message-Id: <m0...@hcf.aloha.net>
> From: dhankins@hcf.aloha.net (David W. Hankins)
> Subject: Re: mod_dld_elf (fwd)
> To: andrew@aaaaaaaa.demon.co.uk (Andrew)
> Date: Sat, 9 Mar 1996 03:18:22 -1000 (HST)
> In-Reply-To: <19...@aaaaaaaa.demon.co.uk> from "Andrew" at Mar 9, 96 11:50:09 am
> X-Mailer: ELM [version 2.4 PL24 PGP3 *ALPHA*]
> MIME-Version: 1.0
> Content-Type: text/plain; charset=US-ASCII
> Content-Transfer-Encoding: 7bit
> Content-Length: 7542      
> 
> Forward this to the others...not sure what your mailing list name is...
> 
> Traffic from Andrew:
> > You can start the server running and then modify the configuration
> > files and send -HUPs to the server to make it reread the module
> > configuration.  The server responds by looking to see which additional
> > modules have been added to the run-time specification and then
> > linking them into place without halting the server.  The dl libraries
> > also allow whole modules to be removed, again following a HUP.
> > Removing modules at run-time, without downing the server is the
> > new functionality that is being exploited in this new version of
> > the module
> 
> Actually, at the moment, it's not intelligent enough to determine what's
> new, or what's missing...I took the shotgun approach and just had it
> re-link everything on HUP.
> 
> So, at the moment, everything is unlinked and removed from the list of
> active modules, then modules are added back in as it receives AddModule
> commands.
> 
> Simple linear list that keeps track of what modules to remove when it
> starts receiving new addmodule commands...you could store the list
> somewhere, and check it when you're adding new modules to see if you
> really need to relink.  Then do an intersection of the two lists (figure
> out what was linked in that isn't in the new 'required' list) and remove
> the listed modules...
> 
> > There are probably may different ways this functionality can be
> > exploited, including:
> > 
> > 1)	never having to down a server when you get hold of a new module
> > 	that you want to play around with
> > 
> > 2)	being able to tune the 'size' of the server - removing 
> > 	unused modules to reduce the process size
> > 
> > 3)	[rather exotic] making the server use a base level of modules 
> > 	it always supports but then giving it the ability to load
> > 	up extra modules to service certain requests, and then
> > 	remove them immediately.  For example the mod_imap.c module
> > 	would only be linked into the process image when someone clicked
> > 	on an image.
> > 
> > 	This doesn't sound as farfetched a procedure as it first
> > 	seems.  Imagine in a years time, when the number of available
> > 	modules has doubled, how much larger the server is going
> > 	to be because of all the damned moules it has to support
> > 	even though some of them are only gonna be used occassionally.
> 
> One could, reasonably, write a rather exotic mod_dld clone that builds its
> own module definition structure as a union of all the modules you want to
> load, then dynamically links in, configures, uses, and unlinks modules on
> certain time-outs.  `AddModule 60 mod_name.so mod_name'
> 
> Anyhow, to add to your list, there was that thing you mentioned to me in
> virtual-person the other day...the security issue.
> 
> It may be a decision of a web admin to remove the compiler (or cripple)
> from a web server...should someone gain entry, it would be one tool they
> would not have the advantage of.
> 
> Further than this, on platforms like Irix where compilers are like $1200
> to be ADDED ON TOP OF the base operating system, you could save yourself
> a pretty good load of money.
> 
> Of course, I have no idea if Irix uses libdl...it is ELF, so the chances
> are pretty good...
> 
> Then there's that friend of mine who broke his compiler and couldn't put
> a simple counter together...if his core was dl capable, I could have just
> compiled mod_counter for him and sent it to him.
> 
> > On my understanding of the dld module's present implementation
> > there is no facility which would allow the server itself to decide
> > which modules it needed, it still requires human intervention to
> > -HUP the beast following a change to the config.
> 
> Yes.  The first mod_dld was a proof-of-concept using gnu dld to dynamically
> link *in* modules at startup...this goes the step further to prove the
> concept of unlinking modules that are no longer to be used.
> 
> It is something of a hack, but it gets the job done.  I'd suggest a few
> revisions to the core server before really spending the time on a fully
> blown 'unlink old modules then relink when in need' version.
> 
> For example, (and I'm only slightly familiar with the core daemon, so forgive
> me), I'm not sure how to determine when a HUP signal is sent...so I used the
> hack that the original dld author used...
> 
> Also, when you are unlinking modules for a short-term to relink them back in
> when you receive webhits that use them, the only way I see right now is to
> cache the configuration commands meant for that module in RAM, then send them
> all when you link in...it would be more optimal if you could have the module
> export its configuration into something mod_dld could hang on to...having the
> `set configuration' hook return a void pointer is not a far off idea.
> 
> Then just have the replacement mod_dld free the pointer when the daemon
> receives a hup...and add something to the core to take a 'use this void
> pointer'...
> 
> Commands could be cached in memory waiting for use of a module before
> configuring the module, then forgotten until the next HUP, etc.
> 
> Possibly a tad more work than you're willing to go through...but considering
> the plan to move to a multithreaded-capable version...might as well go all
> the way, or something.
> 
> > Perhaps a way of registering the capabilities required of the server
> > for various URLs could be used, as in this fragment of imaginary
> > configuration:
> > 
> >     # pages under /Projects/LargeUserbase are protected by a .htaccess
> >     # file that uses DBM authentication.  Any URL under there will need
> >     # modules supporting dbm authentication to be in place
> >     CapabilityRequired /Projects/LargeUserbase/ auth_dbm
> >     # We have a few image map files, let a module handle them
> >     CapabilityRequired *.map image_map
> > 
> >     # map real modules to the abstract capabilities they offer the
> >     # server
> >     CapabilityOffered auth_dbm_module auth_dbm
> >     CapabilityOffered imap_module image_map
> > 
> > Do you see how this sort of approach could let the server tune
> > itself given a few hints from the webadmin?
> 
> This is something I hadn't first thought of when I (re)wrote the code.  It
> is perfectly implementable.
> 
> I'd tend to put in explicit times for timeouts, though, so the daemon can
> leave things linked in when you receive a bunch of hits in the same area.
> 
> > > P.S. BTW, it doesn't compile.  Does it need anything other just being
> > > added to the Configuration file?
> > 
> > Hard to say.  It compiles on the author's (David Hankins) original
> > Linux platform (I watched him doing it) and it ports to FreeBSD by
> > changing a couple of parameters to dl library calls.  Off the top
> > of my head I recall BSDs not supporting the RTLD_LAZY parameter in
> > dlopen.  Changing such calls to:
> 
> The dl* calls are part of the 'libdl' package that was ported to Linux
> from Solaris (the API from Solaris was used).  BSD seems to implement an
> older version of this library...
> 
> GNU DLD is what was originally used, but GNU DLD never had an option to
> unlink libraries.  I don't blame the developers...they have to deal with
> a.out and qmagic.
> 
> The next logical step is to update the code for BSD so you don't have
> to edit it manually again, and to merge mod_dld and mod_dld_elf...
> 
> > I'm not aware of other changes needed to get it spinning on your
> > own platform, but it wouldn't hurt to mail the author directly,
> > he's pretty clued.
> 
> I don't know about that last bit, but I'm usually happy to help in any
> way I can.
> 
> -- 
> David Hankins,             "If you don't do it right the first time,
> Mercenary Spider			you'll just have to do it again."
> The Huddled Masses (Inc)				-- J.T. Hankins
>