[post-1.2] proposed api extension mechanism

[Ian Kluft <ik...@cisco.com>]

Now that some post-1.2 ideas are showing up, I guess it's time for me to
stop lurking and toss in an idea I've been considering...  I haven't been
active on new-httpd yet but I have contributed to mod_rewrite, where I
submitted the "subrequest lookaheads".  I've also written a couple modules
for internal use on servers in Cisco's Central Engineering.

This idea could allow modules to dynamically extend the API with their own
set of functions that other code can use.  In particular, the first thing
I have in mind that could be added with this is a "lookup interface" as a
wrapper for the various DBM, DB, DBMS schemes... and then a modified mod_auth
that would use any of them (and others not yet written) via the new interface.
Other modules, such as mod_rewrite, which currently use DBM could also be
modified so they can use all available lookup methods.

How It Would Work
-----------------
This would require the addition of one field to the "module" type in
http_config.h and some new support functions.  (Plus documentation updates.)

The new field in "module" would be a pointer to a hash structure, or NULL
pointer if not used by the module.  The hash would be keyed on a number or
string representing the "feature" (i.e. lookup interface, authentication
schemes, etc) and return an array of function pointers.  New "features" are
defined as a group by that array of function pointers.  Each feature would
need to specify the number of functions and exact purpose of each in its
array.

Modules would implement features differently using their own functions in
the array.  For example, the lookup interface would define function slots
for open, lookup, close, etc which could each be done differently by modules
for DBM, DB, flat files, LDAP, various DBMS queries, etc.  Other features
would also define what their slots are used for but modules would supply
their own implementations.  So, published feature definitions would become
extensions to the Apache API.

The NULL hash pointer in the module struct is important for compatibility
with existing modules since their module structure would have a NULL filled
in for this field.

I envision these support functions.  (The "feature_id" type is up for
discussion - see implementation choice #2 below.)

* add feature support to a module: (called during module initialization to
  insert a function array in its hash table)

  int register_flex_feature( pool *p, module *mod, feature_id fid,
     array_header *func );

* call a method in every module that supports it: (similar to run_method()
  but checks hashes in each module for support of the feature)

  int run_flex_method( request_rec *r, feature_id fid, int function_index, 
     int run_all, void *param )

Obviously, in order for a feature to get used, there has to be code
(possibly in another module) that calls run_flex_method() for a specific
function from a feature definition.

The initial patch that I would provide would include
* the struct and support functions (API mods) for the Apache core
* a definition for the Lookup Interface feature's functions
* a mod_lookup_htpasswd module to implement htpasswd lookups
* a mod_lookup_dbm module to implement DBM lookups
* a mod_lookup_db module to implement Berkeley DB 1.85 lookups
* a derivative of mod_auth which uses the Lookup Interface feature

(If anyone wants to volunteer to be part of this effort, let me know.
It could go faster if we can farm out some of the pieces.)

I'll also investigate what it might take to patch mod_rewrite so that it can
use the Lookup Interface feature similar to its current dbm functionality.

That should get things started.  More lookup modules can be provided in the
future by anyone (including me.)  Hopefully this will be the last new
derivative of mod_auth and everyone else can add lookup modules. :-)

Implementation Options
----------------------
I'm leaving some of these options open for discussion because there are
tradeoffs.  Also, I realize that since I'm not a member of the Apache Group,
acceptance of the proposal will require convincing members of its merits and
being open to input on these choices.

I'll watch the discussion and then put the design decisions on a web page.
>From that (and possibly more input), I'll start on the proposed patches in
the post-1.2 time frame.  (As with everyone else, this is on the side from
my day job.)

Choice #1: the hashing structure
  Pick one or suggest another alternative:
  a) This could use Apache's "table" structure if the Apache Group is
     amenable to changing "table" to some sort of hashing algorithm in
     2.0.  That would eliminate lots of sequential searches all over the
     server.  (That's a significant factor in making any software scalable!)
     If approved, I would be willing to provide this patch too.

     Possible hashing algorithms could be drawn from Berkeley DB 1.85 or 2.0
     (using in-memory hashing), or Esmond Pitt's public domain dynamic hash
     algorithm from comp.sources.misc volume 6 (with a modification to allow
     multiple instances of the structures.)  I recommend either DB 1.85 or
     2.0, but not both.  Berkeley DB 2.0 is brand new (released in April)
     but much more portable because it uses autoconf.

  b) If Option (a) is not acceptable, this algorithm still needs a hash
     function of its own in order to have variable-size and arbitrarily-keyed
     contents.  I would still suggest DB 1.85 or 2.0 if this alternative is
     needed.
     
In case some platforms can't install the hashing library chosen, it must be
possible to use a preprocessor constant like NO_HAVE_HASHING to omit the
hash library.  In option (a), the existing sequential-search table structs
and functions would be preserved via #ifdefs.  In both cases, NO_HAVE_HASHING
would disable all flexible API extensions.  Sites defining it should still
retain all Apache 1.2 API functionality.

Choice #2: the hash key (feature ID)
  Pick one or suggest another alternative:
  a) The hash key could be an integer representing a specific feature.
     The assignment of these numbers would have to be coordinated by the
     Apache Group.  Using an integer is best for hash search performance
     so I recommend this option.
  b) The hash could also be a string.  It would not require any coordination
     except that the name strings for each feature would have to be unique.
     It could make it easier to understand these changes to the API.  But
     this is slower because of use of strcmp() for hash key matching.
-- 
Ian Kluft  KO6YQ PP-ASEL                                  Cisco Systems, Inc.
ikluft@cisco.com (work)  ikluft@thunder.sbay.org (home)          San Jose, CA

Re: [post-1.2] proposed api extension mechanism

[ra...@bellglobal.com]

> This idea could allow modules to dynamically extend the API with their own
> set of functions that other code can use.  In particular, the first thing
> I have in mind that could be added with this is a "lookup interface" as a
> wrapper for the various DBM, DB, DBMS schemes... and then a modified
> mod_auth that would use any of them (and others not yet written) via the
> new interface. Other modules, such as mod_rewrite, which currently use DBM
> could also be modified so they can use all available lookup methods.

I like the idea.  One thing that might get confusing is how to handle
dependancies between modules.  If modules start to depend on functions in
other modules, I envision problems installing and configuring the
correct modules to get a certain module to work.

There might also be some overlap.  For example, my mod_php module includes
support for a bunch of SQL engines.  The various SQL-based mod_auth modules
that already exist use very similar calls.  Theoretically, the SQL access
functions need only be in one place.  I wouldn't want to force people
who use mod_php to have to use a certain mod_auth module and vice versa.
Perhaps the actual access functions should be separated completely from the
functional modules themselves?  They could perhaps be standalone access
function modules.  ie. mod_func_oracle, mod_func_sybase, mid_func_mysql,
mod_func_snmp, etc.  Then if you were writing a module that needed to access
Oracle, or needed to make snmp queries, you would use the appropriate
access function module.

-Rasmus