Configurable logging formats... EXPERIMENTAL module.

["Robert S. Thau" <rs...@ai.mit.edu>]

I've been playing around for the last few hours with a configurable
logging module (i.e., a replacement for mod_log_common which accepts a
LogFormat directive in addition to TransferLog).  This is still quite
experimental code --- it *seems* to work in my test VirtualHost (well,
main host and 'localhost') configuration, but I don't quite have
enough confidence in it to run with it on the main server yet.  (FWIW,
a crude sanity check --- 'GET /index.html' a hundred times --- didn't
seem to show much a performance hit, though there's always the chance
I screwed up doing the test; also, testing the module did uncover a
one-line thinko in http_protocol.c; see next note).

Assuming I gain enough confidence in it tomorrow to run the thing
myself, there are a few things we could do with it:

1) Reject the thing outright, if it isn't what people want.  I won't
   be crushed --- it represents about six hours worth of work to date, tops.

2) Put it off until after current customer ship.

3) Release it as "experimental and unsupported", but as an indication
   of future directions, and as something which people who are
   interested in new and better logging solutions might want to play
   with.

Anyway, the LogFormat directive is along the same basic lines I
remember people talking about; the LogFormat which reproduces CLF is

  LogFormat "%h %l %u %t \"%r\" %s %b"
  
(those being host, logname, user, time, request, status, bytes-sent).

In addition to these, you can ask for %{Foobar}i and %{Foobar}o, to
get at the contents of some request or response MIME header,
respectively (e.g., you can use %{Referer}i to get the Referer).
Also, you can conditionalize the appearance of certain fields by HTTP
status code; for instance, '%!200,304,302{Referer}i' logs Referer only
on requests which got some sort of nontrivial error, and a '-'
otherwise to keep parsing sane.  (This might be useful if your main
use for Referer: is tracking down pages with bogus links to your site,
and you don't want it taking up space otherwise --- in fact,
%404{Referer}i might be what some people really want; file not found
*only*).

To repeat, I am not wedded to this syntax, to the code, or to anything
at all about this really --- and I don't consider this to be in final
form in any case.  (More formatting directives for URI, filename,
etc., are an obvious and easy addition, for example --- I'm just too
damn lazy for the moment).  I wrote it mainly because if people really
are going down the path of lots of separate log files (which the
module structure does, unfortunately, tend to make easy), it might be
interesting to at least look at an alternative.

rst

PS --- people who want to look at the code, with caveats that it 
       probably still has a lurking bug or two, can find it at
       ftp://ftp.ai.mit.edu/pub/users/rst/mod_log_config.c.

Re: Configurable logging formats... EXPERIMENTAL module.

[Brian Behlendorf <br...@organic.com>]

I would love to marry this with mod_cookie.

	Brian

--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@organic.com  brian@hyperreal.com  http://www.[hyperreal,organic].com/

Re: Configurable logging formats... EXPERIMENTAL module.

[Brian Behlendorf <br...@organic.com>]

On Sat, 29 Jul 1995, Robert S. Thau wrote:
> I've been playing around for the last few hours with a configurable
> logging module (i.e., a replacement for mod_log_common which accepts a
> LogFormat directive in addition to TransferLog).  This is still quite
> experimental code --- it *seems* to work in my test VirtualHost (well,
> main host and 'localhost') configuration, but I don't quite have
> enough confidence in it to run with it on the main server yet. 

Testing it now on port 8001 on hyperreal (conf file 
/usr/local/www.tools/apache/conf/httpd.conf.s for those interested).  
Apparently there's a bug in that an extra copy of every request
gets logged to SERVER_ROOT/logs/access_log, for both virtual and 
non-virtual hosts.

> 3) Release it as "experimental and unsupported", but as an indication
>    of future directions, and as something which people who are
>    interested in new and better logging solutions might want to play
>    with.

Yup, I would support this!  +1

> Anyway, the LogFormat directive is along the same basic lines I
> remember people talking about; the LogFormat which reproduces CLF is
> 
>   LogFormat "%h %l %u %t \"%r\" %s %b"
>   
> (those being host, logname, user, time, request, status, bytes-sent).

There's just a few more I'd like to add:

%U for unix time format
%a for actual object retreived when it differs from the object requested 
	- the heuristics for this need defining I suppose.
%T for total time to deliver file, in milliseconds (okay, I'm dreaming, 
	shoot me)

Hmm.  The rest seem to be covered by your %i and %o (brilliant!)

> In addition to these, you can ask for %{Foobar}i and %{Foobar}o, to
> get at the contents of some request or response MIME header,
> respectively (e.g., you can use %{Referer}i to get the Referer).
> Also, you can conditionalize the appearance of certain fields by HTTP
> status code; for instance, '%!200,304,302{Referer}i' logs Referer only
> on requests which got some sort of nontrivial error, and a '-'
> otherwise to keep parsing sane.  (This might be useful if your main
> use for Referer: is tracking down pages with bogus links to your site,
> and you don't want it taking up space otherwise --- in fact,
> %404{Referer}i might be what some people really want; file not found
> *only*).

I like this, but there is one more condition I'd like to be able to test for
- file type.  In our custom log_common we log referrers for every non-image
access, to save space and ease readability.  If this is hard to do 
efficiently it's not a big deal to me, I'll just hack around it, but it 
might be something others want.

Oh, and we need to bring up the question of escaping again.... sigh.

	Brian

--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@organic.com  brian@hyperreal.com  http://www.[hyperreal,organic].com/