Re: phase II

[Randy Terbush <ra...@zyzzyva.com>]

::Regarding logging..
> The other logging process can then do all the dirty work of 
> writing a logfile entry.
> 
> > and then pipe it to some really good logging program that we write and 
> > distribute with Apache, how likely a contender could that be for CLF2?
> 
> If Apache (and other servers) just dump raw data (as above), CLF2 could
> be unecessary. Log analyzer writers could just bundle their own program
> to create the logfiles.
> 
> Hmm, this has great "pass-the-buck" potential. It's easy to extend, and 
> the dirty work is done by someone else's code. We could write one which
> dumps the current CLF.

Exactly.

Writing to a named pipe works nicely, but here are the issues that I
have encountered.

1. Only one logger daemon can read from a pipe at one time. Aside from 
using sockets, the only solution I see to this is to provide a "spigot"
that is capable of increasing the number of pipes as they are requested.
I am beginning to prefer the idea of the server providing a socket
interface for these little suckers to attach to.  The server could
either be writing the same information to all established sockets,
or we could provide a per socket configuration for the information.

I prefer the idea of the socket and write *everything* to the listening
process. Let the logger determine what to do with it.


2. I would like to see this group create our own log format. It
would be easy to provide a logger daemon that converted it to CLF
and wrote it to a flat file.  I would like to see Apache sending
access dates in integer format and logging the IP address only.

Reducing the output of the present CLF to this.

198.183.2.1 - - 81100035 GET /man/Perl5/perlfunc.html HTTP/1.0 200 135964


Thoughts?  

If we could come to some concensus on this much, we could move forward
with this and implement the addons after this is working.

Re: phase II

[Brian Behlendorf <br...@organic.com>]

On Wed, 24 May 1995, Randy Terbush wrote:
> I prefer the idea of the socket and write *everything* to the listening
> process. Let the logger determine what to do with it.

+1

> 2. I would like to see this group create our own log format. It
> would be easy to provide a logger daemon that converted it to CLF
> and wrote it to a flat file.  I would like to see Apache sending
> access dates in integer format and logging the IP address only.
> 
> Reducing the output of the present CLF to this.
> 
> 198.183.2.1 - - 81100035 GET /man/Perl5/perlfunc.html HTTP/1.0 200 135964

+1 again, with notes:

1) Whatever the separator is, we need to make sure the data escapes that 
character - i.e., this has caused problems at hotwired:

198.183.2.1 - brian behlendorf (etc).... 401 -

I would actually prefer to have tabs separate the data, for a couple 
reasons:
  a) spaces can be preserved (important for user-agent)
  b) it's a standard for importing to other databases.

2) I liked Rob Hartil's MIME-ish dump of the mime headers, but it seems 
like that will dramatically increase the overhead for a logging daemon 
that needs to be sooper efficient.  Maybe have a separate extra-credit 
pipe for anything our CLF2 doesn't handle.

3) How does this look:

IP RFC931 REMOTE_USER TIME METHOD URI PROTOCOL BYTESTRANSFERRED STATUSRESPONSE SESSIONID REALOBJECT SERVERNAME REFERRER USERAGENT

where SESSIONID won't be around until 1.1 at least, REALOBJECT is the
canonical object retreived (if it exists - i.e., a get for "mother" returned
"mother.gif"), SERVERNAME is the virtualhost accessed.  Anything else we 
should be logging?

	Brian


--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@organic.com  brian@hyperreal.com  http://www.[hyperreal,organic].com/

Re: phase II

[Brian Tao <ta...@gate.sinica.edu.tw>]

On Wed, 24 May 1995, Randy Terbush wrote:
> 
> 2. I would like to see this group create our own log format. It
> would be easy to provide a logger daemon that converted it to CLF
> and wrote it to a flat file.  I would like to see Apache sending
> access dates in integer format and logging the IP address only.

    So would I... the current CLF is too geared towards human readers,
IMHO.  Logfile analyzers have to go through some awkward contortions
to correctly parse each line and handle errors or misformats.  If it
weren't for the space-delimited fields and compound fields (like the
HTTP command block, enclosed in quotes), simple shell and awk scripts
would suffice, instead of bulky C or perl utilities.

> Reducing the output of the present CLF to this.
> 
> 198.183.2.1 - - 81100035 GET /man/Perl5/perlfunc.html HTTP/1.0 200 135964

    I would like to see a fixed number of tab-delimited fields that a
simple awk script can post-process into the original CLF or something
more human-readable.
-- 
Brian ("Though this be madness, yet there is method in't") Tao
taob@gate.sinica.edu.tw <-- work ........ play --> taob@io.org