Log Reporting (was RE: Browser detection taglib) (fwd)

[ba...@generationjava.com]

[Cross-posted to Struts-Dev]

---------- Forwarded message ----------
Date: Tue, 25 Sep 2001 18:04:24 +0100 (BST)
From: bayard@generationjava.com
Reply-To: taglibs-dev@jakarta.apache.org
To: "'taglibs-dev@jakarta.apache.org'" <ta...@jakarta.apache.org>
Cc: commons-dev@jakarta.apache.org
Subject: Log Reporting (was RE: Browser detection taglib)

This thread was initially a request for a browser-taglib, then it turned
into a discussion on Commons' util.http.BrowserDectory, and now I'd like
to nudge it in a new direction with a brain dump on an idea I've been
mulling upon.

The reason for a browser-detection taglib is most likely so that the
developer can monitor the types of browser's hitting their website. This
is commonly done by products such as Analog and Webalizer. I believe
Webalizer is a C product and Analog a Perl product.
The general idea is that you point the Web Log analyser at the log files
and it produces pretty tables, statistics and graphs.

Another product that does the same thing is thecounter.com. I imagine
there are many other websites with the same service. I pay them 10 dollars
a year and get some pretty pictures.

All of these have some flaws however.

1) They focus hard on web-logging. 
2) They aren't in Java.
3) They (seem) to be web-server based. ie) A developer can't install it,
   the web server admin has to. 
4) They are log-file based. Pointing them at a database seems difficult.
   What if the log-file is XML.

I could be way off base, I've not gone into the code behind any of them at
all. 

I think a Java Generic Log Maker, Viewer and Reporter would be very very
nice. Hopefully it can reuse as much existing code as possible, ie) XML
parser, Log4J.

Here are some ideas:

1] View a log.
  1.1) Logs must be able to come from anywhere. (interface Log)
  1.2) User-definable log-format. Need to support types, ie) url, date,
       email, number.
  1.3) Need an object structure for a Log which consists of lines and
       fields. 

2] Report on a log.
  2.1) Model-based loglets that manipulate a Log, possibly into 
       another Log. ie) averages, tables, summations etc. Statistics.
  2.2) Reportlets (!). Take a Log and pretty it up. ie) piecharts,
       bar-graphs, ascii.
  2.3) Provide different output formats. Ascii, PDF, JSP/Servlet, Swing.

  2.4) Genericity of Graphing/Imaging solution. Be able to handle many 
       image types.

3] Log Creation. Sits on top of Log4J?
  3.1) Web-targetted. Browser detect, IP etc.
  3.2) Generic so developers can log application logs and still use the 
       product to analyse the logs.

4] Utilities
  4.1) BrowserInfo. Database of browser versions to browser capability?
       ie) IE6/Mac = Java2, IE4/Win = Java1.1.4
  4.2) Portlistener. Use something like this to build up information on 
       browser http headers.


Flow of control
---------------

Application outputs a log to some LogSource. LogSource provides access to
a Log which provides LogEvents. Each LogEvent has a set of LogFields, etc.

Log is passed through a chain of Loglets. These would do mathematics
mainly. Averaging, Summing, etc. One would be a dnslookup.

Result of the Loglets stage is then passed to/through Reportlets. These
would build up a report on the data. In a presentation generic way. So a
Table object, an Image object. Hopefully not a lot more.

Then the result of the Reportlet would be passed to a presentation layer,
JSP/PDF/Email etc. 

=====

I know that's all very pie in the sky, but any ideas people have would be
gratefully received. I would also love to know of any products that either
do the above or are open-source-java and could fill in a part of it. 

I have a basic http snooping tool in Java to look at http headers (yeah I
know you can do it from cmd line *nux, netcat isn't it?). I've a CSV
reader/writer that might be growable to a Log-reader. And there's the
BrowserDetector.

Plus Log4J which this idea should be integrated as close to as
possible. Maybe :)

Bay