[PROPOSAL]: Bytes received v. mod_log_config

[Bojan Smojver <bo...@rexursive.com>]

I'm playing around with a patch for mod_log_config that would enable
logging of the number bytes received per request. That kind of
information would then be very useful for determining input traffic per
virtual host, URL or whatever else, using regular web analysis tools,
e.g. webalizer.

Does something like that sound like functionality that can be rolled
into Apache 1.3/2.0?

Bojan

Re: [PROPOSAL]: Bytes received v. mod_log_config

[Bojan Smojver <bo...@rexursive.com>]

Quoting Chris Taylor <ch...@x-bb.org>:

> Yes, definately.
> 
> FWIW, I find it quite annoying that Webalizer's "amount served" is
> often blindingly inaccurate based on Apache's logs.

I think this is due to the fact that headers are not taken into account or
something like that. And there is also the mod_gzip factor that might change
things significantly. mod_accounting (http://sourceforge.net/projects/mod-acct/)
has some workarounds to produce a bit more accurate numbers. I also had a brief
look at webalizer code and I'll be producing a patch that takes advantage of the
number of input bytes stored in the log file, most likely as the last field, to
remain compatible with current standards.

> I hadn't realised this way possible, but it sounds like a great idea
> to me :)

I had a brief chat to the author of mod_accounting, Simone Tellini, about
pinching his routine(s) that calculates the number of input bytes. It is rather
simple, it just adds the strlen of request, lengths of all headers and specified
content-length, plus relevant CRLF pairs. mod_accounting takes database
approach, however, to record the input/output bytes, but I felt it would be
appropriate to extend the log files with such information as well.

As with anything open source, my motivation is purely selfish - I get charged
for input traffic, so I want my clients to pay accordingly. However, I don't
want to employ proprietary, restricted of closed source solutions for any of
that. So, I started scratching...

Once I'm happy with the code I'll send it to the list.

Bojan

Re: [PROPOSAL]: Bytes received v. mod_log_config

[Chris Taylor <ch...@x-bb.org>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yes, definately.

FWIW, I find it quite annoying that Webalizer's "amount served" is
often blindingly inaccurate based on Apache's logs.

I hadn't realised this way possible, but it sounds like a great idea
to me :)

Chris Taylor - chris@x-bb.org - The guy with the PS2 WebServer -
http://www.x-bb.org/chris.asc

- ----- Original Message ----- 
From: "Bojan Smojver" <bo...@rexursive.com>
To: "Apache Dev List" <de...@httpd.apache.org>
Sent: Sunday, August 18, 2002 11:25 PM
Subject: [PROPOSAL]: Bytes received v. mod_log_config


> I'm playing around with a patch for mod_log_config that would
> enable logging of the number bytes received per request. That kind
> of
> information would then be very useful for determining input traffic
> per virtual host, URL or whatever else, using regular web analysis
> tools, e.g. webalizer.
> 
> Does something like that sound like functionality that can be
> rolled into Apache 1.3/2.0?
> 
> Bojan

-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 7.0.3 for non-commercial use <http://www.pgp.com>

iQA/AwUBPWBQhyqf8lmE2RZkEQJpBACdFH9pi0xUQ2QgI142afS7hKH9weIAoJgn
LEx6VoI7qOUk6bDPI0gUI8gX
=1pJu
-----END PGP SIGNATURE-----

Re: [PROPOSAL]: Bytes received v. mod_log_config

[Justin Erenkrantz <je...@apache.org>]

On Mon, Aug 19, 2002 at 02:27:40PM +1000, Bojan Smojver wrote:
> Thanks for the hint! I never thought of that... Since a filter like that is
> connection based, I'm not sure (mainly because I don't know much about Apache)
> how would that work with persistent connections, chunked encoding and other
> cross connection complications in regards to reporting the info per request. Or
> would that just 'automagic' from the filter's point of view? If I take that
> route, I'll have to do a bit of digging ;-)

The trick is that recently someone (forget who) fixed up the filters
so that a connection-level filter always has a reference to the
'current' request.

So, even in a connection-level filter, filter_rec->r should be valid.
You'd have to determine some place to store how much we read, but
that could be done in a mod_log_config private structure.

Hope this makes some sense.  -- justin

Re: [PROPOSAL]: Bytes received v. mod_log_config

[Bojan Smojver <bo...@rexursive.com>]

Quoting Justin Erenkrantz <je...@apache.org>:

> On Mon, Aug 19, 2002 at 08:25:53AM +1000, Bojan Smojver wrote:
> > I'm playing around with a patch for mod_log_config that would enable
> > logging of the number bytes received per request. That kind of
> > information would then be very useful for determining input traffic
> per
> > virtual host, URL or whatever else, using regular web analysis tools,
> > e.g. webalizer.
> > 
> > Does something like that sound like functionality that can be rolled
> > into Apache 1.3/2.0?
> 
> For 2.0, you only need to add an input filter at the network
> level (AP_FTYPE_NETWORK-1 priority) to track this data.  Really
> easy to do.

Thanks for the hint! I never thought of that... Since a filter like that is
connection based, I'm not sure (mainly because I don't know much about Apache)
how would that work with persistent connections, chunked encoding and other
cross connection complications in regards to reporting the info per request. Or
would that just 'automagic' from the filter's point of view? If I take that
route, I'll have to do a bit of digging ;-)

> For 1.3, it's possible, but probably more complicated.  -- justin

I guess manual calculation, along the lines of mod_accounting, is probably the
only option there.

Bojan

Re: [PROPOSAL]: Bytes received v. mod_log_config

[Justin Erenkrantz <je...@apache.org>]

On Mon, Aug 19, 2002 at 08:25:53AM +1000, Bojan Smojver wrote:
> I'm playing around with a patch for mod_log_config that would enable
> logging of the number bytes received per request. That kind of
> information would then be very useful for determining input traffic per
> virtual host, URL or whatever else, using regular web analysis tools,
> e.g. webalizer.
> 
> Does something like that sound like functionality that can be rolled
> into Apache 1.3/2.0?

For 2.0, you only need to add an input filter at the network
level (AP_FTYPE_NETWORK-1 priority) to track this data.  Really
easy to do.

For 1.3, it's possible, but probably more complicated.  -- justin