Re: apache 2.0.43: %b not showing "bytes sent" but "bytes requested"

[Bojan Smojver <bo...@rexursive.com>]

On Fri, 2002-10-11 at 18:58, David Burry wrote:

> This should also be a concern for anyone who's using mod_logio to charge for
> bandwidth, because customers should be concerned about some serious
> overcharging going on here!

Only if you charge for outgoing bandwidth. On incoming bandwidth, I
think it should be pretty accurate because it won't count what wasn't
received.

Anyway, I think what's causing this problem is the fact that mod_logio
calculates the length of all brigades that are ready to be sent out. If
the sending gets interrupted in the middle, the whole thing gets
reported, instead of just the buckets/brigades that were actually sent
out. Maybe I should use (AP_FTYPE_NETWORK + 1) instead of
(AP_FTYPE_NETWORK - 1) for the output filter (i.e. count after the
sending, not before). I'll give it a try...

Bojan

Re: apache 2.0.43: %b not showing "bytes sent" but "bytes requested"

["William A. Rowe, Jr." <wr...@apache.org>]

At 10:21 PM 10/11/2002, rbb@apache.org wrote:

>What we are learning here is simple.  We need to do the counting in the
>core_output_filter.  If that means adding a field to the conn_rec, or
>somehow getting the request_rec in the core_output_filter doesn't
>matter.  The count needs to be done in the core_output_filter, by tallying
>the amount of data actually written.

How?  The request is destroyed already.  The remaining body is sitting
on the connection scope waiting to flush out.  The model is borked.

A change proposal for 2.1 that I hinted to earlier.  Three scopes, not two...

connection
  request
    handler

(or call them
connection
  response
    request
whichever we prefer.)

The middle tier can live out the body, even as we begin a new
request handler for another pipelined request.  The hard work of
creating the response has already fallen out of scope, so the
impact to the memory footprint shouldn't be all that great.

Bill

Re: apache 2.0.43: %b not showing "bytes sent" but "bytes requested"

[Bojan Smojver <bo...@rexursive.com>]

On Sat, 2002-10-12 at 20:19, Bojan Smojver wrote:

> r->bytes_sent =
>   (total bytes sent) - ( (total size of brigades) - (content length) )

Actually, I think this maths wouldn't work for SSL because content
length is calculated before it. Hmm...

Bojan

Re: apache 2.0.43: %b not showing "bytes sent" but "bytes requested"

[Bojan Smojver <bo...@rexursive.com>]

On Sat, 2002-10-12 at 13:21, rbb@apache.org wrote:

> What we are learning here is simple.  We need to do the counting in the
> core_output_filter.  If that means adding a field to the conn_rec, or
> somehow getting the request_rec in the core_output_filter doesn't
> matter.  The count needs to be done in the core_output_filter, by tallying
> the amount of data actually written.

On a more practical note...

Looks like sendfile_it_all will have to change a bit (lucky it's not a
public function). As it is, it doesn't seem to return the length
actually sent (i.e. &unused_bytes_sent would have to be passed into it).
Otherwise, we should be able to add up the unused_bytes_sent (and change
the name of the variable at the same time :-) to get total bytes sent.

Now, this would give us the total bytes sent but not in the sense of
r->bytes_sent. However, it would be relatively easy to figure it out:

r->bytes_sent =
  (total bytes sent) - ( (total size of brigades) - (content length) )

Total size of brigades is what mod_logio output filter does now, so
that's easy. Content length is what what content_length_filter does now,
that's easy too. The difference between them is obviously the length of
the headers. If the end result (r->bytes_sent) is negative, we just toss
it and keep r->bytes_sent = 0.

Obviously, the functionality, or the filters themselves, from mod_logio
could move into core.c and mod_log_config would make sure the counters
are reset when the transaction ends.

Bojan

Re: apache 2.0.43: %b not showing "bytes sent" but "bytes requested"

[Bojan Smojver <bo...@rexursive.com>]

Funny enough, there is a variable called unused_bytes_sent in that
function - kind of makes it obvious it's not being used ;-)

I thought that making f->r non-NULL was rejected due to complications
with other protocols that don't understand requests? Anyway, we don't
really need to store anything in request, we just need to retrieve from
connection, log it and then reset the counters.

Instead of introducing more fields to conn_rec (to prevent another MMN
bump), why don't we use the same thing you suggested for mod_logio,
which is putting the counting structure of core_output_filter into
f->c->conn_config? This structure would not be a mod_logio option, but
rather always there. mod_logio would then just use it to log. Or, even
better, all the logging would be done through mod_log_config, which
would make mod_logio redundant. By the same token, the input counting
could also go into core_input_filter...

Bojan

On Sat, 2002-10-12 at 13:21, rbb@apache.org wrote:
> 
> What we are learning here is simple.  We need to do the counting in the
> core_output_filter.  If that means adding a field to the conn_rec, or
> somehow getting the request_rec in the core_output_filter doesn't
> matter.  The count needs to be done in the core_output_filter, by tallying
> the amount of data actually written.
> 
> Ryan

Re: apache 2.0.43: %b not showing "bytes sent" but "bytes requested"

[Brian Pane <br...@cnet.com>]

rbb@apache.org wrote:

>What we are learning here is simple.  We need to do the counting in the
>core_output_filter.  If that means adding a field to the conn_rec, or
>somehow getting the request_rec in the core_output_filter doesn't
>matter.  The count needs to be done in the core_output_filter, by tallying
>the amount of data actually written.
>  
>

Yes, exactly.  The only problem is that this will require
a design change.  Currently, the last write can complete
after the request_rec no longer exists (if the connection
is a keepalive).

Solving this is tricky.  One possibility is to transfer
ownership of the the request (including its pool) to
the core output filter as soon as we pass an EOF to that
filter.  That would complicate the memory management a
bit, but it might be worth the effort because it would
give us almost all the infrastructure needed to support
a semi-async MPM.

Brian

Re: apache 2.0.43: %b not showing "bytes sent" but "bytes requested"

[rb...@apache.org]

What we are learning here is simple.  We need to do the counting in the
core_output_filter.  If that means adding a field to the conn_rec, or
somehow getting the request_rec in the core_output_filter doesn't
matter.  The count needs to be done in the core_output_filter, by tallying
the amount of data actually written.

Ryan

On 12 Oct 2002, Bojan Smojver wrote:

> Nope, that doesn't work. The number is always zero.
> 
> Bojan
> 
> On Sat, 2002-10-12 at 12:22, Bojan Smojver wrote:
> 
> > Anyway, I think what's causing this problem is the fact that mod_logio
> > calculates the length of all brigades that are ready to be sent out. If
> > the sending gets interrupted in the middle, the whole thing gets
> > reported, instead of just the buckets/brigades that were actually sent
> > out. Maybe I should use (AP_FTYPE_NETWORK + 1) instead of
> > (AP_FTYPE_NETWORK - 1) for the output filter (i.e. count after the
> > sending, not before). I'll give it a try...
> 

-- 

_______________________________________________________________________________
Ryan Bloom                        	rbb@apache.org
550 Jean St
Oakland CA 94610
-------------------------------------------------------------------------------

Re: apache 2.0.43: %b not showing "bytes sent" but "bytes requested"

[Bojan Smojver <bo...@rexursive.com>]

Nope, that doesn't work. The number is always zero.

Bojan

On Sat, 2002-10-12 at 12:22, Bojan Smojver wrote:

> Anyway, I think what's causing this problem is the fact that mod_logio
> calculates the length of all brigades that are ready to be sent out. If
> the sending gets interrupted in the middle, the whole thing gets
> reported, instead of just the buckets/brigades that were actually sent
> out. Maybe I should use (AP_FTYPE_NETWORK + 1) instead of
> (AP_FTYPE_NETWORK - 1) for the output filter (i.e. count after the
> sending, not before). I'll give it a try...

Re: apache 2.0.43: %b not showing "bytes sent" but "bytes requested"

[rb...@apache.org]

On 12 Oct 2002, Bojan Smojver wrote:

> On Fri, 2002-10-11 at 18:58, David Burry wrote:
> 
> > This should also be a concern for anyone who's using mod_logio to charge for
> > bandwidth, because customers should be concerned about some serious
> > overcharging going on here!
> 
> Only if you charge for outgoing bandwidth. On incoming bandwidth, I
> think it should be pretty accurate because it won't count what wasn't
> received.
> 
> Anyway, I think what's causing this problem is the fact that mod_logio
> calculates the length of all brigades that are ready to be sent out. If
> the sending gets interrupted in the middle, the whole thing gets
> reported, instead of just the buckets/brigades that were actually sent
> out. Maybe I should use (AP_FTYPE_NETWORK + 1) instead of
> (AP_FTYPE_NETWORK - 1) for the output filter (i.e. count after the
> sending, not before). I'll give it a try...

That won't work.  The core_filter will return without calling your
filter.

Ryan

_______________________________________________________________________________
Ryan Bloom                        	rbb@apache.org
550 Jean St
Oakland CA 94610
-------------------------------------------------------------------------------