You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Raul Dias <ra...@dias.com.br> on 2003/12/11 14:49:35 UTC
Documentation update request [WAS Re: [mp2] Memory Leak]
On Wed, 2003-12-10 at 20:42, Stas Bekman wrote:
> > Lastly, why does the memory continue to be consumer after the client has
> > terminated the connection?
>
> Because the server doesn't realize that the connection was aborted. You can
> check that with $c->aborted. It could also be a bug in Apache 2. I'd check
> whether it works without mod_perl, e.g. using INCLUDES or some other core
> filter with a big file.
So it would be nice to have this in the filter (at least) documentation.
So instead of:
while ($f->read(my $buffer, BUFF_LEN)) {
# ....
$f->print($buffer);
}
have this:
while ($f->read(my $buffer, BUFF_LEN)) {
return Apache::DECLINED
if ($f->r->connection->aborted);
$f->print($buffer);
}
Unless there are some caveats that I am unaware of by doing this.
This made a big deal to some of my filters as they would keep processing
large files unecessary after the client aborted.
[]'s
Raul Dias
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
Re: Documentation update request [WAS Re: [mp2] Memory Leak]
Posted by Raul Dias <ra...@dias.com.br>.
On Thu, 2003-12-11 at 22:00, Stas Bekman wrote:
> Raul Dias wrote:
> > I did mean to sound as a rant (if I did). I know that's not easy to
> > keep with the documentation.
>
> what can you do with your subconscious which seems to tell the truth of what
> you've really meant ;)
>
>:)
> Doc patches are always welcome...
>
> > I am using a naive approach to do a traffic shapper.
> > Dividing the available bandwidth by the sum of all "shapped" connections
> > will give the individual connection bandwidth.
>
> What's a "shapped" connection? Is it a filter that tries to measure the data
> transfer rate?
No, this is a http traffic shapper ala mod_bandwidth that can be setup
by virtual domain, dir/location, groups and/or client ip.
>
> I thought there are special tools that can give you traffic
> accounting per socket.
Metering is another module.
But it would be really nice if it could be implemented thru filters.
To have traffic statistics in a per virtual domain / client ip
combination would require a reassemble of the HTTP headers by a sniffing
tool.
> You should be aware of at least one culprit (which I'm not sure if it's still
> the case with httpd-2.0). If you use the front-end/back-end setup and reverse
> proxy the requests, your backend will never know that the connection was
> aborted if it did. proxy doesn't tell that to the backend (it keep the
> connection open). Again this is how things work in 1.0, I don't know about 2.0
> yet. But this is another potential situation where your meter may fail.
>
That's something that I will need to look at in the future.
> > I think I understand it now.
> > You can only use Apache::DECLINED before starting to read the data.
>
> That's right. You can return it after you started reading too, but it'll
> return some of the data that you may have printed out already. which is not a
> problem if the connection has been aborted anyway. I've asked on httpd-dev
> what should be returned and didn't have any follow ups so far. But I think
> APR::FAILURE, or anything else will do fine.
For now I will just turn off the delay.
As another filter might need to get the whole data.
[]'s
Raul Dias
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
Re: Documentation update request [WAS Re: [mp2] Memory Leak]
Posted by Stas Bekman <st...@stason.org>.
Raul Dias wrote:
>>perl-5.8.3-ithread -MApache2 -MModPerl::MethodLookup -e print_object
>>Apache::RequestRec | wc -l
>> 149
>>
>
> Great. This is a good start point for me.
In that case please see:
http://perl.apache.org/docs/2.0/api/ModPerl/MethodLookup.html#top
> I did mean to sound as a rant (if I did). I know that's not easy to
> keep with the documentation.
what can you do with your subconscious which seems to tell the truth of what
you've really meant ;)
Doc patches are always welcome...
> I am using a naive approach to do a traffic shapper.
> Dividing the available bandwidth by the sum of all "shapped" connections
> will give the individual connection bandwidth.
What's a "shapped" connection? Is it a filter that tries to measure the data
transfer rate? I thought there are special tools that can give you traffic
accounting per socket.
> Without detecting that the connection was aborted, it would be counted
> in the sum of the "shapped" connection. In this sense it would still
> slowing down the other connections.
that's exactly where the naive approach is wrong, IMHO. since it tries to
account in the wrong place. It should probably measure the heartbeat on the
pipe and not in the process that feeds the pipe. But without having a good
understanding of your alg, I could be totally wrong.
In any case it's good to always try to detect aborted connections and stop
whatever you were doing.
You should be aware of at least one culprit (which I'm not sure if it's still
the case with httpd-2.0). If you use the front-end/back-end setup and reverse
proxy the requests, your backend will never know that the connection was
aborted if it did. proxy doesn't tell that to the backend (it keep the
connection open). Again this is how things work in 1.0, I don't know about 2.0
yet. But this is another potential situation where your meter may fail.
> I think I understand it now.
> You can only use Apache::DECLINED before starting to read the data.
That's right. You can return it after you started reading too, but it'll
return some of the data that you may have printed out already. which is not a
problem if the connection has been aborted anyway. I've asked on httpd-dev
what should be returned and didn't have any follow ups so far. But I think
APR::FAILURE, or anything else will do fine.
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
Re: Documentation update request [WAS Re: [mp2] Memory Leak]
Posted by Raul Dias <ra...@dias.com.br>.
On Thu, 2003-12-11 at 20:43, Stas Bekman wrote:
> Raul Dias wrote:
> > On Thu, 2003-12-11 at 16:30, Stas Bekman wrote:
> >
> >>Raul Dias wrote:
> >>
> >>>On Wed, 2003-12-10 at 20:42, Stas Bekman wrote:
[...]
> > It would really help to have some section about all the methods
> > available to $filter, $request and $connection handlers.
>
> some section? You are being funny:
>
> perl-5.8.3-ithread -MApache2 -MModPerl::MethodLookup -e print_object
> Apache::RequestRec | wc -l
> 149
>
Great. This is a good start point for me.
> This is only the methods you can invoke on the request object. There are some
> 600 or 700 methods and functions in mp2.
>
> Things are slowly getting documented at:
> http://perl.apache.org/docs/2.0/api/
>
> For example the filter page is pretty complete:
> http://perl.apache.org/docs/2.0/api/Apache/Filter.html
>
> > Most of the time I found some methods by browsing the list history.
> > I just got to know about $c->aborted() in this thread.
> >
> > Sorry if this is documented somewhere, but there is no mention of it by
> > greping the docs dir.
>
> Because it's incomplete. Help to add more things is very welcome. A big chunk
> is the same as mp1 so it should be an easy copy-n-paste.
>
I did mean to sound as a rant (if I did). I know that's not easy to
keep with the documentation.
[...]
> >>I'm not sure what's the right response code to use here, since first it
> >>doesn't matter to the user (the connection is aborted), second a particual
> >>caller of the filter may ignore the response code. I'd think AP_FILTER_ERROR
> >>might be a good choice. Let me ask httpd-dev what response code should be used
> >>in such a case and I'll be back to you. Then we can document it nicely.
> >
> >
> > I think that Apache::DECLINED is the correct behaviour as you explain
> > above.
> >
> > In a bandwidth limiter, there is no point on holding the traffic after
> > the connection has being aborted.
>
> You aren't holding the traffic. If the connection is aborted the process is
> still busy churning data, but it doesn't send any of it out. I don't think
> your bandwidth limiter should be affected at all. It should monitor sockets,
> not processes I believe. what if the process is busy in the cleanup phase, do
> you also consider it consuming bandwidth? It doesn't since Cleanup handler is
> run after the connection has been closed.
>
I am using a naive approach to do a traffic shapper.
Dividing the available bandwidth by the sum of all "shapped" connections
will give the individual connection bandwidth.
Without detecting that the connection was aborted, it would be counted
in the sum of the "shapped" connection. In this sense it would still
slowing down the other connections.
At least until I figure out a way to measure the real speed that the
data is being sent to the client.
> > OTOH, a filter might need to finish
> > parsing a file even if the connection was drop to do some clean up that
> > might be dependent on the file content (e.g. a database transaction).
>
> And that's precisely why I suggested that Apache::DECLINED is not the correct
> value to return, because of its special meaning. It'll try to forward the
> brigade that was already partially read and processed, which most likely mess
> things up (it doesn't since normally nobody cares what it does if the
> connection was aborted).
I think I understand it now.
You can only use Apache::DECLINED before starting to read the data.
[]'s
Raul Dias
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
Re: Documentation update request [WAS Re: [mp2] Memory Leak]
Posted by Stas Bekman <st...@stason.org>.
Raul Dias wrote:
> On Thu, 2003-12-11 at 16:30, Stas Bekman wrote:
>
>>Raul Dias wrote:
>>
>>>On Wed, 2003-12-10 at 20:42, Stas Bekman wrote:
>>>
>>>
>>>
>>>>>Lastly, why does the memory continue to be consumer after the client has
>>>>>terminated the connection?
>>>>
>>>>Because the server doesn't realize that the connection was aborted. You can
>>>>check that with $c->aborted. It could also be a bug in Apache 2. I'd check
>>>>whether it works without mod_perl, e.g. using INCLUDES or some other core
>>>>filter with a big file.
>>>
>>>
>>>So it would be nice to have this in the filter (at least) documentation.
>>
>>why at least? what else did you have on your mind?
>
>
> It would really help to have some section about all the methods
> available to $filter, $request and $connection handlers.
some section? You are being funny:
perl-5.8.3-ithread -MApache2 -MModPerl::MethodLookup -e print_object
Apache::RequestRec | wc -l
149
This is only the methods you can invoke on the request object. There are some
600 or 700 methods and functions in mp2.
Things are slowly getting documented at:
http://perl.apache.org/docs/2.0/api/
For example the filter page is pretty complete:
http://perl.apache.org/docs/2.0/api/Apache/Filter.html
> Most of the time I found some methods by browsing the list history.
> I just got to know about $c->aborted() in this thread.
>
> Sorry if this is documented somewhere, but there is no mention of it by
> greping the docs dir.
Because it's incomplete. Help to add more things is very welcome. A big chunk
is the same as mp1 so it should be an easy copy-n-paste.
>>So you did have it solve your problem? Very good.
>>
>
>
> In a bandwidth limiter it is very important to know that a connection is
> droped, so other clients can get the released bandwidth.
Absolutely.
>>Though I don't think Apache::DECLINED is the right return code here.
>>Apache::DECLINED should be used when your filter decides not to filter the
>>data, in which case mod_perl will forward the data to the next filter in
>>chain. e.g. these two filters are equivalent:
>>
>> sub out_filter {
>> my ($f, $bb) = @_;
>>
>> my $rv = $f->next->pass_brigade($bb);
>> return $rv unless $rv == APR::SUCCESS;
>>
>> return Apache::OK;
>> }
>>
>> sub out_filter {
>> my ($f, $bb) = @_;
>>
>> return Apache::DECLINED;
>> }
>>
>>I'm not sure what's the right response code to use here, since first it
>>doesn't matter to the user (the connection is aborted), second a particual
>>caller of the filter may ignore the response code. I'd think AP_FILTER_ERROR
>>might be a good choice. Let me ask httpd-dev what response code should be used
>>in such a case and I'll be back to you. Then we can document it nicely.
>
>
> I think that Apache::DECLINED is the correct behaviour as you explain
> above.
>
> In a bandwidth limiter, there is no point on holding the traffic after
> the connection has being aborted.
You aren't holding the traffic. If the connection is aborted the process is
still busy churning data, but it doesn't send any of it out. I don't think
your bandwidth limiter should be affected at all. It should monitor sockets,
not processes I believe. what if the process is busy in the cleanup phase, do
you also consider it consuming bandwidth? It doesn't since Cleanup handler is
run after the connection has been closed.
> OTOH, a filter might need to finish
> parsing a file even if the connection was drop to do some clean up that
> might be dependent on the file content (e.g. a database transaction).
And that's precisely why I suggested that Apache::DECLINED is not the correct
value to return, because of its special meaning. It'll try to forward the
brigade that was already partially read and processed, which most likely mess
things up (it doesn't since normally nobody cares what it does if the
connection was aborted).
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
Re: Documentation update request [WAS Re: [mp2] Memory Leak]
Posted by Raul Dias <ra...@dias.com.br>.
On Thu, 2003-12-11 at 16:30, Stas Bekman wrote:
> Raul Dias wrote:
> > On Wed, 2003-12-10 at 20:42, Stas Bekman wrote:
> >
> >
> >>>Lastly, why does the memory continue to be consumer after the client has
> >>>terminated the connection?
> >>
> >>Because the server doesn't realize that the connection was aborted. You can
> >>check that with $c->aborted. It could also be a bug in Apache 2. I'd check
> >>whether it works without mod_perl, e.g. using INCLUDES or some other core
> >>filter with a big file.
> >
> >
> > So it would be nice to have this in the filter (at least) documentation.
>
> why at least? what else did you have on your mind?
It would really help to have some section about all the methods
available to $filter, $request and $connection handlers.
Most of the time I found some methods by browsing the list history.
I just got to know about $c->aborted() in this thread.
Sorry if this is documented somewhere, but there is no mention of it by
greping the docs dir.
[..]
> So you did have it solve your problem? Very good.
>
In a bandwidth limiter it is very important to know that a connection is
droped, so other clients can get the released bandwidth.
> Though I don't think Apache::DECLINED is the right return code here.
> Apache::DECLINED should be used when your filter decides not to filter the
> data, in which case mod_perl will forward the data to the next filter in
> chain. e.g. these two filters are equivalent:
>
> sub out_filter {
> my ($f, $bb) = @_;
>
> my $rv = $f->next->pass_brigade($bb);
> return $rv unless $rv == APR::SUCCESS;
>
> return Apache::OK;
> }
>
> sub out_filter {
> my ($f, $bb) = @_;
>
> return Apache::DECLINED;
> }
>
> I'm not sure what's the right response code to use here, since first it
> doesn't matter to the user (the connection is aborted), second a particual
> caller of the filter may ignore the response code. I'd think AP_FILTER_ERROR
> might be a good choice. Let me ask httpd-dev what response code should be used
> in such a case and I'll be back to you. Then we can document it nicely.
I think that Apache::DECLINED is the correct behaviour as you explain
above.
In a bandwidth limiter, there is no point on holding the traffic after
the connection has being aborted. OTOH, a filter might need to finish
parsing a file even if the connection was drop to do some clean up that
might be dependent on the file content (e.g. a database transaction).
[]'s
Raul Dias
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
Re: Documentation update request [WAS Re: [mp2] Memory Leak]
Posted by Stas Bekman <st...@stason.org>.
Raul Dias wrote:
> On Wed, 2003-12-10 at 20:42, Stas Bekman wrote:
>
>
>>>Lastly, why does the memory continue to be consumer after the client has
>>>terminated the connection?
>>
>>Because the server doesn't realize that the connection was aborted. You can
>>check that with $c->aborted. It could also be a bug in Apache 2. I'd check
>>whether it works without mod_perl, e.g. using INCLUDES or some other core
>>filter with a big file.
>
>
> So it would be nice to have this in the filter (at least) documentation.
why at least? what else did you have on your mind?
> So instead of:
>
> while ($f->read(my $buffer, BUFF_LEN)) {
> # ....
> $f->print($buffer);
> }
>
> have this:
>
> while ($f->read(my $buffer, BUFF_LEN)) {
> return Apache::DECLINED
> if ($f->r->connection->aborted);
> $f->print($buffer);
> }
>
>
> Unless there are some caveats that I am unaware of by doing this.
>
> This made a big deal to some of my filters as they would keep processing
> large files unecessary after the client aborted.
So you did have it solve your problem? Very good.
Though I don't think Apache::DECLINED is the right return code here.
Apache::DECLINED should be used when your filter decides not to filter the
data, in which case mod_perl will forward the data to the next filter in
chain. e.g. these two filters are equivalent:
sub out_filter {
my ($f, $bb) = @_;
my $rv = $f->next->pass_brigade($bb);
return $rv unless $rv == APR::SUCCESS;
return Apache::OK;
}
sub out_filter {
my ($f, $bb) = @_;
return Apache::DECLINED;
}
I'm not sure what's the right response code to use here, since first it
doesn't matter to the user (the connection is aborted), second a particual
caller of the filter may ignore the response code. I'd think AP_FILTER_ERROR
might be a good choice. Let me ask httpd-dev what response code should be used
in such a case and I'll be back to you. Then we can document it nicely.
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html