You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Stas Bekman <st...@stason.org> on 2005/05/02 08:48:39 UTC
Re: advice needed: mod_perl reverse proxy
allan juul wrote:
> hi stas
>
> Stas Bekman wrote:
>
>> allan juul wrote:
>> [...]
>
>
>>>> But if you use a mod_perl filter you will still hit the issue of
>>>> unknown content-length header.
>>>
>>>
>>>
>>>
>>> yes, of course that's true.
>>> there goes caching (:
>>
>>
>>
>> Not really. Nothing prevents you from buffering up the response,
>> process it, set the content-length header and make the document
>> cache-able.
>
>
> ok, eh how do i that. you mean instead of printing to STDOUT, collect
> data in a buffer, then set the calculated Content-Length, then print data?
That's right.
> anyway, it's pretty strange. it seems i'm able to set the Content-Length
> when i use the mod_perl_filter and do *not* reverse proxy. see both
> headers below. the strange things is that i'm not allowed at all to set
> the standard Content-Length, but indeed allowed to set a custom one
> called Content-Length2. and even stranger is that this custom header
> presents a correct value when *not* proxying but "0" when proxying. i
> use the exact same mod_perl code, also supplied below. the actual
> filtering of data content works in both cases.
Use must use $r->set_content_length(). See the mp2 test suite for examples.
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: advice needed: mod_perl reverse proxy
Posted by Jeff Ambrosino <jb...@gmail.com>.
I've had some similar issues, but gotten around them by using
erro_headers_out. This worked for me, so you might give it a try:
$f->r->headers_out->unset('Content-Length');
$f->r->err_headers_out->set('Content-Length' => length($NewBody));
Obviously "$NewBody" is whatever you're using to buffer the content.
FYI this is used with an HTTP output filter on mp2 (RC4) within a
reverse mod_proxy.
Jeff
On 5/3/05, allan juul <al...@muly.dk> wrote:
> but about collecting data in a buffer variable. it seems i can actually
> $f->print that buffer, but not actually calculate the length of it. or
> rather: i can calculate the length but when i set any header the value
> is 0 (whether i set it before the $f->print statement or after).
> it seems i must admit that i don't quite get what is going on and when.
>
> can anyone supply a simple example i then can check on our reverse proxy ?
Re: advice needed: mod_perl reverse proxy
Posted by Stas Bekman <st...@stason.org>.
allan@muly.dk wrote:
[...]
>>>>> can anyone supply a simple example i then can check on our reverse
>>>>> proxy ?
>>>>
>>>>
>>>>
>>>> Try: t/response/TestApache/content_length_header.pm
>>>> Though I haven't tried to call it from the filter, so may be Jeff's
>>>> suggestion will work.
>>>
>>>
>>>
>>>
>>> Jeff's suggestion does indeed work. oddly enough ;
>>
>>
>> It just happens to work in certain conditions. The correct solution is
>> to use a bucket brigade-based filter, which gives you a complete
>> control. What's happening is that streaming filter API passes FLUSH
>> buckets through, and you can't control that. We have to do that to
>> conform to the FLUSH requests, to be a well-behaved filter by default.
>> I've revealed that using 2 debug filters plugged before and after the
>> filter in question. This is a very useful debugging tool:
>> http://search.cpan.org/dist/Apache-DebugFilter/
>
>
> well its not available yet on win32.
what do you mean? It's a pure perl module.
>> So the following solution works just fine, it'll be added shortly to
>> the mp2 test suite as a pair of tests: t/filter/out_str_buffer.t
>> t/filter/TestFilter/out_str_buffer.pm
>
>
>
> thanks for the example andf replies. however, i get this error:
>
> Can't locate object method "first" via package "APR::Brigade" i have
> prior to that just uninstalled mod_perl [RC5] and installed mod_perl
> [RC6] via ppm (randys repository i beleve)
sorry, I didn't paste the part that loads the modules, please see:
http://perl.apache.org/docs/2.0/user/handlers/filters.html#Setting_the_Content_Length_Header_in_Request_Output_Filters
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: advice needed: mod_perl reverse proxy
Posted by al...@muly.dk.
Quoting Stas Bekman <st...@stason.org>:
> allan@muly.dk wrote:
>> Quoting Stas Bekman <st...@stason.org>:
>>
>>> allan juul wrote:
>>> [...]
>>>
>>>>> Use must use $r->set_content_length(). See the mp2 test suite for
>>>>> examples.
>>>>
>>>>
>>>>
>>>> (i don't have that method available in my mod_perl2)
>>>
>>>
>>> You sure do :)
>>>
>>> % lookup set_content_length
>>> To use method 'set_content_length' add:
>>> use Apache2::Response ();
>>> http://perl.apache.org/docs/2.0/api/Apache2/Response.html#C_set_content_length_
>>
>>
>> ok maybe i have a screwed installiation or a missing use/namespace .
>> "method not found" message in my error.log.
>>
>>>> but about collecting data in a buffer variable. it seems i can
>>>> actually $f->print that buffer, but not actually calculate the
>>>> length of it. or rather: i can calculate the length but when i set
>>>> any header the value is 0 (whether i set it before the $f->print
>>>> statement or after).
>>>> it seems i must admit that i don't quite get what is going on and when.
>>>>
>>>> can anyone supply a simple example i then can check on our reverse proxy ?
>>>
>>>
>>> Try: t/response/TestApache/content_length_header.pm
>>> Though I haven't tried to call it from the filter, so may be Jeff's
>>> suggestion will work.
>>
>>
>>
>> Jeff's suggestion does indeed work. oddly enough ;
>
> It just happens to work in certain conditions. The correct solution
> is to use a bucket brigade-based filter, which gives you a complete
> control. What's happening is that streaming filter API passes FLUSH
> buckets through, and you can't control that. We have to do that to
> conform to the FLUSH requests, to be a well-behaved filter by default.
> I've revealed that using 2 debug filters plugged before and after the
> filter in question. This is a very useful debugging tool:
> http://search.cpan.org/dist/Apache-DebugFilter/
well its not available yet on win32.
> So the following solution works just fine, it'll be added shortly to
> the mp2 test suite as a pair of tests: t/filter/out_str_buffer.t
> t/filter/TestFilter/out_str_buffer.pm
thanks for the example andf replies. however, i get this error:
Can't locate object method "first" via package "APR::Brigade" i have
prior to that just uninstalled mod_perl [RC5] and installed mod_perl
[RC6] via ppm (randys repository i beleve)
./allan
> and I'll document it too.
>
> sub flatten_bb {
> my ($bb) = shift;
>
> my $seen_eos = 0;
>
> my @data;
> for (my $b = $bb->first; $b; $b = $bb->next($b)) {
> $seen_eos++, last if $b->is_eos;
> $b->read(my $bdata);
> push @data, $bdata;
> }
> return (join('', @data), $seen_eos);
> }
>
> sub handler {
> my($filter, $bb) = @_;
>
> my $ctx = $filter->ctx;
>
> # no need to unset the C-L header, since this filter makes sure to
> # correct it before any headers go out.
> #unless ($ctx) {
> # $filter->r->headers_out->unset('Content-Length');
> #}
>
> my $data = exists $ctx->{data} ? $ctx->{data} : '';
> $ctx->{invoked}++;
> my($bdata, $seen_eos) = flatten_bb($bb);
> $bdata =~ s/-//g;
> $data .= $bdata if $bdata;
>
> if ($seen_eos) {
> my $len = length $data;
> $filter->r->headers_out->set('Content-Length', $len);
> $filter->print($data) if $data;
> }
> else {
> # store context for all but the last invocation
> $ctx->{data} = $data;
> $filter->ctx($ctx);
> }
>
> return Apache2::Const::OK;
> }
>
>
> --
> __________________________________________________________________
> Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
> http://stason.org/ mod_perl Guide ---> http://perl.apache.org
> mailto:stas@stason.org http://use.perl.org http://apacheweek.com
> http://modperlbook.org http://apache.org http://ticketmaster.com
>
Re: advice needed: mod_perl reverse proxy
Posted by Stas Bekman <st...@stason.org>.
allan@muly.dk wrote:
> Quoting Stas Bekman <st...@stason.org>:
>
>> allan juul wrote:
>> [...]
>>
>>>> Use must use $r->set_content_length(). See the mp2 test suite for
>>>> examples.
>>>
>>>
>>>
>>> (i don't have that method available in my mod_perl2)
>>
>>
>> You sure do :)
>>
>> % lookup set_content_length
>> To use method 'set_content_length' add:
>> use Apache2::Response ();
>> http://perl.apache.org/docs/2.0/api/Apache2/Response.html#C_set_content_length_
>>
>
>
> ok maybe i have a screwed installiation or a missing use/namespace .
> "method not found" message in my error.log.
>
>>> but about collecting data in a buffer variable. it seems i can
>>> actually $f->print that buffer, but not actually calculate the length
>>> of it. or rather: i can calculate the length but when i set any
>>> header the value is 0 (whether i set it before the $f->print
>>> statement or after).
>>> it seems i must admit that i don't quite get what is going on and when.
>>>
>>> can anyone supply a simple example i then can check on our reverse
>>> proxy ?
>>
>>
>> Try: t/response/TestApache/content_length_header.pm
>> Though I haven't tried to call it from the filter, so may be Jeff's
>> suggestion will work.
>
>
>
> Jeff's suggestion does indeed work. oddly enough ;
It just happens to work in certain conditions. The correct solution is to
use a bucket brigade-based filter, which gives you a complete control.
What's happening is that streaming filter API passes FLUSH buckets
through, and you can't control that. We have to do that to conform to the
FLUSH requests, to be a well-behaved filter by default.
I've revealed that using 2 debug filters plugged before and after the
filter in question. This is a very useful debugging tool:
http://search.cpan.org/dist/Apache-DebugFilter/
So the following solution works just fine, it'll be added shortly to the
mp2 test suite as a pair of tests: t/filter/out_str_buffer.t
t/filter/TestFilter/out_str_buffer.pm
and I'll document it too.
sub flatten_bb {
my ($bb) = shift;
my $seen_eos = 0;
my @data;
for (my $b = $bb->first; $b; $b = $bb->next($b)) {
$seen_eos++, last if $b->is_eos;
$b->read(my $bdata);
push @data, $bdata;
}
return (join('', @data), $seen_eos);
}
sub handler {
my($filter, $bb) = @_;
my $ctx = $filter->ctx;
# no need to unset the C-L header, since this filter makes sure to
# correct it before any headers go out.
#unless ($ctx) {
# $filter->r->headers_out->unset('Content-Length');
#}
my $data = exists $ctx->{data} ? $ctx->{data} : '';
$ctx->{invoked}++;
my($bdata, $seen_eos) = flatten_bb($bb);
$bdata =~ s/-//g;
$data .= $bdata if $bdata;
if ($seen_eos) {
my $len = length $data;
$filter->r->headers_out->set('Content-Length', $len);
$filter->print($data) if $data;
}
else {
# store context for all but the last invocation
$ctx->{data} = $data;
$filter->ctx($ctx);
}
return Apache2::Const::OK;
}
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: advice needed: mod_perl reverse proxy
Posted by al...@muly.dk.
Quoting Stas Bekman <st...@stason.org>:
> allan juul wrote:
> [...]
>>> Use must use $r->set_content_length(). See the mp2 test suite for examples.
>>
>>
>> (i don't have that method available in my mod_perl2)
>
> You sure do :)
>
> % lookup set_content_length
> To use method 'set_content_length' add:
> use Apache2::Response ();
> http://perl.apache.org/docs/2.0/api/Apache2/Response.html#C_set_content_length_
ok maybe i have a screwed installiation or a missing use/namespace .
"method not found" message in my error.log.
>> but about collecting data in a buffer variable. it seems i can
>> actually $f->print that buffer, but not actually calculate the
>> length of it. or rather: i can calculate the length but when i set
>> any header the value is 0 (whether i set it before the $f->print
>> statement or after).
>> it seems i must admit that i don't quite get what is going on and when.
>>
>> can anyone supply a simple example i then can check on our reverse proxy ?
>
> Try: t/response/TestApache/content_length_header.pm
> Though I haven't tried to call it from the filter, so may be Jeff's
> suggestion will work.
Jeff's suggestion does indeed work. oddly enough ;
./a
Re: advice needed: mod_perl reverse proxy
Posted by Stas Bekman <st...@stason.org>.
allan@muly.dk wrote:
> hi i don't get it. the below filter does output the content alright it
> seems, but the setting of the header *value* is incorrect. (?)
> so the $f->print statement prints correct output
> but the calcualtion length(output) is incorrect (since it evaluates
> length of this exact string "<html><head></head><body></body></html>\n" )
> why is that and how to fix this ??
Allan, I suggest that you spend time reading this document:
http://perl.apache.org/docs/2.0/user/handlers/filters.html
Clearly you don't realize that filters are invoked multiple times.
See my other reply for an example of a working code. If after reading the
doc you have questions, I'd be glad to answer those.
In particular notice that as soon as something is sent from the request
handler out, apache will generate its HTTP response headers and you can't
change those afterwards, since they were already sent to the client.
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: advice needed: mod_perl reverse proxy
Posted by al...@muly.dk.
hi i don't get it. the below filter does output the content alright it
seems, but the setting of the header *value* is incorrect. (?)
so the $f->print statement prints correct output
but the calcualtion length(output) is incorrect (since it evaluates
length of this exact string "<html><head></head><body></body></html>\n"
)
why is that and how to fix this ??
many thanks
./allan
# Apache/2.0.54 (Win32) mod_ssl/2.0.53 OpenSSL/0.9.7f proxy_html/2.4
mod_perl/1.999.22-dev Perl/v5.8.6 configured
# this is the actual 11 bytes of content on URL:
# hello world
# $ get localhost
# hello world
# $ head localhost
# ...
# Server: Microsoft-IIS/6.0
# Content-Length: 40
# Content-Type: text/html; charset=utf-8
# ...
# excerpt of code
my $context;
unless ($f->ctx) {
$r->headers_out->unset( "Content-Length" );
}
$context ||= $f->ctx;
my $content_length = 0;
my $str = "";
while ($f->read(my $buffer, 1024)) {
$buffer = $context->{extra} . $buffer if $context->{extra};
if (($context->{extra}) = $buffer =~ m/(<[^>]*)$/) {
$buffer = substr($buffer, 0, - length($context->{extra}));
}
$str .= $buffer;
}
if ($f->seen_eos) {
if ( $context->{extra} ) {
$str .= $context->{extra};
}
}
else {
$f->ctx($context);
}
$str = Rewrite::replace_links ( $str );
# at this point $str equals: <html><head></head><body></body></html>
$content_length += length( $str );
# set header
$r->err_headers_out->set('Content-Length', $content_length );
$f->print( $str );
# at this point $str equals: all 42KB of correct html
Re: advice needed: mod_perl reverse proxy
Posted by Stas Bekman <st...@stason.org>.
allan juul wrote:
[...]
>> Use must use $r->set_content_length(). See the mp2 test suite for
>> examples.
>
>
> (i don't have that method available in my mod_perl2)
You sure do :)
% lookup set_content_length
To use method 'set_content_length' add:
use Apache2::Response ();
http://perl.apache.org/docs/2.0/api/Apache2/Response.html#C_set_content_length_
> but about collecting data in a buffer variable. it seems i can actually
> $f->print that buffer, but not actually calculate the length of it. or
> rather: i can calculate the length but when i set any header the value
> is 0 (whether i set it before the $f->print statement or after).
> it seems i must admit that i don't quite get what is going on and when.
>
> can anyone supply a simple example i then can check on our reverse proxy ?
Try: t/response/TestApache/content_length_header.pm
Though I haven't tried to call it from the filter, so may be Jeff's
suggestion will work.
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: advice needed: mod_perl reverse proxy
Posted by allan juul <al...@muly.dk>.
Stas Bekman wrote:
> allan juul wrote:
>
>> hi stas
>>
>> Stas Bekman wrote:
>>
>>> allan juul wrote:
>>> [...]
>>
>>
>>
>>>>> But if you use a mod_perl filter you will still hit the issue of
>>>>> unknown content-length header.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> yes, of course that's true.
>>>> there goes caching (:
>>>
>>>
>>>
>>>
>>> Not really. Nothing prevents you from buffering up the response,
>>> process it, set the content-length header and make the document
>>> cache-able.
>>
>>
>>
>> ok, eh how do i that. you mean instead of printing to STDOUT, collect
>> data in a buffer, then set the calculated Content-Length, then print
>> data?
>
>
> That's right.
>
>> anyway, it's pretty strange. it seems i'm able to set the
>> Content-Length when i use the mod_perl_filter and do *not* reverse
>> proxy. see both headers below. the strange things is that i'm not
>> allowed at all to set the standard Content-Length, but indeed allowed
>> to set a custom one called Content-Length2. and even stranger is that
>> this custom header presents a correct value when *not* proxying but
>> "0" when proxying. i use the exact same mod_perl code, also supplied
>> below. the actual filtering of data content works in both cases.
>
>
> Use must use $r->set_content_length(). See the mp2 test suite for examples.
(i don't have that method available in my mod_perl2)
but about collecting data in a buffer variable. it seems i can actually
$f->print that buffer, but not actually calculate the length of it. or
rather: i can calculate the length but when i set any header the value
is 0 (whether i set it before the $f->print statement or after).
it seems i must admit that i don't quite get what is going on and when.
can anyone supply a simple example i then can check on our reverse proxy ?
thanks
./allan