You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by James Lee <mo...@gmail.com> on 2010/08/24 15:20:21 UTC
mp2] [mod_proxy] [filter] mod_proxy is not playing with my request filter.
Hi mod_perl community, this is my first post so be gentle with me.
I'm trying to create something which translates a GET request into a POST
(creating the body dynamically) and then hand off the POST to another
server.
I have created a mod_perl connection filter to change the GET to a POST in
the request line. It also adds Content-Length and Content-Type headers and
this works fine.
I then have a request filter which creates the POST body. This works when
the request is handled by a PerlResponseHandler but I'm trying to offload
the request to mod_proxy as the responses can be quite large but I just
can't get it to play ball. I see the request filter being called but the
POST body never arrives at the target server. I suspect this is a timing
issue, ie: mod_proxy kicking in before my request filter but I'm not
certain.
Can anybody shed some light on this or correct my approach. I was going to
add the POST body in the connection filter but it seemed cleaner/easier to
do it in a request filter.
I've looked on the mailing list and found a few things that touch on this (
http://tech.groups.yahoo.com/group/modperl/message/54541) but nothing that's
close enough to help.
Config below ... I've not included perl code as this message is quite long
anyway. Please let me know if it would be helpful.
Thanks in advance, James.
***
httpd.conf extract:
PerlInputFilterHandler Sample::RequestTweaker::change_get_to_post
<Location /reports>
PerlInputFilterHandler Sample::RequestTweaker::inject_post_data
ProxyPass http://appserver/reports-engine
</Location>
Re: mp2] [mod_proxy] [filter] mod_proxy is not playing with my
request filter.
Posted by James Lee <mo...@gmail.com>.
Hi, thanks for the response.
To my knowledge (which is about a weeks worth) you don't need or indeed see
an end_of_stream bucket in a connection filter till the connection is closed
and then only in the output filter.
Conveniently the request headers seem to be passed from the core filter one
brigade at a time which makes my code quite easy, no need for buffering
buckets between calls. As I said the code works fine until I append the
'data=test' to the end of the headers which should be my request body.
It's been driving me crazy all day and I can't understand why its doing what
it is doing! It's just frustrating because it sounds like something I should
be able to do.
James.
2010/8/26 Torsten Förtsch <to...@gmx.net>
> On Thursday, August 26, 2010 18:28:47 James Lee wrote:
>
> A few comments below.
>
> The goal of the code is to fill up $bb. Nothing is copied there unless you
> do.
> The way you do it is to start with an empty brigade.
>
> > sub forward_get_as_post :FilterConnectionHandler
> > {
> > my ($f, $bb, $mode, $block, $readbytes) = @_;
> > my $ctx = $f->ctx || { 'state' => 'waiting_for_request_line' };
> >
> > warn "state = ".$ctx->{'state'}."\r\n";
> >
> > # check whether we need to process this request.
> > return Apache2::Const::DECLINED if ($ctx->{'state'} eq 'ignore');
> >
> > # read into a tmp brigade.
> > my $connection = $f->c;
> > my $tmp_bb = APR::Brigade->new($connection->pool,
> > $connection->bucket_alloc);
> > my $rv = $f->next->get_brigade($tmp_bb, $mode, $block, $readbytes);
> >
> > return $rv unless $rv == APR::Const::SUCCESS;
> >
> > while (!$tmp_bb->is_empty)
> > {
> > # pop buckets from this brigade.
> > my $bucket = $tmp_bb->first;
> > $bucket->remove();
> >
> > if ($ctx->{'state'} eq 'waiting_for_request_line')
> > {
> > # assumes request line is first bucket.
> > $bucket->read(my $request_line);
> > my ($method, $uri, $version) = ($request_line =~ m|^(.*?)
> (.*?)
> > HTTP/(.*?)\r\n$|);
> >
> > if (defined ($method) and $method eq "GET" and $uri =~
> > m|^/echo|)
> > {
> > my $new_uri = 'POST '.$uri.' HTTP/'.$version."\r\n";
> > my $new_uri_bucket =
> > APR::Bucket->new($connection->bucket_alloc, $new_uri);
> >
> > $bb->insert_tail($new_uri_bucket);
> >
> > my $bucket2 = APR::Bucket->new($connection->bucket_alloc,
> > "Content-Type: application/x-www-form-urlencoded\r\n");
> > $bb->insert_tail($bucket2);
> >
> > my $bucket3 = APR::Bucket->new($connection->bucket_alloc,
> > "Content-Length: 9\r\n");
> > $bb->insert_tail($bucket3);
>
> by now you have inserted these lines:
>
> POST /echo HTTP/1.1
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 9
>
> You have read the 1st bucket of the input. That bucket may (or may not)
> contain the whole request including the empty line that signals end-of-
> headers. So, you have to check also the rest of the bucket.
>
> >
> > $ctx->{'state'} = 'waiting_for_end_of_headers';
> > }
> > else
> > {
> > $bb->insert_tail($bucket);
> > $ctx->{'state'} = 'ignore';
> > }
> > }
> > elsif ($ctx->{'state'} eq 'waiting_for_end_of_headers')
> > {
> > $bucket->read(my $header);
> > warn "received header ... ".$header."\r\n";
> >
> > if ($header =~ m|^\r\n$|)
> > {
> > warn "detected end_of_headers\r\n";
> >
> > my $post_data = &get_post_data();
> >
> >
> > ### as soon as I add 'data=test' to this bucket the
> request
> > appears to hang.
> > ############################################
> >
> >
> > my $end_of_headers_bucket =
> > APR::Bucket->new($connection->bucket_alloc, "\r\ndata=test");
> >
> > $bb->insert_tail($end_of_headers_bucket);
>
> by now you have sent
>
> POST /echo HTTP/1.1
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 9
>
> data=test
>
> Correct me if I am wrong but don't you have to insert an EOS marker?
> Haven't
> tried connection input filter yet. But normally a stream is closed with a
>
> $bb->insert_tail(APR::Bucket::eos_create $connection->bucket_alloc);
>
> > $ctx->{'state'} = 'finished';
> > }
> > else
> > {
> > $bb->insert_tail($bucket);
> > }
> > }
> > }
> >
> > # set context.
> > $f->ctx($ctx);
> >
> > return Apache2::Const::OK;
> > }
>
> If it works with the EOS bucket it will probably work for most of the
> requests. But RFC2616 doesn't forbid GET requests to have a request body.
> That
> means you should read the input until EOS. Don't know if your code does
> that
> actually. Otherwise your client may still send the request while your
> handler
> is already done with the response.
>
> Torsten Förtsch
>
> --
> Need professional modperl support? Hire me! (http://foertsch.name)
>
> Like fantasy? http://kabatinte.net
>
Re: mp2] [mod_proxy] [filter] mod_proxy is not playing with my request filter.
Posted by Torsten Förtsch <to...@gmx.net>.
On Thursday, August 26, 2010 18:28:47 James Lee wrote:
A few comments below.
The goal of the code is to fill up $bb. Nothing is copied there unless you do.
The way you do it is to start with an empty brigade.
> sub forward_get_as_post :FilterConnectionHandler
> {
> my ($f, $bb, $mode, $block, $readbytes) = @_;
> my $ctx = $f->ctx || { 'state' => 'waiting_for_request_line' };
>
> warn "state = ".$ctx->{'state'}."\r\n";
>
> # check whether we need to process this request.
> return Apache2::Const::DECLINED if ($ctx->{'state'} eq 'ignore');
>
> # read into a tmp brigade.
> my $connection = $f->c;
> my $tmp_bb = APR::Brigade->new($connection->pool,
> $connection->bucket_alloc);
> my $rv = $f->next->get_brigade($tmp_bb, $mode, $block, $readbytes);
>
> return $rv unless $rv == APR::Const::SUCCESS;
>
> while (!$tmp_bb->is_empty)
> {
> # pop buckets from this brigade.
> my $bucket = $tmp_bb->first;
> $bucket->remove();
>
> if ($ctx->{'state'} eq 'waiting_for_request_line')
> {
> # assumes request line is first bucket.
> $bucket->read(my $request_line);
> my ($method, $uri, $version) = ($request_line =~ m|^(.*?) (.*?)
> HTTP/(.*?)\r\n$|);
>
> if (defined ($method) and $method eq "GET" and $uri =~
> m|^/echo|)
> {
> my $new_uri = 'POST '.$uri.' HTTP/'.$version."\r\n";
> my $new_uri_bucket =
> APR::Bucket->new($connection->bucket_alloc, $new_uri);
>
> $bb->insert_tail($new_uri_bucket);
>
> my $bucket2 = APR::Bucket->new($connection->bucket_alloc,
> "Content-Type: application/x-www-form-urlencoded\r\n");
> $bb->insert_tail($bucket2);
>
> my $bucket3 = APR::Bucket->new($connection->bucket_alloc,
> "Content-Length: 9\r\n");
> $bb->insert_tail($bucket3);
by now you have inserted these lines:
POST /echo HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Content-Length: 9
You have read the 1st bucket of the input. That bucket may (or may not)
contain the whole request including the empty line that signals end-of-
headers. So, you have to check also the rest of the bucket.
>
> $ctx->{'state'} = 'waiting_for_end_of_headers';
> }
> else
> {
> $bb->insert_tail($bucket);
> $ctx->{'state'} = 'ignore';
> }
> }
> elsif ($ctx->{'state'} eq 'waiting_for_end_of_headers')
> {
> $bucket->read(my $header);
> warn "received header ... ".$header."\r\n";
>
> if ($header =~ m|^\r\n$|)
> {
> warn "detected end_of_headers\r\n";
>
> my $post_data = &get_post_data();
>
>
> ### as soon as I add 'data=test' to this bucket the request
> appears to hang.
> ############################################
>
>
> my $end_of_headers_bucket =
> APR::Bucket->new($connection->bucket_alloc, "\r\ndata=test");
>
> $bb->insert_tail($end_of_headers_bucket);
by now you have sent
POST /echo HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Content-Length: 9
data=test
Correct me if I am wrong but don't you have to insert an EOS marker? Haven't
tried connection input filter yet. But normally a stream is closed with a
$bb->insert_tail(APR::Bucket::eos_create $connection->bucket_alloc);
> $ctx->{'state'} = 'finished';
> }
> else
> {
> $bb->insert_tail($bucket);
> }
> }
> }
>
> # set context.
> $f->ctx($ctx);
>
> return Apache2::Const::OK;
> }
If it works with the EOS bucket it will probably work for most of the
requests. But RFC2616 doesn't forbid GET requests to have a request body. That
means you should read the input until EOS. Don't know if your code does that
actually. Otherwise your client may still send the request while your handler
is already done with the response.
Torsten Förtsch
--
Need professional modperl support? Hire me! (http://foertsch.name)
Like fantasy? http://kabatinte.net
Re: mp2] [mod_proxy] [filter] mod_proxy is not playing with my
request filter.
Posted by James Lee <mo...@gmail.com>.
I'm still having issues attempting to modify the body of an http request -
this time using a FilterConnectionHandler.
I've had to park this due to deadlines and write it using LWP in a
PerlResponseHandler. This is far from ideal as some of the responses are
upwards of 5MB, and what I'm trying to do does feel very doable in mod_perl.
What I really want is to be able to do is pass the request off to mod_proxy
(which I assume is better setup for streaming responses through Apache ... I
may be wrong) and then through mod_deflate to compress the results.
I've attached the code and configuration if anybody has a second and could
shed some light over what I'm doing wrong.
I'm basically trying to convert a GET request to 'http://hostname/echo' into
a POST request and attach a payload, however as it hangs mod_proxy becomes
largely academic.
Again any help would be much appreciated.
Thanks, James.
PerlModule Example::Echo
PerlModule Example::ConnectionFilter
PerlInputFilterHandler Example::ConnectionFilter::forward_get_as_post
<Location /echo>
SetHandler modperl
PerlResponseHandler Example::Echo
</Location>
package Example::Echo;
use strict;
use warnings;
use Apache2::Const -compile => qw(OK);
use Apache2::RequestIO;
use Apache2::RequestRec;
sub handler
{
my $r = shift;
$r->content_type('text/plain');
$r->read(my $buffer, 1024);
$r->print("received post data: '".$buffer."'");
return Apache2::Const::OK;
}
1;
package Example::ConnectionFilter;
use strict;
use warnings;
use base qw/Apache2::Filter/;
use Apache2::Connection;
use Apache2::Const -compile => qw(OK DECLINED);
use Apache2::FilterRec;
use Apache2::Log;
use Apache2::RequestRec;
use Apache2::RequestIO;
use APR::Const -compile => ':common';
use APR::Brigade;
use APR::Bucket;
use APR::BucketType;
use APR::Error;
sub forward_get_as_post :FilterConnectionHandler
{
my ($f, $bb, $mode, $block, $readbytes) = @_;
my $ctx = $f->ctx || { 'state' => 'waiting_for_request_line' };
warn "state = ".$ctx->{'state'}."\r\n";
# check whether we need to process this request.
return Apache2::Const::DECLINED if ($ctx->{'state'} eq 'ignore');
# read into a tmp brigade.
my $connection = $f->c;
my $tmp_bb = APR::Brigade->new($connection->pool,
$connection->bucket_alloc);
my $rv = $f->next->get_brigade($tmp_bb, $mode, $block, $readbytes);
return $rv unless $rv == APR::Const::SUCCESS;
while (!$tmp_bb->is_empty)
{
# pop buckets from this brigade.
my $bucket = $tmp_bb->first;
$bucket->remove();
if ($ctx->{'state'} eq 'waiting_for_request_line')
{
# assumes request line is first bucket.
$bucket->read(my $request_line);
my ($method, $uri, $version) = ($request_line =~ m|^(.*?) (.*?)
HTTP/(.*?)\r\n$|);
if (defined ($method) and $method eq "GET" and $uri =~
m|^/echo|)
{
my $new_uri = 'POST '.$uri.' HTTP/'.$version."\r\n";
my $new_uri_bucket =
APR::Bucket->new($connection->bucket_alloc, $new_uri);
$bb->insert_tail($new_uri_bucket);
my $bucket2 = APR::Bucket->new($connection->bucket_alloc,
"Content-Type: application/x-www-form-urlencoded\r\n");
$bb->insert_tail($bucket2);
my $bucket3 = APR::Bucket->new($connection->bucket_alloc,
"Content-Length: 9\r\n");
$bb->insert_tail($bucket3);
$ctx->{'state'} = 'waiting_for_end_of_headers';
}
else
{
$bb->insert_tail($bucket);
$ctx->{'state'} = 'ignore';
}
}
elsif ($ctx->{'state'} eq 'waiting_for_end_of_headers')
{
$bucket->read(my $header);
warn "received header ... ".$header."\r\n";
if ($header =~ m|^\r\n$|)
{
warn "detected end_of_headers\r\n";
my $post_data = &get_post_data();
### as soon as I add 'data=test' to this bucket the request
appears to hang.
############################################
my $end_of_headers_bucket =
APR::Bucket->new($connection->bucket_alloc, "\r\ndata=test");
$bb->insert_tail($end_of_headers_bucket);
$ctx->{'state'} = 'finished';
}
else
{
$bb->insert_tail($bucket);
}
}
}
# set context.
$f->ctx($ctx);
return Apache2::Const::OK;
}
1;
On 25 August 2010 09:41, James Lee <mo...@gmail.com> wrote:
> Hi Andre, thanks for the response.
>
> I don't actually want to use a PerlResponseHandler, I was just using that
> to make sure my filter did what I wanted it to do.
>
> I actually wanted the request filter to add the POST body expecting then
> that mod_proxy would do the rest. I expected mod_proxy to kick in around
> PerlTransHandler time but wasn't sure when my request filter got called.
> Either way it wasn't working :)
>
> I may see whether I can get the connection filter to add the content I was
> just after a little clarification when filters were called in relation to
> mod_proxy.
>
> Thanks again.
>
>
>
> On 25 August 2010 07:12, André Warnier <aw...@ice-sa.com> wrote:
>
>> James Lee wrote:
>>
>>> Hi mod_perl community, this is my first post so be gentle with me.
>>>
>>> I'm trying to create something which translates a GET request into a POST
>>> (creating the body dynamically) and then hand off the POST to another
>>> server.
>>>
>>> I have created a mod_perl connection filter to change the GET to a POST
>>> in
>>> the request line. It also adds Content-Length and Content-Type headers
>>> and
>>> this works fine.
>>>
>>> I then have a request filter which creates the POST body. This works when
>>> the request is handled by a PerlResponseHandler but I'm trying to offload
>>> the request to mod_proxy as the responses can be quite large but I just
>>> can't get it to play ball. I see the request filter being called but the
>>> POST body never arrives at the target server. I suspect this is a timing
>>> issue, ie: mod_proxy kicking in before my request filter but I'm not
>>> certain.
>>>
>>
>> Exactly. By the time your response handler is invoked, the time for
>> mod_proxy is long past.
>>
>> One approach would be : have your PerlResponseHandler send the request
>> directly to the back-end server, using LWP, read the response, and return
>> this response as the response for your response handler. In other words, do
>> yourself what mod_proxy would do.
>>
>>
>>
>>> Can anybody shed some light on this or correct my approach. I was going
>>> to
>>> add the POST body in the connection filter but it seemed cleaner/easier
>>> to
>>> do it in a request filter.
>>>
>>
>> In the mod_perl documentation, there is somewhere a schema of the
>> different Apache phases, and when the different handlers get invoked. I
>> beliebe mod_proxygets invoked somewhere around the Fixup phase, so if you
>> want to beat it, you have to be earlier than that.
>>
>>
>>
>>> I've looked on the mailing list and found a few things that touch on this
>>> (
>>> http://tech.groups.yahoo.com/group/modperl/message/54541) but nothing
>>> that's
>>> close enough to help.
>>>
>>> Config below ... I've not included perl code as this message is quite
>>> long
>>> anyway. Please let me know if it would be helpful.
>>> Thanks in advance, James.
>>>
>>>
>>> ***
>>> httpd.conf extract:
>>>
>>>
>>> PerlInputFilterHandler Sample::RequestTweaker::change_get_to_post
>>>
>>> <Location /reports>
>>> PerlInputFilterHandler Sample::RequestTweaker::inject_post_data
>>> ProxyPass http://appserver/reports-engine
>>> </Location>
>>>
>>>
>>
>
Re: mp2] [mod_proxy] [filter] mod_proxy is not playing with my
request filter.
Posted by James Lee <mo...@gmail.com>.
Hi Andre, thanks for the response.
I don't actually want to use a PerlResponseHandler, I was just using that to
make sure my filter did what I wanted it to do.
I actually wanted the request filter to add the POST body expecting then
that mod_proxy would do the rest. I expected mod_proxy to kick in around
PerlTransHandler time but wasn't sure when my request filter got called.
Either way it wasn't working :)
I may see whether I can get the connection filter to add the content I was
just after a little clarification when filters were called in relation to
mod_proxy.
Thanks again.
On 25 August 2010 07:12, André Warnier <aw...@ice-sa.com> wrote:
> James Lee wrote:
>
>> Hi mod_perl community, this is my first post so be gentle with me.
>>
>> I'm trying to create something which translates a GET request into a POST
>> (creating the body dynamically) and then hand off the POST to another
>> server.
>>
>> I have created a mod_perl connection filter to change the GET to a POST in
>> the request line. It also adds Content-Length and Content-Type headers and
>> this works fine.
>>
>> I then have a request filter which creates the POST body. This works when
>> the request is handled by a PerlResponseHandler but I'm trying to offload
>> the request to mod_proxy as the responses can be quite large but I just
>> can't get it to play ball. I see the request filter being called but the
>> POST body never arrives at the target server. I suspect this is a timing
>> issue, ie: mod_proxy kicking in before my request filter but I'm not
>> certain.
>>
>
> Exactly. By the time your response handler is invoked, the time for
> mod_proxy is long past.
>
> One approach would be : have your PerlResponseHandler send the request
> directly to the back-end server, using LWP, read the response, and return
> this response as the response for your response handler. In other words, do
> yourself what mod_proxy would do.
>
>
>
>> Can anybody shed some light on this or correct my approach. I was going to
>> add the POST body in the connection filter but it seemed cleaner/easier to
>> do it in a request filter.
>>
>
> In the mod_perl documentation, there is somewhere a schema of the different
> Apache phases, and when the different handlers get invoked. I beliebe
> mod_proxygets invoked somewhere around the Fixup phase, so if you want to
> beat it, you have to be earlier than that.
>
>
>
>> I've looked on the mailing list and found a few things that touch on this
>> (
>> http://tech.groups.yahoo.com/group/modperl/message/54541) but nothing
>> that's
>> close enough to help.
>>
>> Config below ... I've not included perl code as this message is quite long
>> anyway. Please let me know if it would be helpful.
>> Thanks in advance, James.
>>
>>
>> ***
>> httpd.conf extract:
>>
>>
>> PerlInputFilterHandler Sample::RequestTweaker::change_get_to_post
>>
>> <Location /reports>
>> PerlInputFilterHandler Sample::RequestTweaker::inject_post_data
>> ProxyPass http://appserver/reports-engine
>> </Location>
>>
>>
>
Re: mp2] [mod_proxy] [filter] mod_proxy is not playing with my request
filter.
Posted by André Warnier <aw...@ice-sa.com>.
James Lee wrote:
> Hi mod_perl community, this is my first post so be gentle with me.
>
> I'm trying to create something which translates a GET request into a POST
> (creating the body dynamically) and then hand off the POST to another
> server.
>
> I have created a mod_perl connection filter to change the GET to a POST in
> the request line. It also adds Content-Length and Content-Type headers and
> this works fine.
>
> I then have a request filter which creates the POST body. This works when
> the request is handled by a PerlResponseHandler but I'm trying to offload
> the request to mod_proxy as the responses can be quite large but I just
> can't get it to play ball. I see the request filter being called but the
> POST body never arrives at the target server. I suspect this is a timing
> issue, ie: mod_proxy kicking in before my request filter but I'm not
> certain.
Exactly. By the time your response handler is invoked, the time for mod_proxy is long past.
One approach would be : have your PerlResponseHandler send the request directly to the
back-end server, using LWP, read the response, and return this response as the response
for your response handler. In other words, do yourself what mod_proxy would do.
>
> Can anybody shed some light on this or correct my approach. I was going to
> add the POST body in the connection filter but it seemed cleaner/easier to
> do it in a request filter.
In the mod_perl documentation, there is somewhere a schema of the different Apache phases,
and when the different handlers get invoked. I beliebe mod_proxygets invoked somewhere
around the Fixup phase, so if you want to beat it, you have to be earlier than that.
>
> I've looked on the mailing list and found a few things that touch on this (
> http://tech.groups.yahoo.com/group/modperl/message/54541) but nothing that's
> close enough to help.
>
> Config below ... I've not included perl code as this message is quite long
> anyway. Please let me know if it would be helpful.
> Thanks in advance, James.
>
>
> ***
> httpd.conf extract:
>
>
> PerlInputFilterHandler Sample::RequestTweaker::change_get_to_post
>
> <Location /reports>
> PerlInputFilterHandler Sample::RequestTweaker::inject_post_data
> ProxyPass http://appserver/reports-engine
> </Location>
>