You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Nick Kew <ni...@webthing.com> on 2007/10/28 02:10:53 UTC
Proxying subrequests
Our subrequest API currently handles only local subrequests,
so for example mod_include doesn't support
<!--#include virtual="http://example.com/example.txt"-->.
I've worked around this in the past with some ugly hacks.
Looking at the code, it's actually very simple to build
in support for remote URLs into the subrequest mechanism
using mod_proxy. I've just test-driven a patch comprising
just six lines of code (attached), and it works.
The patch still needs attention to escaping, but before
I deal with that, can I ask for votes on the concept?
Is there any reason not to add this feature?
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/
Re: Proxying subrequests
Posted by Graham Leggett <mi...@sharp.fm>.
Nick Kew wrote:
> Our subrequest API currently handles only local subrequests,
> so for example mod_include doesn't support
> <!--#include virtual="http://example.com/example.txt"-->.
> I've worked around this in the past with some ugly hacks.
>
> Looking at the code, it's actually very simple to build
> in support for remote URLs into the subrequest mechanism
> using mod_proxy. I've just test-driven a patch comprising
> just six lines of code (attached), and it works.
>
> The patch still needs attention to escaping, but before
> I deal with that, can I ask for votes on the concept?
> Is there any reason not to add this feature?
I was under the impression this worrked, if it doesn't work, a definite
+1 to fix it.
Regards,
Graham
--
Re: Proxying subrequests
Posted by "Akins, Brian" <Br...@turner.com>.
On 11/1/07 6:48 PM, "Ian Holsman" <li...@holsman.net> wrote:
> Akins, Brian wrote:
>> -1 from me (if that counts.)
>>
>> Using ProxyPass should be fine for 95% of the use cases??
>>
>> ProxyPass /cnn http://www.cnn.com/
>>
>> <!--#include virtual="/cnn/WORLD/index.html" -->
>>
>
> yes.
> if you:
> a. have a static small number of hosts
> b. those hosts don't change often
>
> if either of these 2 conditions aren't met, then proxypass is next to
> useless.
True. (You could manage with a decent config tool, however.)
Couldn't some creative rewrite rules do the same thing:
RewriteCond %{IS_SUBREQ} true
RewriteRule ^/proxy/(.*) http://$1 [P,L]
Note: not sure if that will actually work.
I just foresee horrible cross-site type "vulnerabilities" in this.
If it is configurable on/off, then I'm -0.
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: Proxying subrequests
Posted by Ian Holsman <li...@holsman.net>.
Akins, Brian wrote:
> -1 from me (if that counts.)
>
> Using ProxyPass should be fine for 95% of the use cases??
>
> ProxyPass /cnn http://www.cnn.com/
>
> <!--#include virtual="/cnn/WORLD/index.html" -->
>
yes.
if you:
a. have a static small number of hosts
b. those hosts don't change often
if either of these 2 conditions aren't met, then proxypass is next to
useless.
Re: Proxying subrequests
Posted by "Akins, Brian" <Br...@turner.com>.
-1 from me (if that counts.)
Using ProxyPass should be fine for 95% of the use cases??
ProxyPass /cnn http://www.cnn.com/
<!--#include virtual="/cnn/WORLD/index.html" -->
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: Proxying subrequests
Posted by Ian Holsman <li...@holsman.net>.
Nick Kew wrote:
> Our subrequest API currently handles only local subrequests,
> so for example mod_include doesn't support
> <!--#include virtual="http://example.com/example.txt"-->.
> I've worked around this in the past with some ugly hacks.
>
> Looking at the code, it's actually very simple to build
> in support for remote URLs into the subrequest mechanism
> using mod_proxy. I've just test-driven a patch comprising
> just six lines of code (attached), and it works.
>
> The patch still needs attention to escaping, but before
> I deal with that, can I ask for votes on the concept?
> Is there any reason not to add this feature?
>
>
+1
I've wanted this feature in mod-include for years, and have been too
lazy to do it.
(relying on custom module hacks to implement it)
as for PHP security holes.. maybe a config option or something to
enable/disable it.
Re: Proxying subrequests
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
Mads Toftum wrote:
> On Sun, Oct 28, 2007 at 04:17:50PM +0200, Graham Leggett wrote:
>> I am not so sure. mod_proxy already supports client behaviour, and has the
>> bonus of using bucket brigades natively allowing zero copy natively.
>>
> That'd be true for serf as well, right? The whole point of getting
> something into apr was to get bucket brigedes.
We really need both pull and push model brigades, which I understand that svn
and/or serf solved. Pity those didn't make it back into apr-util.
Re: Proxying subrequests
Posted by Mads Toftum <ma...@toftum.dk>.
On Sun, Oct 28, 2007 at 04:17:50PM +0200, Graham Leggett wrote:
> I am not so sure. mod_proxy already supports client behaviour, and has the
> bonus of using bucket brigades natively allowing zero copy natively.
>
That'd be true for serf as well, right? The whole point of getting
something into apr was to get bucket brigedes.
vh
Mads Toftum
--
http://soulfood.dk
Re: Proxying subrequests
Posted by Graham Leggett <mi...@sharp.fm>.
Mads Toftum wrote:
> The need for http client code seems to pop up every so often. I think it
> would make a lot of sense to get a generic client lib in place (possibly
> in apr-land), before bolting on another bit of client code.
I am not so sure. mod_proxy already supports client behaviour, and has
the bonus of using bucket brigades natively allowing zero copy natively.
Creating a request on demand is usually as simple as creating a
subrequest and firing it off, as mod_include does.
Regards,
Graham
--
Re: Proxying subrequests
Posted by Mads Toftum <ma...@toftum.dk>.
On Sun, Oct 28, 2007 at 12:20:47PM +0000, Nick Kew wrote:
> Well, it involved my own HTTP Client code (duplication) and
> in the case of SSI, a separate <!--#include url="..."-->.
> (The duplication dates back to before I had commit here:-)
> Basically I want to eliminate that duplication.
>
The need for http client code seems to pop up every so often. I think it
would make a lot of sense to get a generic client lib in place (possibly
in apr-land), before bolting on another bit of client code.
Another example is
http://svn.apache.org/viewvc/httpd/mod_cache_requester/trunk/ which is
using curl.
I think perhaps a directive like SSIProxyURL defaulting to off would be
an absolute minimal requirement to keep unsspecting people out of
trouble.
vh
Mads Toftum
--
http://soulfood.dk
Re: Proxying subrequests
Posted by Nick Kew <ni...@webthing.com>.
On Sun, 28 Oct 2007 02:32:31 +0200
André Malo <nd...@perlig.de> wrote:
> * Nick Kew wrote:
>
> > Our subrequest API currently handles only local subrequests,
> > so for example mod_include doesn't support
> > <!--#include virtual="http://example.com/example.txt"-->.
> > I've worked around this in the past with some ugly hacks.
>
> How ugly are they? I'd rather accept them than...
Well, it involved my own HTTP Client code (duplication) and
in the case of SSI, a separate <!--#include url="..."-->.
(The duplication dates back to before I had commit here:-)
Basically I want to eliminate that duplication.
> It opens a big can of worms. See the neverending story of faulty php
scripts.
Is that because the PHP scripts are all reinventing this wheel?
> -0 from here. I think this is a point where mod_include should stop.
I'll answer that in a followup to another post.
> Besides that, shouldn't your patch trigger a gateway request instead
> of regular proxy?
That already works, in the sense that if a subrequest triggers a
ProxyPass, then that applies. I used PROXYREQ_PROXY because that
seemed appropriate to a request starting from a full URI.
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/
Re: Proxying subrequests
Posted by Ruediger Pluem <rp...@apache.org>.
On 10/28/2007 12:29 PM, Dr Stephen Henson wrote:
> Ruediger Pluem wrote:
>> Possibly mod_include should stop there, but "remote" subrequest are valuable
>> for other modules (e.g. mod_ssl and OCSP). So +1 to add this to the API,
>> -0 for enabling it for mod_include.
>>
>
> Does anyone have any pointers how OCSP might be done using subrequests
> in mod_ssl? I've been looking into this possibility.
>
> Other than a lack of documentation I hit a brick wall because
> request_rec is not available (and possibly hasn't been created yet?) at
> the time it is needed which would be during the initial connection
> before any HTTP headers were sent.
Damn it. I forgot that we do not have a request_rec at this point of time
(not created at this point of time).
Currently I have no idea, but I will let you know if I have one.
Regards
Rüdiger
Re: Proxying subrequests
Posted by Dr Stephen Henson <sh...@oss-institute.org>.
Ruediger Pluem wrote:
>
> Possibly mod_include should stop there, but "remote" subrequest are valuable
> for other modules (e.g. mod_ssl and OCSP). So +1 to add this to the API,
> -0 for enabling it for mod_include.
>
Does anyone have any pointers how OCSP might be done using subrequests
in mod_ssl? I've been looking into this possibility.
Other than a lack of documentation I hit a brick wall because
request_rec is not available (and possibly hasn't been created yet?) at
the time it is needed which would be during the initial connection
before any HTTP headers were sent.
Steve.
--
Dr Stephen N. Henson.
Core developer of the OpenSSL project: http://www.openssl.org/
Freelance consultant see: http://www.drh-consultancy.co.uk/
Email: shenson@drh-consultancy.co.uk, PGP key: via homepage.
Re: Proxying subrequests
Posted by Ruediger Pluem <rp...@apache.org>.
On 10/28/2007 02:32 AM, André Malo wrote:
> * Nick Kew wrote:
>
>> Our subrequest API currently handles only local subrequests,
>> so for example mod_include doesn't support
>> <!--#include virtual="http://example.com/example.txt"-->.
>> I've worked around this in the past with some ugly hacks.
>
> How ugly are they? I'd rather accept them than...
>
>> Looking at the code, it's actually very simple to build
>> in support for remote URLs into the subrequest mechanism
>> using mod_proxy. I've just test-driven a patch comprising
>> just six lines of code (attached), and it works.
>>
>> The patch still needs attention to escaping, but before
>> I deal with that, can I ask for votes on the concept?
>> Is there any reason not to add this feature?
>
> It opens a big can of worms. See the neverending story of faulty php
> scripts.
> -0 from here. I think this is a point where mod_include should stop.
Possibly mod_include should stop there, but "remote" subrequest are valuable
for other modules (e.g. mod_ssl and OCSP). So +1 to add this to the API,
-0 for enabling it for mod_include.
>
> Besides that, shouldn't your patch trigger a gateway request instead of
> regular proxy?
Thats also my feeling. PROXYREQ_REVERSE seems more appropriate.
Regards
Rüdiger
Re: Proxying subrequests
Posted by André Malo <nd...@perlig.de>.
* Nick Kew wrote:
> Our subrequest API currently handles only local subrequests,
> so for example mod_include doesn't support
> <!--#include virtual="http://example.com/example.txt"-->.
> I've worked around this in the past with some ugly hacks.
How ugly are they? I'd rather accept them than...
> Looking at the code, it's actually very simple to build
> in support for remote URLs into the subrequest mechanism
> using mod_proxy. I've just test-driven a patch comprising
> just six lines of code (attached), and it works.
>
> The patch still needs attention to escaping, but before
> I deal with that, can I ask for votes on the concept?
> Is there any reason not to add this feature?
It opens a big can of worms. See the neverending story of faulty php
scripts.
-0 from here. I think this is a point where mod_include should stop.
Besides that, shouldn't your patch trigger a gateway request instead of
regular proxy?
nd
--
s s^saaaaaoaaaoaaaaooooaaoaaaomaaaa a alataa aaoat a a
a maoaa a laoata a oia a o a m a o alaoooat aaool aaoaa
matooololaaatoto aaa o a o ms;s;\s;s;g;y;s;:;s;y#mailto: #
\51/\134\137| http://www.perlig.de #;print;# > nd@perlig.de
Re: Proxying subrequests
Posted by Nick Kew <ni...@webthing.com>.
On Sat, 27 Oct 2007 22:56:47 -0700
Paul Querna <ch...@force-elite.com> wrote:
> +0.5 in concept.
>
> -0.9 on enabling this by default in mod_includes. Make it possible to
> turn it on via httpd.conf, but never on by default....
That makes sense. In fact, the patch already accomplishes that,
in the sense that it requires explicit configuration of mod_proxy.
I used:
ProxyRequests On
<Proxy *>
Order Deny,Allow
Deny from all
Allow from 127.0.0.1
</Proxy>
And yes, if we enable it, we need an additional directive so that
it can be enabled without ProxyRequests, and the open invitation
for lusers to shoot themselves in the foot.
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/
Re: Proxying subrequests
Posted by Graham Leggett <mi...@sharp.fm>.
On Mon, October 29, 2007 3:34 pm, Jim Jagielski wrote:
> I tend to agree... This seems to open up a huge can
> of worms, and makes it v easy to people to use these "neat"
> feature and open themselves up to all kinds of
> nasty, nasty things.
As I understand it, mod_include is capable of including an URL that is
ProxyPass'ed, which does the same job surely?
Regards,
Graham
--
Re: Proxying subrequests
Posted by Arturo 'Buanzo' Busleiman <bu...@buanzo.com.ar>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Jim Jagielski wrote:
> I tend to agree... This seems to open up a huge can
> of worms, and makes it v easy to people to use these "neat"
> feature and open themselves up to all kinds of
> nasty, nasty things.
If being properly documented, and disabled-by-default, and also ACL-limited (domain, path, dstip of
foreign server), then this is a useful feature. In any case, PHP can already do that. An apache
level equivalent is perfect in terms of performance, and consistency.
And, any case, security is not a software-provided, but software-aided I guess, feature. Yes,
clueless admins might be open to XSI attacks, but... PHP for instance, provides register_globals in
Off default. Knowledgeable admins still can enable it if they know (or believe to know) what they're
doing. Sounds pretty similar a discussion to me.
- --
Arturo "Buanzo" Busleiman - Consultor Independiente en Seguridad Informatica
Servicios Ofrecidos: http://www.buanzo.com.ar/pro/
Unase a los Foros GNU/Buanzo - La palabra Comunidad en su maxima expresion.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFHJe9pAlpOsGhXcE0RCkDfAJ9rRg0z7DEl7kaP73+WZ1SRVFzoZQCdFmED
fCQY+V/qH6ye0Qwp3ole0uM=
=/XF4
-----END PGP SIGNATURE-----
Re: Proxying subrequests
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Oct 28, 2007, at 4:12 AM, Niklas Edmundsson wrote:
> On Sat, 27 Oct 2007, Paul Querna wrote:
>
>> -0.9 on enabling this by default in mod_includes. Make it
>> possible to
>> turn it on via httpd.conf, but never on by default....
>
> I agree.
>
> And it should have huge warning signs, and a long descriptive name
> that does not invite to "let's try this and see if it solves my
> problem".
>
> Cross-site-include-holes are nasty, and I see it as a feature that
> they are not "supported" ;)
I tend to agree... This seems to open up a huge can
of worms, and makes it v easy to people to use these "neat"
feature and open themselves up to all kinds of
nasty, nasty things.
Re: Proxying subrequests
Posted by Niklas Edmundsson <ni...@acc.umu.se>.
On Sat, 27 Oct 2007, Paul Querna wrote:
> -0.9 on enabling this by default in mod_includes. Make it possible to
> turn it on via httpd.conf, but never on by default....
I agree.
And it should have huge warning signs, and a long descriptive name
that does not invite to "let's try this and see if it solves my
problem".
Cross-site-include-holes are nasty, and I see it as a feature that
they are not "supported" ;)
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se
---------------------------------------------------------------------------
"The only one who can destroy your Tasha now, is you." Q
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: Proxying subrequests
Posted by Paul Querna <ch...@force-elite.com>.
Nick Kew wrote:
> Our subrequest API currently handles only local subrequests,
> so for example mod_include doesn't support
> <!--#include virtual="http://example.com/example.txt"-->.
> I've worked around this in the past with some ugly hacks.
>
> Looking at the code, it's actually very simple to build
> in support for remote URLs into the subrequest mechanism
> using mod_proxy. I've just test-driven a patch comprising
> just six lines of code (attached), and it works.
>
> The patch still needs attention to escaping, but before
> I deal with that, can I ask for votes on the concept?
> Is there any reason not to add this feature?
+0.5 in concept.
-0.9 on enabling this by default in mod_includes. Make it possible to
turn it on via httpd.conf, but never on by default....
-Paul