You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Geoffrey Kneale <ge...@exceleratesoftware.com> on 2012/02/01 01:09:17 UTC
HTTP Headers in a forward proxy
Hello All,
Is it possible to manipulate the HTTP Headers in Trafficserver configured
to be a forward proxy? My goal is to support the YouTube Education filter,
which utilizes either A or B below.
A) Add new HTTP header rule
Modify your hardware filter or proxy server settings so that all outgoing
traffic to youtube.com contains the following custom HTTP header. The ID to
use in the HTTP header configuration, written below, is an example of a
unique ID for your school’s network only. If your school is blocked at the
district level, this HTTP header is then unique to the district network.
Example:
X-YouTube-Edu-Filter:ABCD1234567890abcdef
Field Name: X-YouTube-Edu-Filter
Description: When YouTube sees this header and an accompanying valid school
ID in the incoming traffic, YouTube will serve a limited EDU-only site to
all computers behind the school hardware filter.
Field Value Format: alphanumeric [a-z][A-Z][0-9]
Field Value Length: up to 44 characters
B) Create URL parameter rewrite rule
If your hardware filter does not support HTTP header modification, please
rewrite all outgoing URLs to youtube.com by appending the parameter
“edufilter” at the end. Exclude the &edufilter parameter from these file
types: .css, .gif, .png, .js, .xml
Example:
http://youtube.com/?edufilter=ABCD1234567890abcdef
http://youtube.com/watch?v=gM95HHI4gLk&edufilter=ABCD1234567890afbcdef
Thanks in advance,
Geoffrey
Re: HTTP Headers in a forward proxy
Posted by Leif Hedstrom <zw...@apache.org>.
On 1/31/12 9:19 PM, James Peach wrote:
> On 31/01/2012, at 8:13 PM, Alan M. Carroll wrote:
>
>> I think you would need to write a plugin, but both of those should be easy to do. The plugin would not be very complex (50-100 lines of code). A Type A plugin would be simpler to implement.
> Could the header_filter plugin do this?
>
> <http://trafficserver.apache.org/docs/trunk/admin/plugins/header_filter/index.en.html>
It would, unless the header value is "dynamic" in nature (e.g. generated on
the fly for each request).
-- Leif
Re: HTTP Headers in a forward proxy
Posted by Conan Wang <bu...@gmail.com>.
I didn't try the header_filter plugin. I wrote my plugin because I prefer
"remap" only mode and I can write config in remap.config (for easy
management and reloading support).
And I guess "global" plugin is not faster than "remap" plugin if have many
map rules. Because every request will go through the global part of the
"global" plugin even if we don't want plugin to process that request. (
just my general opinion about global and remap plugin, not for
header_filter plugin)
Anyway, I'm sure the global plugin is fast enough according to my test
before. :)
2012/2/6 Leif Hedstrom <zw...@apache.org>
> On 2/5/12 7:40 PM, Conan Wang wrote:
>
>> btw, I'v written a simple remap plugin to do this stuff ( add one header
>> to origin server ). I modify code in "ats/example/add-header" from
>> "global" plugin to "remap" plugin, so you don't need to configure
>> plugin.config.
>>
>> https://github.com/wkl/add_**header <https://github.com/wkl/add_header>
>> (the header "key:value" should not contain space)
>>
>
> Did the header_filter plugin not work? If so, I'm curious why so I can fix
> it :).
>
> -- Leif
>
>
--
Regards,
Conan
Re: HTTP Headers in a forward proxy
Posted by Leif Hedstrom <zw...@apache.org>.
On 2/5/12 7:40 PM, Conan Wang wrote:
> btw, I'v written a simple remap plugin to do this stuff ( add one header
> to origin server ). I modify code in "ats/example/add-header" from
> "global" plugin to "remap" plugin, so you don't need to configure
> plugin.config.
>
> https://github.com/wkl/add_header
> (the header "key:value" should not contain space)
Did the header_filter plugin not work? If so, I'm curious why so I can fix
it :).
-- Leif
Re: HTTP Headers in a forward proxy
Posted by Conan Wang <bu...@gmail.com>.
btw, I'v written a simple remap plugin to do this stuff ( add one header to
origin server ). I modify code in "ats/example/add-header" from "global"
plugin to "remap" plugin, so you don't need to configure plugin.config.
https://github.com/wkl/add_header
(the header "key:value" should not contain space)
2012/2/3 Geoffrey Kneale <ge...@exceleratesoftware.com>
> Thanks Leif!
>
> I have a working config. To note, I also needed to configure the
> records.config to NOT force pristine headers.
>
> In summary:
> 1. Configure records.config to not force remap and not force pristine
> headers.
> 2. Download and install header filter plugin.
> 3. Configure plugins to use header filter (global w/o configuration file).
> 4. Configure the configuration file for the header filter.
> 5. Configure remap to use the header filter with Youtube requests.
>
> Invaluable tool:
> 1. Wireshark with display filter: http.response !=0 || http.request.method
> != "TRACE" and check the Hypertext Transfer Protocol section for the header
> modifications.
>
>
>
> On Wed, Feb 1, 2012 at 7:24 PM, Leif Hedstrom <zw...@apache.org> wrote:
>
>> On 2/1/12 5:33 PM, Leif Hedstrom wrote:
>>
>>> On 2/1/12 3:35 PM, Alan M. Carroll wrote:
>>>
>>>> Wednesday, February 1, 2012, 3:22:11 PM, you wrote:
>>>>
>>>> --- remap.config ---
>>>>> regex_map http://www.youtube.comhttp://**www.youtube.com@plugin=/**
>>>>> libexec/trafficserver/header_**filter.so @pparam=/etc/trafficserver/**
>>>>> youtube.config
>>>>> regex_map http://(.*) http://$1
>>>>>
>>>>
>>>>
>> Two more things while I remember it:
>>
>> 1) You shouldn't have to type the full path to the plugin, as long as you
>> installed it in the normal install directory. I.e. @plugin=header_filter.so
>> should work.
>>
>> 2) This particular plugin is a bit "strange", in that it is both a remap
>> plugin, and normal (hooks) plugin. This means for it to function properly,
>> you also have to add a line to plugins.config, to make sure all the hooks
>> are activated. I don't know if the documentation specifies this or not?
>>
>> Cheers,
>>
>> -- Leif
>>
>>
>
--
Regards,
Conan
Re: HTTP Headers in a forward proxy
Posted by Geoffrey Kneale <ge...@exceleratesoftware.com>.
Thanks Leif!
I have a working config. To note, I also needed to configure the
records.config to NOT force pristine headers.
In summary:
1. Configure records.config to not force remap and not force pristine
headers.
2. Download and install header filter plugin.
3. Configure plugins to use header filter (global w/o configuration file).
4. Configure the configuration file for the header filter.
5. Configure remap to use the header filter with Youtube requests.
Invaluable tool:
1. Wireshark with display filter: http.response !=0 || http.request.method
!= "TRACE" and check the Hypertext Transfer Protocol section for the header
modifications.
On Wed, Feb 1, 2012 at 7:24 PM, Leif Hedstrom <zw...@apache.org> wrote:
> On 2/1/12 5:33 PM, Leif Hedstrom wrote:
>
>> On 2/1/12 3:35 PM, Alan M. Carroll wrote:
>>
>>> Wednesday, February 1, 2012, 3:22:11 PM, you wrote:
>>>
>>> --- remap.config ---
>>>> regex_map http://www.youtube.comhttp://**www.youtube.com@plugin=/**
>>>> libexec/trafficserver/header_**filter.so @pparam=/etc/trafficserver/**
>>>> youtube.config
>>>> regex_map http://(.*) http://$1
>>>>
>>>
>>>
> Two more things while I remember it:
>
> 1) You shouldn't have to type the full path to the plugin, as long as you
> installed it in the normal install directory. I.e. @plugin=header_filter.so
> should work.
>
> 2) This particular plugin is a bit "strange", in that it is both a remap
> plugin, and normal (hooks) plugin. This means for it to function properly,
> you also have to add a line to plugins.config, to make sure all the hooks
> are activated. I don't know if the documentation specifies this or not?
>
> Cheers,
>
> -- Leif
>
>
Re: HTTP Headers in a forward proxy
Posted by Leif Hedstrom <zw...@apache.org>.
On 2/1/12 5:33 PM, Leif Hedstrom wrote:
> On 2/1/12 3:35 PM, Alan M. Carroll wrote:
>> Wednesday, February 1, 2012, 3:22:11 PM, you wrote:
>>
>>> --- remap.config ---
>>> regex_map
>>> http://www.youtube.comhttp://www.youtube.com@plugin=/libexec/trafficserver/header_filter.so
>>> @pparam=/etc/trafficserver/youtube.config
>>> regex_map http://(.*) http://$1
>>
Two more things while I remember it:
1) You shouldn't have to type the full path to the plugin, as long as
you installed it in the normal install directory. I.e.
@plugin=header_filter.so should work.
2) This particular plugin is a bit "strange", in that it is both a remap
plugin, and normal (hooks) plugin. This means for it to function
properly, you also have to add a line to plugins.config, to make sure
all the hooks are activated. I don't know if the documentation specifies
this or not?
Cheers,
-- Leif
Re: HTTP Headers in a forward proxy
Posted by Leif Hedstrom <zw...@apache.org>.
On 2/1/12 3:35 PM, Alan M. Carroll wrote:
> Wednesday, February 1, 2012, 3:22:11 PM, you wrote:
>
>> --- remap.config ---
>> regex_map http://www.youtube.comhttp://www.youtube.com@plugin=/libexec/trafficserver/header_filter.so @pparam=/etc/trafficserver/youtube.config
>> regex_map http://(.*) http://$1
> Are there really no spaces in that? You need one in "comhttp" ("com http") and one before the '@'. Any reason for using
I think your MUA messed it up, it had spaces in my client :).
In any case, I'm guessing that the URL(s) don't match that first rule (which
is a strange rule to have as a regex, since it has no regexes in it). I also
think the last rule is unnecessary, either use "map /" or no map rule at all
(since you disabled remap required).
> My experience is that all the You Tube URLS end in youtube.com. He would want to check his logs, though, to be sure.
>
> I have to be careful, I've never actually used the remap stuff, but something more like
>
> map http://youtube.com http://youtube.com @plugin=headerfilter.so @pparam=/etc/trafficserver/youtube.config
> regex_map http://(.+)[.]youtube[.]com/(.*) http://$1.youtube.com/$2 @plugin=header_filter.so @pparam=/etc/trafficserver/youtube.config
Yeah, that makes more sense.
-- Leif
Re: HTTP Headers in a forward proxy
Posted by "Alan M. Carroll" <am...@network-geographics.com>.
Wednesday, February 1, 2012, 3:22:11 PM, you wrote:
> --- remap.config ---
> regex_map http://www.youtube.comhttp://www.youtube.com@plugin=/libexec/trafficserver/header_filter.so @pparam=/etc/trafficserver/youtube.config
> regex_map http://(.*) http://$1
Are there really no spaces in that? You need one in "comhttp" ("com http") and one before the '@'. Any reason for using regex_map instead of map? You should use map for straight text, and regex_map for actual regular expressions (note the difference in Leif's original email). If you're not familiar with regular expressions, we can point you documentation for that.
My experience is that all the You Tube URLS end in youtube.com. He would want to check his logs, though, to be sure.
I have to be careful, I've never actually used the remap stuff, but something more like
map http://youtube.com http://youtube.com @plugin=headerfilter.so @pparam=/etc/trafficserver/youtube.config
regex_map http://(.+)[.]youtube[.]com/(.*) http://$1.youtube.com/$2 @plugin=header_filter.so @pparam=/etc/trafficserver/youtube.config
Put the header_filter.so in the libexec directory of ATS, that is where it will look w/o a path. Otherwise you need to be sure that ATS is installed in the root directory for what you had. If you didn't configure that, it's more likely to be in /usr/local/.
Also, did you actually compile the header_filter.so so that it can be placed in the libexec directory?
Re: HTTP Headers in a forward proxy
Posted by Geoffrey Kneale <ge...@exceleratesoftware.com>.
Hello Alan and Leif,
Thank you for your insight.
I have downloaded and installed the header_filter plugin. However, I am
having some difficulty with making it work. I can visit YouTube and other
pages, but it seems to be ignoring the header manipulation. If you see any
glaring mistakes, I would appreciate and advise you may have.
So far, I have the following configuration items:
--- records.config ---
CONFIG proxy.config.url_remap.remap_required INT 0
--- remap.config ---
regex_map http://www.youtube.com
http://www.youtube.com@plugin=/libexec/trafficserver/header_filter.so
@pparam=/etc/trafficserver/youtube.config
regex_map http://(.*) http://$1
--- youtube.config ---
[SEND_REQUEST_HDR]
X-YouTube-Edu-Filter +*youtube_code_here*+
On Wed, Feb 1, 2012 at 6:56 AM, Leif Hedstrom <zw...@apache.org> wrote:
> On 2/1/12 7:28 AM, Alan M. Carroll wrote:
>
>> I think it would be a good start, but the problem would be that he wanted
>> this in a forward proxy. He would have to set up remap rules to trigger the
>> header filtering only on the appropriate targets. I'm not sure if it would
>> be simpler to tweak the plugin to check the URL or not.
>>
>
> Yeah. He could (as you said) create one (or a few) remap.config rules for
> youtube (he has to make sure remap.config is not required, but enabled via
> the reverse proxy config). E.g.
>
> map http://static.youtube.com http://static.youtube.com@plugin=header_filter.so @pparam=youtube.config
>
>
> or some such. Also remember he could use regular expressions here (on the
> host part). e.g.
>
> regex_map http://(.+)\.youtube\.(.+) http://$1.youtube.$2
> @plugin=header_filter.so
>
>
> (or some such, I don't know what the various youtube URLs looks like).
>
> Cheers,
>
> -- Leif
>
>
Re: HTTP Headers in a forward proxy
Posted by Leif Hedstrom <zw...@apache.org>.
On 2/1/12 7:28 AM, Alan M. Carroll wrote:
> I think it would be a good start, but the problem would be that he wanted this in a forward proxy. He would have to set up remap rules to trigger the header filtering only on the appropriate targets. I'm not sure if it would be simpler to tweak the plugin to check the URL or not.
Yeah. He could (as you said) create one (or a few) remap.config rules for
youtube (he has to make sure remap.config is not required, but enabled via
the reverse proxy config). E.g.
map http://static.youtube.com http://static.youtube.com
@plugin=header_filter.so @pparam=youtube.config
or some such. Also remember he could use regular expressions here (on the
host part). e.g.
regex_map http://(.+)\.youtube\.(.+) http://$1.youtube.$2
@plugin=header_filter.so
(or some such, I don't know what the various youtube URLs looks like).
Cheers,
-- Leif
Re: HTTP Headers in a forward proxy
Posted by "Alan M. Carroll" <am...@network-geographics.com>.
I think it would be a good start, but the problem would be that he wanted this in a forward proxy. He would have to set up remap rules to trigger the header filtering only on the appropriate targets. I'm not sure if it would be simpler to tweak the plugin to check the URL or not.
Tuesday, January 31, 2012, 10:19:31 PM, you wrote:
> On 31/01/2012, at 8:13 PM, Alan M. Carroll wrote:
>> I think you would need to write a plugin, but both of those should be easy to do. The plugin would not be very complex (50-100 lines of code). A Type A plugin would be simpler to implement.
> Could the header_filter plugin do this?
> <http://trafficserver.apache.org/docs/trunk/admin/plugins/header_filter/index.en.html>
Re: HTTP Headers in a forward proxy
Posted by James Peach <jp...@apache.org>.
On 31/01/2012, at 8:13 PM, Alan M. Carroll wrote:
> I think you would need to write a plugin, but both of those should be easy to do. The plugin would not be very complex (50-100 lines of code). A Type A plugin would be simpler to implement.
Could the header_filter plugin do this?
<http://trafficserver.apache.org/docs/trunk/admin/plugins/header_filter/index.en.html>
>
> Tuesday, January 31, 2012, 6:09:17 PM, you wrote:
>
>> Is it possible to manipulate the HTTP Headers in Trafficserver configured to be a forward proxy? My goal is to support the YouTube Education filter, which utilizes either A or B below.
>
>
>> A) Add new HTTP header rule
>> Modify your hardware filter or proxy server settings so that all outgoing traffic to youtube.com contains the following custom HTTP header.
>> Example:
>> X-YouTube-Edu-Filter:ABCD1234567890abcdef
>
>> B) Create URL parameter rewrite rule
>> If your hardware filter does not support HTTP header modification, please rewrite all outgoing URLs to youtube.com by appending the parameter “edufilter” at the end. Exclude the &edufilter parameter from these file types: .css, .gif, .png, .js, .xml
>>
>> Example:
>> http://youtube.com/?edufilter=ABCD1234567890abcdef
>> http://youtube.com/watch?v=gM95HHI4gLk&edufilter=ABCD1234567890afbcdef
>>
>
Re: HTTP Headers in a forward proxy
Posted by "Alan M. Carroll" <am...@network-geographics.com>.
I think you would need to write a plugin, but both of those should be easy to do. The plugin would not be very complex (50-100 lines of code). A Type A plugin would be simpler to implement.
Tuesday, January 31, 2012, 6:09:17 PM, you wrote:
> Is it possible to manipulate the HTTP Headers in Trafficserver configured to be a forward proxy? My goal is to support the YouTube Education filter, which utilizes either A or B below.
> A) Add new HTTP header rule
> Modify your hardware filter or proxy server settings so that all outgoing traffic to youtube.com contains the following custom HTTP header.
> Example:
> X-YouTube-Edu-Filter:ABCD1234567890abcdef
> B) Create URL parameter rewrite rule
> If your hardware filter does not support HTTP header modification, please rewrite all outgoing URLs to youtube.com by appending the parameter “edufilter” at the end. Exclude the &edufilter parameter from these file types: .css, .gif, .png, .js, .xml
>
> Example:
> http://youtube.com/?edufilter=ABCD1234567890abcdef
> http://youtube.com/watch?v=gM95HHI4gLk&edufilter=ABCD1234567890afbcdef
>