faking host headers

[Michael Turner <jm...@apple.com>]

Hi all,

Leif, we spoke yesterday in IRC about this but I wanted to follow up.  You mentioned faking the host headers (which we can do with curl) to load the cache with the vip address on both the independent real traffic servers.

Doing this, I could only get 404 results, and it never seemed to actually fetch from the origin.

$ curl -H "Host: <real_ats_server> -o /dev/null http://<ats server vip>/path_to_cache

If I removed the "Host: <real ats server>" component, it works.

Am I missing something? :)

Thanks,

Mikey

Re: faking host headers

[Leif Hedstrom <zw...@apache.org>]

On 4/26/12 4:16 PM, Michael Turner wrote:
> Hi all,
>
> Leif, we spoke yesterday in IRC about this but I wanted to follow up.  You mentioned faking the host headers (which we can do with curl) to load the cache with the vip address on both the independent real traffic servers.
>
> Doing this, I could only get 404 results, and it never seemed to actually fetch from the origin.
>
> $ curl -H "Host:<real_ats_server>  -o /dev/null http://<ats server vip>/path_to_cache
>
> If I removed the "Host:<real ats server>" component, it works.
>
> Am I missing something? :)
>

Hmmm, I do this all the time. Lets make sure I understand, and I'll make a 
fictitious setup:

     a. The SLB has a VIP of 192.168.201.69.

     b. The REALs behind the SLB are 192.168.202.2 and 192.168.202.3

     c. The SLB VIP has a DNS record:    ats.example.com. IN A 192.168.201.69


Normally, a user would fetch e.g. http://ats.example.com/file.jpg. For that 
to work , your ATS boxes would have a remap.config like

     map http://ats.example.com  http://origin.example.com


To fetch file.jpg from each host behind the SLB, you would do e.g.

     for i in 192.168.202.2 192.168.202.3; do
         curl -s -o /dev/null -D - -H "Host: ats.example.com" http://$i/file.jpg
     done


This is where it can get sketchy, your REALs must obviously have a way to 
route directly here, at yahoo we did tricks with iptables and route tables 
and what not, since the VIPs for the CDN were in-line and not DSR.

Now, for your specific error, I'm guessing, but not sure, that your 
remap.config is not matching what the Host: header is that you set via -H? 
Typically, a 404 means that the server can't figure out how to map the 
request to an origin (or, the origin is generating a 404, but that's easy to 
figure out).

Another thing to look at is if your "pristine" host header setting in 
records.config is on or off. What should it be? Well, it depends on what 
your origin server expects to see! If the origin expects to see Host: 
ats.example.com, then pristine host headers should be set to "1", it the 
origin (in this example) expects to see Host: origin.example.com, then 
pristine host headers should be set to "0".

I hope that helps.

Cheers,

-- Leif

Re: faking host headers

[tom ryan <to...@gmail.com>]

Hi Michael,

On Thu, Apr 26, 2012 at 11:16 PM, Michael Turner <jm...@apple.com> wrote:
> Hi all,
>
> Leif, we spoke yesterday in IRC about this but I wanted to follow up.  You mentioned faking the host headers (which we can do with curl) to load the cache with the vip address on both the independent real traffic servers.
>
> Doing this, I could only get 404 results, and it never seemed to actually fetch from the origin.
>
> $ curl -H "Host: <real_ats_server> -o /dev/null http://<ats server vip>/path_to_cache

Firstly, the line you pasted is missing a closing quote mark - but no
doubt this is just a typo in the mail if you are seeing a 404.

Secondly, have you considered listening with netcat to see what your
command actually sends?

$ nc -l -p 12345
meanwhile...
$ curl -H "Host: <real_ats_server>" -o /dev/null
http://localhost:12345/path_to_cache

I'm not sure if it's fair to call this method more or rather less
scientific than trying to do it The Right Way... but I find it useful
when two settings (eg your -H to curl) produce unexpectedly different
results.

It looks like you have a hardware load balancer in front of multiple
ats machines in front of some webserver/s, and it's that middle ats
layer you want to warm for all nodes at that layer... is that right?

best,
tom