You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Daniel Biazus <da...@azion.com> on 2014/05/13 20:14:49 UTC

traffic server cache key

Hi Guys,

    We've been using traffic server as a reverse proxy to delivery static
content across multiple CDN nodes, though the regex purge works as expected
through web UI tool, We've been facing some challenges regarding the cache
keys when the objects are stored in cache.
    In order to have a better understanding of our scenario, let's take a
look at this example:

Scenario 1:

- traffic server is working as a reverse proxy:

#remap.config

map http://finalurl.foo.bar/   http://originserver.foo.bar/
reverse_map  http://originserver.foo.bar/  http://finalurl.foo.bar/

- We have to purge all css files under the URL http://finalurl.foo.bar/

Considering this, we have to purge the following regex:

- http://finalurl.foo.bar/*.css

The example above doesn't work due the fact that traffic server uses the
origin URL to calculate the hash key, then, in this case we have to make a
purge request using the origin server URL (that's not nice with our
clients, right ?):

- http://originserver.foo.bar/*.css

So guys, anyone knows if there is a way to change the hash key in traffic
server, or even if is there some workaround to deal with this scenario ?


Thanks in advance,

-- 
Daniel Biazus

Porto Alegre, Brasil + 55 51 3012 3005
Miami, USA + 1 305 704 8816

Quaisquer informações contidas neste e-mail e anexos podem ser
confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma
de utilização deste documento depende de autorização do emissor, sujeito as
penalidades cabíveis.

Any information in this e-mail and attachments may be confidential and
privileged, protected by legal confidentiality. The use of this document
require authorization by the issuer, subject to penalties.

Re: traffic server cache key

Posted by Acácio Centeno <ac...@azion.com>.
Found open issue https://issues.apache.org/jira/browse/TS-1767 that
describes exactly this case.


On Thu, May 15, 2014 at 1:38 PM, Acácio Centeno <ac...@azion.com>wrote:

> Hello,
>
> I work on the same team as Daniel, and this was what I observed. Using
> pristine_host_hdr set to 1 the original URL is indeed used as cache key
> (I'm calling the original URL the one the client sends). Therefore, if we
> do a cache lookup or delete using the original URL, the object is found.
>
> However, if we do a _regex_ lookup/delete using the original URL, then
> it's not found.
>
> I read the code for the regex procedures (ShowCache::
> handleCacheScanCallback) and found out that it will read every object in
> the cache, and for each one it will print its URL to a buffer, then
> comparing this buffer with the informed regex. To build this buffer, the
> code gets a pointer to a HTTPHdr object, and if it has the field
> m_target_in_url set to true, the code will not consider the Host header. It
> turns out that in our tests, the object does have this field set, because
> the request being sent to the origin has this format:
>
> GET http://<remapped_domain><URI> HTTP/1.1
> ...
> Host: <original_domain>
> ...
>
> That is, the GET line has the remapped URL, instead of just the URI or the
> original URL.
>
> Therefore, using the remapped domain, the regex procedure does not match
> the expression we're sending to it, as our expression uses the original
> domain.
>
> Is there a configuration we could do to force ATS to send just the URI on
> the GET line? I think that if such a config exists, we would get the
> behavior we were expecting, as the UrlPrintHack ctor does consider the Host
> header if m_target_in_url is not set.
>
> If such a configuration does not exist, do you think it would be the case
> to patch the URL building done by UrlPrintHack to check if
> pristine_host_hdr is set and, if so, build the URL using the Host header
> instead of the one in the GET line?
>
> Thanks,
> Acácio.
>
>
> On Wed, May 14, 2014 at 6:16 PM, Leif Hedstrom <zw...@apache.org> wrote:
>
>>
>> On May 13, 2014, at 12:14 PM, Daniel Biazus <da...@azion.com>
>> wrote:
>>
>> Hi Guys,
>>
>>     We've been using traffic server as a reverse proxy to delivery static
>> content across multiple CDN nodes, though the regex purge works as expected
>> through web UI tool, We've been facing some challenges regarding the cache
>> keys when the objects are stored in cache.
>>     In order to have a better understanding of our scenario, let's take a
>> look at this example:
>>
>> Scenario 1:
>>
>> - traffic server is working as a reverse proxy:
>>
>> #remap.config
>>
>> map http://finalurl.foo.bar/   http://originserver.foo.bar/
>> reverse_map  http://originserver.foo.bar/  http://finalurl.foo.bar/
>>
>> - We have to purge all css files under the URL http://finalurl.foo.bar/
>>
>> Considering this, we have to purge the following regex:
>>
>> - http://finalurl.foo.bar/*.css
>>
>> The example above doesn't work due the fact that traffic server uses the
>> origin URL to calculate the hash key, then, in this case we have to make a
>> purge request using the origin server URL (that's not nice with our
>> clients, right ?):
>>
>> - http://originserver.foo.bar/*.css
>>
>> So guys, anyone knows if there is a way to change the hash key in traffic
>> server, or even if is there some workaround to deal with this scenario ?
>>
>>
>>
>> The common way of doing this is to have a plugin, that generates cache
>> keys based on a combination of the URL and a generation ID. This would
>> easily be configurable per remap rule, such that you can then instantly
>> (O(1)) purge all objects for that remap rule simply by modifying
>> (increasing) the generation ID. There is one such plugin in the public
>> domain, although it probably needs a little loving. :) An easy adoption to
>> that plugin would be to take the generation ID as an argument on the remap
>> rule.
>>
>> — Leif
>>
>>
>
>
> --
> Acácio Centeno
>
> Porto Alegre, Brasil + 55 51 3012 3005
> Miami, USA + 1 305 704 8816
>
> Quaisquer informações contidas neste e-mail e anexos podem ser
> confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma
> de utilização deste documento depende de autorização do emissor, sujeito as
> penalidades cabíveis.
>
> Any information in this e-mail and attachments may be confidential and
> privileged, protected by legal confidentiality. The use of this document
> require authorization by the issuer, subject to penalties.
>



-- 
Acácio Centeno

Porto Alegre, Brasil + 55 51 3012 3005
Miami, USA + 1 305 704 8816

Quaisquer informações contidas neste e-mail e anexos podem ser
confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma
de utilização deste documento depende de autorização do emissor, sujeito as
penalidades cabíveis.

Any information in this e-mail and attachments may be confidential and
privileged, protected by legal confidentiality. The use of this document
require authorization by the issuer, subject to penalties.

Re: traffic server cache key

Posted by "Alan M. Carroll" <am...@network-geographics.com>.
I tried looking at this today but when I do the tests the request sent to the origin server does *not* have a host name in the GET line.

However, there does seem to be something funky about how it's stored in the cache which (I think) makes it appear as if the URL has the host in it. I need to investigate a bit more but it's plausible the problem is the m_target_in_url is set incorrectly as others have noted.>  

> GET http://<remapped_domain><URI> HTTP/1.1
> ...
> Host: <original_domain>
> ...


Re: traffic server cache key

Posted by Acácio Centeno <ac...@azion.com>.
Hello,

I work on the same team as Daniel, and this was what I observed. Using
pristine_host_hdr set to 1 the original URL is indeed used as cache key
(I'm calling the original URL the one the client sends). Therefore, if we
do a cache lookup or delete using the original URL, the object is found.

However, if we do a _regex_ lookup/delete using the original URL, then it's
not found.

I read the code for the regex procedures (ShowCache::
handleCacheScanCallback) and found out that it will read every object in
the cache, and for each one it will print its URL to a buffer, then
comparing this buffer with the informed regex. To build this buffer, the
code gets a pointer to a HTTPHdr object, and if it has the field
m_target_in_url set to true, the code will not consider the Host header. It
turns out that in our tests, the object does have this field set, because
the request being sent to the origin has this format:

GET http://<remapped_domain><URI> HTTP/1.1
...
Host: <original_domain>
...

That is, the GET line has the remapped URL, instead of just the URI or the
original URL.

Therefore, using the remapped domain, the regex procedure does not match
the expression we're sending to it, as our expression uses the original
domain.

Is there a configuration we could do to force ATS to send just the URI on
the GET line? I think that if such a config exists, we would get the
behavior we were expecting, as the UrlPrintHack ctor does consider the Host
header if m_target_in_url is not set.

If such a configuration does not exist, do you think it would be the case
to patch the URL building done by UrlPrintHack to check if
pristine_host_hdr is set and, if so, build the URL using the Host header
instead of the one in the GET line?

Thanks,
Acácio.


On Wed, May 14, 2014 at 6:16 PM, Leif Hedstrom <zw...@apache.org> wrote:

>
> On May 13, 2014, at 12:14 PM, Daniel Biazus <da...@azion.com>
> wrote:
>
> Hi Guys,
>
>     We've been using traffic server as a reverse proxy to delivery static
> content across multiple CDN nodes, though the regex purge works as expected
> through web UI tool, We've been facing some challenges regarding the cache
> keys when the objects are stored in cache.
>     In order to have a better understanding of our scenario, let's take a
> look at this example:
>
> Scenario 1:
>
> - traffic server is working as a reverse proxy:
>
> #remap.config
>
> map http://finalurl.foo.bar/   http://originserver.foo.bar/
> reverse_map  http://originserver.foo.bar/  http://finalurl.foo.bar/
>
> - We have to purge all css files under the URL http://finalurl.foo.bar/
>
> Considering this, we have to purge the following regex:
>
> - http://finalurl.foo.bar/*.css
>
> The example above doesn't work due the fact that traffic server uses the
> origin URL to calculate the hash key, then, in this case we have to make a
> purge request using the origin server URL (that's not nice with our
> clients, right ?):
>
> - http://originserver.foo.bar/*.css
>
> So guys, anyone knows if there is a way to change the hash key in traffic
> server, or even if is there some workaround to deal with this scenario ?
>
>
>
> The common way of doing this is to have a plugin, that generates cache
> keys based on a combination of the URL and a generation ID. This would
> easily be configurable per remap rule, such that you can then instantly
> (O(1)) purge all objects for that remap rule simply by modifying
> (increasing) the generation ID. There is one such plugin in the public
> domain, although it probably needs a little loving. :) An easy adoption to
> that plugin would be to take the generation ID as an argument on the remap
> rule.
>
> — Leif
>
>


-- 
Acácio Centeno

Porto Alegre, Brasil + 55 51 3012 3005
Miami, USA + 1 305 704 8816

Quaisquer informações contidas neste e-mail e anexos podem ser
confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma
de utilização deste documento depende de autorização do emissor, sujeito as
penalidades cabíveis.

Any information in this e-mail and attachments may be confidential and
privileged, protected by legal confidentiality. The use of this document
require authorization by the issuer, subject to penalties.

Re: traffic server cache key

Posted by Leif Hedstrom <zw...@apache.org>.
On May 13, 2014, at 12:14 PM, Daniel Biazus <da...@azion.com> wrote:

> Hi Guys,
> 
>     We've been using traffic server as a reverse proxy to delivery static content across multiple CDN nodes, though the regex purge works as expected through web UI tool, We've been facing some challenges regarding the cache keys when the objects are stored in cache.
>     In order to have a better understanding of our scenario, let's take a look at this example:
> 
> Scenario 1:
> 
> - traffic server is working as a reverse proxy:
> 
> #remap.config
> 
> map http://finalurl.foo.bar/   http://originserver.foo.bar/
> reverse_map  http://originserver.foo.bar/  http://finalurl.foo.bar/
> 
> - We have to purge all css files under the URL http://finalurl.foo.bar/
> 
> Considering this, we have to purge the following regex:
> 
> - http://finalurl.foo.bar/*.css
>     
> The example above doesn't work due the fact that traffic server uses the origin URL to calculate the hash key, then, in this case we have to make a purge request using the origin server URL (that's not nice with our clients, right ?):
> 
> - http://originserver.foo.bar/*.css
> 
> So guys, anyone knows if there is a way to change the hash key in traffic server, or even if is there some workaround to deal with this scenario ?


The common way of doing this is to have a plugin, that generates cache keys based on a combination of the URL and a generation ID. This would easily be configurable per remap rule, such that you can then instantly (O(1)) purge all objects for that remap rule simply by modifying (increasing) the generation ID. There is one such plugin in the public domain, although it probably needs a little loving. :) An easy adoption to that plugin would be to take the generation ID as an argument on the remap rule.

— Leif


Re: traffic server cache key

Posted by Daniel Biazus <da...@azion.com>.
Hey Alan,

    Thanks for the hint! Although this plugin can be easily configured to
change the cache key, the objects are still stored into the cache
considering the origin URL (dont know why). My guess is, that behaviour is
a inheritance when traffic server was just a forward proxy and the hash
algorithm was hardcoded to use the origin URL as part of the cache key.
    If you guys have any another suggestions, please let us know! :-)

Thanks & Regards,
Daniel


On Wed, May 14, 2014 at 3:05 AM, Alan M. Carroll <
amc@network-geographics.com> wrote:

>  Daniel,
>
> There are plugin API functions to do that. You can look at
> plugins/cacheurl for an example that uses regexes to match URLs and modify
> the cache key.
>
>
> Tuesday, May 13, 2014, 11:14:49 AM, you wrote:
>
>
>
>
> So guys, anyone knows if there is a way to change the hash key in traffic
> server, or even if is there some workaround to deal with this scenario ?
>
>
>
>


-- 
Daniel Biazus

Porto Alegre, Brasil + 55 51 3012 3005
Miami, USA + 1 305 704 8816

Quaisquer informações contidas neste e-mail e anexos podem ser
confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma
de utilização deste documento depende de autorização do emissor, sujeito as
penalidades cabíveis.

Any information in this e-mail and attachments may be confidential and
privileged, protected by legal confidentiality. The use of this document
require authorization by the issuer, subject to penalties.