You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Conan <co...@gmail.com> on 2011/03/21 18:13:00 UTC
how to cache object with different urls (with question mark) just once?
hi users,
I have a object to cache. This object's url is like "
http://example.com/a.jpg?id=xxx". The parameters after question mark(?) is
different and dynamic, but the object is identical so it's better to cache
only one copy of the object.
What I except is: ATS ignore the parameters in url and serve the object (if
cached), or pass the complete url to original server if the object is not
cached.
When I enable "proxy.config.http.cache_urls_that_look_dynamic", it seemed
ATS treated these urls as different objects. If I disable it, ATS did not
cache any urls with question mark.
Any suggestions? Or what kind of plugin could I write to support that?
Thanks in advance.
Re: how to cache object with different urls (with question mark)
just once?
Posted by Leif Hedstrom <zw...@apache.org>.
On 03/23/2011 09:46 AM, Leif Hedstrom wrote:
> On 03/23/2011 09:06 AM, Mark Harrison wrote:
>> I have the plugin working with 2.1.7 and it is currently available at
>> http://ats-cache-proxy.googlecode.com/svn/trunk/trafficserver_cacheurl/
>> .
>>
>> It builds with tsxs (tsxs -c cacheurl.c -o cacheurl.so).
>>
>> To enable it, add cacheurl.so to your plugin.config file. You will
>> also need to create a cacheurl.config file which goes in the plugin
>> directory (libexec usually) that contains patterns/replacements - one
>> per line. Comments are allowed using '#'.
>>
>> An example config file is available here:
>> http://ats-cache-proxy.googlecode.com/svn/trunk/install/cacheurl.config
>
> Interesting. What is the ATSINTERNAL string in that example config?
> Is that just something you add to add some entropy to the cache key?
> Also, do you really intend to keep the "omnit" references in the
> plugin initialisation? :)
Oh, never mind on the OmniTi reference. But, curious, why is the code
copyright Google?
-- Leif
Re: how to cache object with different urls (with question mark) just once?
Posted by Mark Harrison <ma...@omniti.com>.
> Interesting. What is the ATSINTERNAL string in that example config? Is that
> just something you add to add some entropy to the cache key?
It's just something to make it clear that it's a rewritten cache key
and to guarantee it won't clash with a real url by mistake. Those
examples were taken from a similar feature in squid where they had
SQUIDINTERNAL as part of the cache key. It's not really necessary.
Re: how to cache object with different urls (with question mark)
just once?
Posted by Leif Hedstrom <zw...@apache.org>.
On 03/23/2011 09:06 AM, Mark Harrison wrote:
> I have the plugin working with 2.1.7 and it is currently available at
> http://ats-cache-proxy.googlecode.com/svn/trunk/trafficserver_cacheurl/
> .
>
> It builds with tsxs (tsxs -c cacheurl.c -o cacheurl.so).
>
> To enable it, add cacheurl.so to your plugin.config file. You will
> also need to create a cacheurl.config file which goes in the plugin
> directory (libexec usually) that contains patterns/replacements - one
> per line. Comments are allowed using '#'.
>
> An example config file is available here:
> http://ats-cache-proxy.googlecode.com/svn/trunk/install/cacheurl.config
Interesting. What is the ATSINTERNAL string in that example config? Is
that just something you add to add some entropy to the cache key? Also,
do you really intend to keep the "omnit" references in the plugin
initialisation? :)
Cheers,
-- Leif
Re: how to cache object with different urls (with question mark) just once?
Posted by Conan <co...@gmail.com>.
This plugin works for me.
I have some questions. After modifying the cacheurl.config file, I must
restart traffic_server or use "traffic_line -b".
1. Will "traffic_line -b" block the requests during "Bounce local
traffic_server"?
2. How can I improve the plugin if I want the plugin to reread the config
files when "traffic_line -x"?
2011/3/24 Mark Harrison <ma...@omniti.com>
> > This meas that evey modified url you want to cached you have to write
> into
> > this file?
>
> > For example, i have a url like http://example.com/a.jpg?id=xxx, if i
> want
> > cache this object, does mean i have to wirte patterns/replacements
> > in cacheurl.config?
>
> Yes, the file specified a list of URLs and what you want the cache key
> to be. In your case if you just wanted to strip off the query string
> you could probably do something like:
>
> http://example.com/([^?]*)\? http://example.com/$1
>
> This would replace http://example.com/foo.jpg?some=1&query=2&string=3
> with simply http://example.com/foo.jpg
>
> (I've not tested this and it might need a little tweaking)
>
> > it's possible use your plugin to catche some unforeseeable modified url
> ?
>
> I'm not sure what you mean by this. You need to specify (via patterns)
> which URLs you wish to consider the same. You can however use regular
> expressions to match a group of URLs as in the example above where
> anything in example.com is matched, and it will ignore the query
> string for the purposes of matching.
>
> The other use case for this plugin is for when you have multiple
> mirrors of the same content - e.g. http://server1.example.com/foo.jpg,
> http://server2.example.com/foo.jpg. You can make a pattern such as:
>
> http://server[12].example.com/(.*)
> http://server.example.com.ATSINTERNAL/$1
>
> (The ATSINTERNAL here is simply something so that if
> http://server.example.com/foo.jpg exists, but has different content
> from http://server1.example.com/foo.jpg, then ATS doesn't consider
> them to be the same. Any unique string could be used here).
>
> Mark
>
Re: how to cache object with different urls (with question mark) just once?
Posted by Mark Harrison <ma...@omniti.com>.
> This meas that evey modified url you want to cached you have to write into
> this file?
> For example, i have a url like http://example.com/a.jpg?id=xxx, if i want
> cache this object, does mean i have to wirte patterns/replacements
> in cacheurl.config?
Yes, the file specified a list of URLs and what you want the cache key
to be. In your case if you just wanted to strip off the query string
you could probably do something like:
http://example.com/([^?]*)\? http://example.com/$1
This would replace http://example.com/foo.jpg?some=1&query=2&string=3
with simply http://example.com/foo.jpg
(I've not tested this and it might need a little tweaking)
> it's possible use your plugin to catche some unforeseeable modified url ?
I'm not sure what you mean by this. You need to specify (via patterns)
which URLs you wish to consider the same. You can however use regular
expressions to match a group of URLs as in the example above where
anything in example.com is matched, and it will ignore the query
string for the purposes of matching.
The other use case for this plugin is for when you have multiple
mirrors of the same content - e.g. http://server1.example.com/foo.jpg,
http://server2.example.com/foo.jpg. You can make a pattern such as:
http://server[12].example.com/(.*) http://server.example.com.ATSINTERNAL/$1
(The ATSINTERNAL here is simply something so that if
http://server.example.com/foo.jpg exists, but has different content
from http://server1.example.com/foo.jpg, then ATS doesn't consider
them to be the same. Any unique string could be used here).
Mark
RE: how to cache object with different urls (with question mark)
just once?
Posted by ChangCheng <cc...@msn.com>.
Hi, MarkThanks your sharing, I have some queries about cacheurl.config, Could you explan moare about what supoort to do for this file?This meas that evey modified url you want to cached you have to write into this file?For example, i have a url like http://example.com/a.jpg?id=xxx, if i want cache this object, does mean i have to wirte patterns/replacements in cacheurl.config?it's possible use your plugin to catche some unforeseeable modified url ?Thanks
> From: mark@omniti.com
> Date: Wed, 23 Mar 2011 11:06:28 -0400
> Subject: Re: how to cache object with different urls (with question mark) just once?
> To: users@trafficserver.apache.org
>
> I have the plugin working with 2.1.7 and it is currently available at
> http://ats-cache-proxy.googlecode.com/svn/trunk/trafficserver_cacheurl/
> .
>
> It builds with tsxs (tsxs -c cacheurl.c -o cacheurl.so).
>
> To enable it, add cacheurl.so to your plugin.config file. You will
> also need to create a cacheurl.config file which goes in the plugin
> directory (libexec usually) that contains patterns/replacements - one
> per line. Comments are allowed using '#'.
>
> An example config file is available here:
> http://ats-cache-proxy.googlecode.com/svn/trunk/install/cacheurl.config
>
> Let me know if there are any issues getting things working,
>
> Mark
Re: how to cache object with different urls (with question mark) just once?
Posted by Mark Harrison <ma...@omniti.com>.
I have the plugin working with 2.1.7 and it is currently available at
http://ats-cache-proxy.googlecode.com/svn/trunk/trafficserver_cacheurl/
.
It builds with tsxs (tsxs -c cacheurl.c -o cacheurl.so).
To enable it, add cacheurl.so to your plugin.config file. You will
also need to create a cacheurl.config file which goes in the plugin
directory (libexec usually) that contains patterns/replacements - one
per line. Comments are allowed using '#'.
An example config file is available here:
http://ats-cache-proxy.googlecode.com/svn/trunk/install/cacheurl.config
Let me know if there are any issues getting things working,
Mark
Re: how to cache object with different urls (with question mark)
just once?
Posted by Leif Hedstrom <zw...@apache.org>.
On 03/21/2011 02:29 PM, Mark Harrison wrote:
> The plugin was complete and working for 2.1.4, but there have been a
> number of API changes since then. I'll work on getting it built
> against 2.1.6 and mail a link to the code here.
Go for v2.1.7.
-- leif
Re: how to cache object with different urls (with question mark) just once?
Posted by Igor Galić <i....@brainsware.org>.
----- Original Message -----
Hi Mark,
> The plugin was complete and working for 2.1.4, but there have been a
> number of API changes since then. I'll work on getting it built
> against 2.1.6 and mail a link to the code here.
My suggestion is to always use either the latest release or trunk until
we've released a GA (3.0). After that point we guarantee a stable API.
The changes so far:
http://s.apache.org/ts-2.1.5
http://s.apache.org/ts-2.1.6
http://s.apache.org/ts-2.1.7
http://s.apache.org/ts-2.1.8
A summary can also be found here:
http://trafficserver.apache.org/downloads.html
The biggest change is: s/INK/TS/ (with some exceptions) -- for the core
API, and restructuring of the core API to be internally consistent:
https://cwiki.apache.org/confluence/display/TS/APICleanup
i
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
Re: how to cache object with different urls (with question mark) just once?
Posted by Mark Harrison <ma...@omniti.com>.
The plugin was complete and working for 2.1.4, but there have been a
number of API changes since then. I'll work on getting it built
against 2.1.6 and mail a link to the code here.
Currently the plugin takes a list of regex/replacement pairs which is
apples to all URLs, and changes the URL that they're cached under.
Hopefully that should cover most people's use cases. A couple of
examples:
http://example.com/(.+)\?.* http://example.com/$1
http://server[1-4].example.com/(.*)?id=([0-9]+) http://test.ATSINTERNAL/$1_$2
this works with any part of the URL, including the query string, and
was inspired by the store_url feature in a past version of squid.
2011/3/21 ChangCheng <cc...@msn.com>:
> Hi, Mark
> I am working on this plugin as well. but i still haven't find better way to
> solve this problem. could you give some suggestion for this problem if
> possible?
> Thanks lots
>
>> From: mark@omniti.com
>> Date: Mon, 21 Mar 2011 13:52:35 -0400
>> Subject: Re: how to cache object with different urls (with question mark)
>> just once?
>> To: users@trafficserver.apache.org
>>
>> I've been working on a plugin that does just that, but it hasn't yet
>> been updated to work with anything past 2.1.4 (It's on my todo list).
>> However, I seem to remember somebody else also working on this also
>> which may be further along. Let me know if you'd be interested in the
>> code for the plugin.
>>
>> Regards,
>>
>> Mark
>>
>> On Mon, Mar 21, 2011 at 1:46 PM, Leif Hedstrom <zw...@apache.org> wrote:
>> > On 03/21/2011 11:13 AM, Conan wrote:
>> >
>> > hi users,
> &g t; >
>> >
>> > I have a object to cache. This object's url is like
>> > "http://example.com/a.jpg?id=xxx". The parameters after question mark(?)
>> > is
>> > different and dynamic, but the object is identical so it's better to
>> > cache
>> > only one copy of the object.
>> > What I except is: ATS ignore the parameters in url and serve the object
>> > (if
>> > cached), or pass the complete url to original server if the object is
>> > not
>> > cached.
>> >
>> > When I enable "proxy.config.http.cache_urls_that_look_dynamic", it
>> > seemed
>> > ATS treated these urls as different objects. If I disable it, ATS did
>> > not
>> > cache any urls with question mark.
>> > Any suggestions? Or what kind of plugin could I write to support that?
>> >
>> > You'd have to write a plugin that calculates a different cache key,
>> > based on
>> > your "normalization" rules. In your case, presumably only the host and
>> > the
>> > path.
>> >
>> > -- Leif
>> >
>> >
>
RE: how to cache object with different urls (with question mark)
just once?
Posted by ChangCheng <cc...@msn.com>.
Hi, MarkI am working on this plugin as well. but i still haven't find better way to solve this problem. could you give some suggestion for this problem if possible?Thanks lots
> From: mark@omniti.com
> Date: Mon, 21 Mar 2011 13:52:35 -0400
> Subject: Re: how to cache object with different urls (with question mark) just once?
> To: users@trafficserver.apache.org
>
> I've been working on a plugin that does just that, but it hasn't yet
> been updated to work with anything past 2.1.4 (It's on my todo list).
> However, I seem to remember somebody else also working on this also
> which may be further along. Let me know if you'd be interested in the
> code for the plugin.
>
> Regards,
>
> Mark
>
> On Mon, Mar 21, 2011 at 1:46 PM, Leif Hedstrom <zw...@apache.org> wrote:
> > On 03/21/2011 11:13 AM, Conan wrote:
> >
> > hi users,
> >
> >
> > I have a object to cache. This object's url is like
> > "http://example.com/a.jpg?id=xxx". The parameters after question mark(?) is
> > different and dynamic, but the object is identical so it's better to cache
> > only one copy of the object.
> > What I except is: ATS ignore the parameters in url and serve the object (if
> > cached), or pass the complete url to original server if the object is not
> > cached.
> >
> > When I enable "proxy.config.http.cache_urls_that_look_dynamic", it seemed
> > ATS treated these urls as different objects. If I disable it, ATS did not
> > cache any urls with question mark.
> > Any suggestions? Or what kind of plugin could I write to support that?
> >
> > You'd have to write a plugin that calculates a different cache key, based on
> > your "normalization" rules. In your case, presumably only the host and the
> > path.
> >
> > -- Leif
> >
> >
Re: how to cache object with different urls (with question mark) just once?
Posted by Mark Harrison <ma...@omniti.com>.
I've been working on a plugin that does just that, but it hasn't yet
been updated to work with anything past 2.1.4 (It's on my todo list).
However, I seem to remember somebody else also working on this also
which may be further along. Let me know if you'd be interested in the
code for the plugin.
Regards,
Mark
On Mon, Mar 21, 2011 at 1:46 PM, Leif Hedstrom <zw...@apache.org> wrote:
> On 03/21/2011 11:13 AM, Conan wrote:
>
> hi users,
>
>
> I have a object to cache. This object's url is like
> "http://example.com/a.jpg?id=xxx". The parameters after question mark(?) is
> different and dynamic, but the object is identical so it's better to cache
> only one copy of the object.
> What I except is: ATS ignore the parameters in url and serve the object (if
> cached), or pass the complete url to original server if the object is not
> cached.
>
> When I enable "proxy.config.http.cache_urls_that_look_dynamic", it seemed
> ATS treated these urls as different objects. If I disable it, ATS did not
> cache any urls with question mark.
> Any suggestions? Or what kind of plugin could I write to support that?
>
> You'd have to write a plugin that calculates a different cache key, based on
> your "normalization" rules. In your case, presumably only the host and the
> path.
>
> -- Leif
>
>
Re: how to cache object with different urls (with question mark)
just once?
Posted by Leif Hedstrom <zw...@apache.org>.
On 03/21/2011 11:13 AM, Conan wrote:
> hi users,
>
> I have a object to cache. This object's url is like
> "http://example.com/a.jpg?id=xxx". The parameters after question
> mark(?) is different and dynamic, but the object is identical so it's
> better to cache only one copy of the object.
>
> What I except is: ATS ignore the parameters in url and serve the
> object (if cached), or pass the complete url to original server if the
> object is not cached.
>
>
> When I enable "proxy.config.http.cache_urls_that_look_dynamic", it
> seemed ATS treated these urls as different objects. If I disable it,
> ATS did not cache any urls with question mark.
>
> Any suggestions? Or what kind of plugin could I write to support that?
>
You'd have to write a plugin that calculates a different cache key,
based on your "normalization" rules. In your case, presumably only the
host and the path.
-- Leif