You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by Sebastian Annies <se...@castlabs.com> on 2013/08/03 21:16:02 UTC
Plugin transforming between cache and end-user
Hi,
we have huge files that will be slightly different for most users - they
are 1 to 8GB and only 1k differs - so we want to change them with a
transformation. Unfortunately the null-transform plugin is of no help since
it transforms the content coming from the origin server and NOT the content
that is going to the browser of the final user. Is there anywhere any
example on doing transforms at this place?
Basically it’s like this:
http://ats/huge_video_a?userspecific=abc
http://ats/huge_video_a?userspecific=xyz
and of course we only want to store the untransformed ‘huge_video_a’ once
and perform the transformation (inserting the user specific info) on every
delivery. Where to start? Any hints?
Best Regards,
Sebastian
AW: Plugin transforming between cache and end-user
Posted by Sebastian Annies <se...@castlabs.com>.
Hi Leif,
On the first range request it works in deed, the problem arises - at least
in my case - when the full file is cached by a previous request and the
range is served from the cache. I attached my heavily modified gzip (now
null-transform) plugin I used to test it.
Best Regards,
Sebastian
-----Ursprüngliche Nachricht-----
Von: Leif Hedstrom [mailto:zwoop@apache.org]
Gesendet: Freitag, 9. August 2013 12:34
An: dev@trafficserver.apache.org
Betreff: Re: Plugin transforming between cache and end-user
On Aug 8, 2013, at 12:56 PM, Sebastian Annies
<se...@castlabs.com> wrote:
> Hi,
>
> So far I'm happy with everything except HTTP range requests.
> Whenever the plugin is null-transforming a HTTP range request the
> following is happening:
>
> * the return code is set to 200 (instead of 206)
> * the content-range header is omitted
> * nevertheless the plugin only 'sees' the requested bytes
Hmmm, I tested this with 3.3.5, and I'm not able to reproduce. I'm running
with the standard null-transform that comes in the examples/ directory.
For example:
loki (13:32) 424/0 $ curl -D - -o /dev/null -s -H "Range: bytes=0-100" -x
localhost:80
http://www.ogre.com/~leif/ats/Screenshot%202013-07-11%20at%2012.46.21%20PM
.png
HTTP/1.1 206 Partial Content
Date: Fri, 09 Aug 2013 19:32:20 GMT
Server: ATS/3.3.5-dev
Last-Modified: Thu, 11 Jul 2013 18:52:25 GMT
ETag: "51ab-4e140e52805cd"
Accept-Ranges: bytes
Content-Length: 101
Content-Range: bytes 0-100/20907
This seems exactly as expected (and I get the same result without the
null-transform.so plugin).
More details on how to reproduce this ? Are you serving out of ATS's
cache, or is it fetching from origin ? (My test fetches from Origin,
because the default null-transform.so doesn't apply to cache hits).
-- Leif
Re: Plugin transforming between cache and end-user
Posted by Leif Hedstrom <zw...@apache.org>.
On Aug 8, 2013, at 12:56 PM, Sebastian Annies <se...@castlabs.com> wrote:
> Hi,
>
> So far I'm happy with everything except HTTP range requests.
> Whenever the plugin is null-transforming a HTTP range request the
> following is happening:
>
> * the return code is set to 200 (instead of 206)
> * the content-range header is omitted
> * nevertheless the plugin only 'sees' the requested bytes
Hmmm, I tested this with 3.3.5, and I'm not able to reproduce. I'm running with the standard null-transform that comes in the examples/ directory. For example:
loki (13:32) 424/0 $ curl -D - -o /dev/null -s -H "Range: bytes=0-100" -x localhost:80 http://www.ogre.com/~leif/ats/Screenshot%202013-07-11%20at%2012.46.21%20PM.png
HTTP/1.1 206 Partial Content
Date: Fri, 09 Aug 2013 19:32:20 GMT
Server: ATS/3.3.5-dev
Last-Modified: Thu, 11 Jul 2013 18:52:25 GMT
ETag: "51ab-4e140e52805cd"
Accept-Ranges: bytes
Content-Length: 101
Content-Range: bytes 0-100/20907
This seems exactly as expected (and I get the same result without the null-transform.so plugin).
More details on how to reproduce this ? Are you serving out of ATS's cache, or is it fetching from origin ? (My test fetches from Origin, because the default null-transform.so doesn't apply to cache hits).
-- Leif
Re: AW: AW: Plugin transforming between cache and end-user
Posted by "Alan M. Carroll" <am...@network-geographics.com>.
Realistically, I would estimate end of August as a reasonable time frame.
Thursday, August 8, 2013, 7:36:43 PM, you wrote:
> I'm glad that the problem is at least known.
> Do you have a gut-feeling when you will find the time? Is it rather this
> month, in the next 3 months, this year or rather in 2014?
AW: AW: Plugin transforming between cache and end-user
Posted by Sebastian Annies <se...@castlabs.com>.
I'm glad that the problem is at least known.
Do you have a gut-feeling when you will find the time? Is it rather this
month, in the next 3 months, this year or rather in 2014?
Best Regards,
Sebastian
-----Ursprüngliche Nachricht-----
Von: Alan M. Carroll [mailto:amc@network-geographics.com]
Gesendet: Donnerstag, 8. August 2013 14:50
An: Sebastian Annies
Betreff: Re: AW: Plugin transforming between cache and end-user
The range transforms worked at for 3.0 but were broken for 3.2. This was
masked by the addition of range acceleration but when that was determined
to be too risky the original range transform issue was re-exposed. I'm
currently trying to get some time to work on that, I see what the problem
is, a fix has proven somewhat more difficult to discover.
https://issues.apache.org/jira/browse/TS-1776
Thursday, August 8, 2013, 1:56:11 PM, you wrote:
> * the return code is set to 200 (instead of 206)
> * the content-range header is omitted
> * nevertheless the plugin only 'sees' the requested bytes
Re: AW: Plugin transforming between cache and end-user
Posted by "Alan M. Carroll" <am...@network-geographics.com>.
The range transforms worked at for 3.0 but were broken for 3.2. This was masked by the addition of range acceleration but when that was determined to be too risky the original range transform issue was re-exposed. I'm currently trying to get some time to work on that, I see what the problem is, a fix has proven somewhat more difficult to discover.
https://issues.apache.org/jira/browse/TS-1776
Thursday, August 8, 2013, 1:56:11 PM, you wrote:
> * the return code is set to 200 (instead of 206)
> * the content-range header is omitted
> * nevertheless the plugin only 'sees' the requested bytes
AW: Plugin transforming between cache and end-user
Posted by Sebastian Annies <se...@castlabs.com>.
Hi,
So far I'm happy with everything except HTTP range requests.
Whenever the plugin is null-transforming a HTTP range request the
following is happening:
* the return code is set to 200 (instead of 206)
* the content-range header is omitted
* nevertheless the plugin only 'sees' the requested bytes
I would need to make it work:
* return code 206 & content-range response need to be there
* the transformation must know the offset with the original response, it
doesn't seem to be available
Do transformation and range-requests already work or is it still a todo on
ATS side?
Grüße,
Sebastian
-----Ursprüngliche Nachricht-----
Von: James Peach [mailto:jpeach@apache.org]
Gesendet: Dienstag, 6. August 2013 16:15
An: dev@trafficserver.apache.org
Cc: briang@apache.org
Betreff: Re: Plugin transforming between cache and end-user
On Aug 6, 2013, at 4:01 PM, Sebastian Annies
<se...@castlabs.com> wrote:
> Hi,
>
> On IRC someone suggested (sorry I forgot your name!) to start looking
> at the gzip plugin as it stores the full data and perform the
> compression/transformation on the fly. I removed everything and
> modified so that I now have a null-transform that also works on cached
> content. I have a good starting point now. The key seems to be to add
> the transformation on TS_EVENT_HTTP_CACHE_LOOKUP_COMPLETE, too.
Nice!
>
> Best Regards,
> Sebastian
> -----Ursprüngliche Nachricht-----
> Von: James Peach [mailto:jpeach@apache.org]
> Gesendet: Dienstag, 6. August 2013 10:33
> An: dev@trafficserver.apache.org
> Cc: briang@apache.org
> Betreff: Re: Plugin transforming between cache and end-user
>
>
> On Aug 3, 2013, at 12:16 PM, Sebastian Annies
> <se...@castlabs.com> wrote:
>
>> Hi,
>>
>>
>>
>> we have huge files that will be slightly different for most users -
>> they are 1 to 8GB and only 1k differs - so we want to change them
>> with a transformation. Unfortunately the null-transform plugin is of
>> no help since it transforms the content coming from the origin server
>> and NOT the content that is going to the browser of the final user.
>> Is there anywhere any example on doing transforms at this place?
>>
>>
>>
>> Basically it’s like this:
>>
>>
>>
>> http://ats/huge_video_a?userspecific=abc
>>
>> http://ats/huge_video_a?userspecific=xyz
>>
>>
>>
>> and of course we only want to store the untransformed ‘huge_video_a’
>> once and perform the transformation (inserting the user specific
>> info) on every delivery. Where to start? Any hints?
>
> I'm not very familiar with transform, but AFAICS you get to do
> TS_HTTP_REQUEST_TRANSFORM_HOOK or TS_HTTP_RESPONSE_TRANSFORM_HOOK, and
> the cache always stores the transformed response. Maybe Brian (CC'ed)
> has some other ideas ...
>
> J
Re: Plugin transforming between cache and end-user
Posted by James Peach <jp...@apache.org>.
On Aug 6, 2013, at 4:01 PM, Sebastian Annies <se...@castlabs.com> wrote:
> Hi,
>
> On IRC someone suggested (sorry I forgot your name!) to start looking at
> the gzip plugin as it stores the full data and perform the
> compression/transformation on the fly. I removed everything and modified
> so that I now have a null-transform that also works on cached content. I
> have a good starting point now. The key seems to be to add the
> transformation on TS_EVENT_HTTP_CACHE_LOOKUP_COMPLETE, too.
Nice!
>
> Best Regards,
> Sebastian
> -----Ursprüngliche Nachricht-----
> Von: James Peach [mailto:jpeach@apache.org]
> Gesendet: Dienstag, 6. August 2013 10:33
> An: dev@trafficserver.apache.org
> Cc: briang@apache.org
> Betreff: Re: Plugin transforming between cache and end-user
>
>
> On Aug 3, 2013, at 12:16 PM, Sebastian Annies
> <se...@castlabs.com> wrote:
>
>> Hi,
>>
>>
>>
>> we have huge files that will be slightly different for most users -
>> they are 1 to 8GB and only 1k differs - so we want to change them with
>> a transformation. Unfortunately the null-transform plugin is of no
>> help since it transforms the content coming from the origin server and
>> NOT the content that is going to the browser of the final user. Is
>> there anywhere any example on doing transforms at this place?
>>
>>
>>
>> Basically it’s like this:
>>
>>
>>
>> http://ats/huge_video_a?userspecific=abc
>>
>> http://ats/huge_video_a?userspecific=xyz
>>
>>
>>
>> and of course we only want to store the untransformed ‘huge_video_a’
>> once and perform the transformation (inserting the user specific info)
>> on every delivery. Where to start? Any hints?
>
> I'm not very familiar with transform, but AFAICS you get to do
> TS_HTTP_REQUEST_TRANSFORM_HOOK or TS_HTTP_RESPONSE_TRANSFORM_HOOK, and the
> cache always stores the transformed response. Maybe Brian (CC'ed) has some
> other ideas ...
>
> J
AW: Plugin transforming between cache and end-user
Posted by Sebastian Annies <se...@castlabs.com>.
Hi,
On IRC someone suggested (sorry I forgot your name!) to start looking at
the gzip plugin as it stores the full data and perform the
compression/transformation on the fly. I removed everything and modified
so that I now have a null-transform that also works on cached content. I
have a good starting point now. The key seems to be to add the
transformation on TS_EVENT_HTTP_CACHE_LOOKUP_COMPLETE, too.
Best Regards,
Sebastian
-----Ursprüngliche Nachricht-----
Von: James Peach [mailto:jpeach@apache.org]
Gesendet: Dienstag, 6. August 2013 10:33
An: dev@trafficserver.apache.org
Cc: briang@apache.org
Betreff: Re: Plugin transforming between cache and end-user
On Aug 3, 2013, at 12:16 PM, Sebastian Annies
<se...@castlabs.com> wrote:
> Hi,
>
>
>
> we have huge files that will be slightly different for most users -
> they are 1 to 8GB and only 1k differs - so we want to change them with
> a transformation. Unfortunately the null-transform plugin is of no
> help since it transforms the content coming from the origin server and
> NOT the content that is going to the browser of the final user. Is
> there anywhere any example on doing transforms at this place?
>
>
>
> Basically it’s like this:
>
>
>
> http://ats/huge_video_a?userspecific=abc
>
> http://ats/huge_video_a?userspecific=xyz
>
>
>
> and of course we only want to store the untransformed ‘huge_video_a’
> once and perform the transformation (inserting the user specific info)
> on every delivery. Where to start? Any hints?
I'm not very familiar with transform, but AFAICS you get to do
TS_HTTP_REQUEST_TRANSFORM_HOOK or TS_HTTP_RESPONSE_TRANSFORM_HOOK, and the
cache always stores the transformed response. Maybe Brian (CC'ed) has some
other ideas ...
J
Re: Plugin transforming between cache and end-user
Posted by James Peach <jp...@apache.org>.
On Aug 3, 2013, at 12:16 PM, Sebastian Annies <se...@castlabs.com> wrote:
> Hi,
>
>
>
> we have huge files that will be slightly different for most users - they
> are 1 to 8GB and only 1k differs - so we want to change them with a
> transformation. Unfortunately the null-transform plugin is of no help since
> it transforms the content coming from the origin server and NOT the content
> that is going to the browser of the final user. Is there anywhere any
> example on doing transforms at this place?
>
>
>
> Basically it’s like this:
>
>
>
> http://ats/huge_video_a?userspecific=abc
>
> http://ats/huge_video_a?userspecific=xyz
>
>
>
> and of course we only want to store the untransformed ‘huge_video_a’ once
> and perform the transformation (inserting the user specific info) on every
> delivery. Where to start? Any hints?
I'm not very familiar with transform, but AFAICS you get to do TS_HTTP_REQUEST_TRANSFORM_HOOK or TS_HTTP_RESPONSE_TRANSFORM_HOOK, and the cache always stores the transformed response. Maybe Brian (CC'ed) has some other ideas ...
J