You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Daniel Morilha <dm...@gmail.com> on 2013/11/25 19:59:56 UTC
gzip
Hi,
I am writing a transformation plug-in and I bumped up into the problem of
deflating the content after the transformation kicks in.
I believe that's a common problem that most of the transformation plug-ins
may suffer with.
I am aware ESI implements gzip itself and after giving a quick look into
the GZIP plug-in and realizing it will be tough to integrate into another
transformation plug-in I am inclined to do the same.
Couldn't we have the gzip capabilities baked into the C API. The Linked'in
atscppapi provides a nice way where you can inflate before the transformers
and defalte after all of them had passed. Only problem is the dependency it
creates on their APIs.
thanks,
--
Daniel Morilha (dmorilha@gmail.com)
Re: gzip
Posted by Daniel Morilha <dm...@gmail.com>.
To follow up on this old thread I've started.
Once I had no success in integrating the available gzip plug-in into my
scenario I came up with different implementation:
- It was based on gzip/gunzip provided by ESI plug-in.
- The main plug-in was forked from null-transform and uses the same
model.
- I couldn't resist to make some parts more C++ friendly (vectors
instead of strings, less mallocs, etc...).
In fact I have two plug-ins, one to firstly inflate the incoming response
content and the other to deflate (after the transformation pipeline has
been applied), both plug-ins supporting streaming.
Differently from the gzip plug-in, they don't prevent the origin from
deflating the original content (by hiding the accept-encoding request
field). Instead I believe this decision is up to the origin. We may have
cases the proxy is far from the origin where applying gzip between both
could be beneficial. Therefore it is up to the origin to honor the field or
not.
It is worth to mention I am using the plug-ins in a very untraditional way.
I am dynamically opening them from my transformation plug-in and including
them into the transformation pipeline only when the content will in fact be
subjected to additional transformations (a run-time decision). They also
work when included on plugins.conf file. Care was taken to apply the
plug-in only once per request (similarly to what has been done on the
original gzip plug-in).
A subset of this approach (only suboptimal gzip) can already be seen when
accessing any page under https://search.yahoo.com/ The described
implementation should be available on the same domain soon. More work needs
to be done to make the plug-in broader (support deflate, etc) more
debuggable and configurable.
This will be the model I will follow for next transformation plug-ins I may
write. I would be glad to open source the work so others could benefit from
it if there is interest.
thanks,
On Tue, Nov 26, 2013 at 9:25 AM, Daniel Morilha <dm...@gmail.com> wrote:
> I think I got it, I tried to use the gzip plug-in with a simple
> configuration as exemplified on the docs but I wasn't able to stream the
> content any longer. It looks like the plug-in buffers everything so it can
> compress and flush at once. Is there a way to make the gzip plug-in to
> compress and flush the chunks? That's a requirement for my use case.
>
> Thanks guys!
>
>
> On Tue, Nov 26, 2013 at 6:16 AM, Igor Galić <i....@brainsware.org>wrote:
>
>>
>>
>> ------------------------------
>>
>> > Also, I lied about the headers.. but, really, this should be a feature.
>>
>> That was what I was hinting at as well, and I think it should work in its
>> current form. Other plugins should be able to control compression through
>> the accept encoding header
>>
>> > See, ugh.. Otto, this https://trafficserver.readthedocs.org/en/latest/>reference/plugins/gzip.en.html
>> needs some polishing!
>>
>> I see :-) I'll fix that.
>>
>>
>> Thanks!
>>
>> --
>> Igor Galić
>>
>> Tel: +43 (0) 664 886 22 883
>> Mail: i.galic@brainsware.org
>> URL: http://brainsware.org/
>> GPG: 8716 7A9F 989B ABD5 100F 4008 F266 55D6 2998 1641
>>
>>
>
>
> --
> Daniel Morilha (dmorilha@gmail.com)
>
--
Daniel Morilha (dmorilha@gmail.com)
Re: gzip
Posted by Daniel Morilha <dm...@gmail.com>.
I think I got it, I tried to use the gzip plug-in with a simple
configuration as exemplified on the docs but I wasn't able to stream the
content any longer. It looks like the plug-in buffers everything so it can
compress and flush at once. Is there a way to make the gzip plug-in to
compress and flush the chunks? That's a requirement for my use case.
Thanks guys!
On Tue, Nov 26, 2013 at 6:16 AM, Igor Galić <i....@brainsware.org> wrote:
>
>
> ------------------------------
>
> > Also, I lied about the headers.. but, really, this should be a feature.
>
> That was what I was hinting at as well, and I think it should work in its
> current form. Other plugins should be able to control compression through
> the accept encoding header
>
> > See, ugh.. Otto, this https://trafficserver.readthedocs.org/en/latest/>reference/plugins/gzip.en.html
> needs some polishing!
>
> I see :-) I'll fix that.
>
>
> Thanks!
>
> --
> Igor Galić
>
> Tel: +43 (0) 664 886 22 883
> Mail: i.galic@brainsware.org
> URL: http://brainsware.org/
> GPG: 8716 7A9F 989B ABD5 100F 4008 F266 55D6 2998 1641
>
>
--
Daniel Morilha (dmorilha@gmail.com)
Re: gzip
Posted by Igor Galić <i....@brainsware.org>.
----- Original Message -----
> > Also, I lied about the headers.. but, really, this should be a feature.
> That was what I was hinting at as well, and I think it should work in its
> current form. Other plugins should be able to control compression through
> the accept encoding header
> > See, ugh.. Otto, this https://trafficserver.readthedocs.org/en/latest/
> > >reference/plugins/gzip.en.html needs some polishing!
> I see :-) I'll fix that.
Thanks!
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 8716 7A9F 989B ABD5 100F 4008 F266 55D6 2998 1641
Re: gzip
Posted by Otto van der Schaaf <os...@gmail.com>.
> Also, I lied about the headers.. but, really, this should be a feature.
That was what I was hinting at as well, and I think it should work in its
current form. Other plugins should be able to control compression through
the accept encoding header
> See, ugh.. Otto, this https://trafficserver.readthedocs.org/en/latest/>reference/plugins/gzip.en.html
needs some polishing!
I see :-) I'll fix that.
2013/11/25 Igor Galić <i....@brainsware.org>
> I think what you're looking for is to trigger the gzip plugin through
> headers.
> Currently that happens automatically if you just enable it, and there are
> Accept-Encoding: gzip headers.
>
> Your transformation chain could set a trigger, that the gzip plugin could
> react to.
>
> See, ugh.. Otto, this
> https://trafficserver.readthedocs.org/en/latest/reference/plugins/gzip.en.htmlneeds some polishing!
>
> Also, I lied about the headers.. but, really, this should be a feature.
>
> So long,
>
> i
>
> ------------------------------
>
> so then might question would be is it possible to generate the gzip plugin
> configuration into another transformation plug-in or this is only based on
> the configuration file provided on plugins.conf ?
>
> thanks,
>
>
> On Mon, Nov 25, 2013 at 11:25 AM, Otto van der Schaaf <os...@gmail.com>wrote:
>
>> The gzip plugin will run when configured to do so, plus when the
>> accept-encoding request header meets the criteria (e.g. 'gzip' is specified)
>>
>> Regrettably, it doesn't support inflation at this moment.
>> I think that should go into a separate plugin, so that can be configured
>> into its own place into the plugin chain (probably before any
>> transformations).
>>
>> Otto
>>
>>
>> 2013/11/25 Daniel Morilha <dm...@gmail.com>
>>
>>> also AFAICT gzip plug-in has no inflate code.
>>>
>>>
>>> On Mon, Nov 25, 2013 at 11:19 AM, Daniel Morilha <dm...@gmail.com>wrote:
>>>
>>>> the transformation runs for some of the requests and I didn't see how
>>>> to tell the gzip to run or not... I don't want to unecessarily
>>>> inflate/deflate. Does it make sense?
>>>>
>>>> I was inclined into call the gzip plug-in internal APIs myself based on
>>>> those conditions but that would mean I would create the gzip headers myself
>>>> and also make sure the plug-in gets loaded correctly, which sounds hackish.
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> On Mon, Nov 25, 2013 at 11:12 AM, Otto van der Schaaf <
>>>> oschaaf@gmail.com> wrote:
>>>>
>>>>> Wouldn't it be a solution to configure the gzip plugin to run after
>>>>> your own transformation?
>>>>>
>>>>> Otto van der Schaaf
>>>>>
>>>>>
>>>>> 2013/11/25 Daniel Morilha <dm...@gmail.com>
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am writing a transformation plug-in and I bumped up into the
>>>>>> problem of deflating the content after the transformation kicks in.
>>>>>>
>>>>>> I believe that's a common problem that most of the transformation
>>>>>> plug-ins may suffer with.
>>>>>>
>>>>>> I am aware ESI implements gzip itself and after giving a quick look
>>>>>> into the GZIP plug-in and realizing it will be tough to integrate into
>>>>>> another transformation plug-in I am inclined to do the same.
>>>>>>
>>>>>> Couldn't we have the gzip capabilities baked into the C API. The
>>>>>> Linked'in atscppapi provides a nice way where you can inflate before the
>>>>>> transformers and defalte after all of them had passed. Only problem is the
>>>>>> dependency it creates on their APIs.
>>>>>>
>>>>>> thanks,
>>>>>> --
>>>>>> Daniel Morilha (dmorilha@gmail.com)
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Daniel Morilha (dmorilha@gmail.com)
>>>>
>>>
>>>
>>>
>>> --
>>> Daniel Morilha (dmorilha@gmail.com)
>>>
>>
>>
>
>
> --
> Daniel Morilha (dmorilha@gmail.com)
>
>
>
>
> --
> Igor Galić
>
> Tel: +43 (0) 664 886 22 883
> Mail: i.galic@brainsware.org
> URL: http://brainsware.org/
> GPG: 8716 7A9F 989B ABD5 100F 4008 F266 55D6 2998 1641
>
>
Re: gzip
Posted by Igor Galić <i....@brainsware.org>.
I think what you're looking for is to trigger the gzip plugin through headers.
Currently that happens automatically if you just enable it, and there are Accept-Encoding: gzip headers.
Your transformation chain could set a trigger, that the gzip plugin could react to.
See, ugh.. Otto, this https://trafficserver.readthedocs.org/en/latest/reference/plugins/gzip.en.html needs some polishing!
Also, I lied about the headers.. but, really, this should be a feature.
So long,
i
----- Original Message -----
> so then might question would be is it possible to generate the gzip plugin
> configuration into another transformation plug-in or this is only based on
> the configuration file provided on plugins.conf ?
> thanks,
> On Mon, Nov 25, 2013 at 11:25 AM, Otto van der Schaaf < oschaaf@gmail.com >
> wrote:
> > The gzip plugin will run when configured to do so, plus when the
> > accept-encoding request header meets the criteria (e.g. 'gzip' is
> > specified)
>
> > Regrettably, it doesn't support inflation at this moment.
>
> > I think that should go into a separate plugin, so that can be configured
> > into
> > its own place into the plugin chain (probably before any transformations).
>
> > Otto
>
> > 2013/11/25 Daniel Morilha < dmorilha@gmail.com >
>
> > > also AFAICT gzip plug-in has no inflate code.
> >
>
> > > On Mon, Nov 25, 2013 at 11:19 AM, Daniel Morilha < dmorilha@gmail.com >
> > > wrote:
> >
>
> > > > the transformation runs for some of the requests and I didn't see how
> > > > to
> > > > tell
> > > > the gzip to run or not... I don't want to unecessarily inflate/deflate.
> > > > Does
> > > > it make sense?
> > >
> >
>
> > > > I was inclined into call the gzip plug-in internal APIs myself based on
> > > > those
> > > > conditions but that would mean I would create the gzip headers myself
> > > > and
> > > > also make sure the plug-in gets loaded correctly, which sounds hackish.
> > >
> >
>
> > > > Thanks,
> > >
> >
>
> > > > On Mon, Nov 25, 2013 at 11:12 AM, Otto van der Schaaf <
> > > > oschaaf@gmail.com
> > > > >
> > > > wrote:
> > >
> >
>
> > > > > Wouldn't it be a solution to configure the gzip plugin to run after
> > > > > your
> > > > > own
> > > > > transformation?
> > > >
> > >
> >
>
> > > > > Otto van der Schaaf
> > > >
> > >
> >
>
> > > > > 2013/11/25 Daniel Morilha < dmorilha@gmail.com >
> > > >
> > >
> >
>
> > > > > > Hi,
> > > > >
> > > >
> > >
> >
>
> > > > > > I am writing a transformation plug-in and I bumped up into the
> > > > > > problem
> > > > > > of
> > > > > > deflating the content after the transformation kicks in.
> > > > >
> > > >
> > >
> >
>
> > > > > > I believe that's a common problem that most of the transformation
> > > > > > plug-ins
> > > > > > may suffer with.
> > > > >
> > > >
> > >
> >
>
> > > > > > I am aware ESI implements gzip itself and after giving a quick look
> > > > > > into
> > > > > > the
> > > > > > GZIP plug-in and realizing it will be tough to integrate into
> > > > > > another
> > > > > > transformation plug-in I am inclined to do the same.
> > > > >
> > > >
> > >
> >
>
> > > > > > Couldn't we have the gzip capabilities baked into the C API. The
> > > > > > Linked'in
> > > > > > atscppapi provides a nice way where you can inflate before the
> > > > > > transformers
> > > > > > and defalte after all of them had passed. Only problem is the
> > > > > > dependency
> > > > > > it
> > > > > > creates on their APIs.
> > > > >
> > > >
> > >
> >
>
> > > > > > thanks,
> > > > >
> > > >
> > >
> >
>
> > > > > > --
> > > > >
> > > >
> > >
> >
>
> > > > > > Daniel Morilha ( dmorilha@gmail.com )
> > > > >
> > > >
> > >
> >
>
> > > > --
> > >
> >
>
> > > > Daniel Morilha ( dmorilha@gmail.com )
> > >
> >
>
> > > --
> >
>
> > > Daniel Morilha ( dmorilha@gmail.com )
> >
>
> --
> Daniel Morilha ( dmorilha@gmail.com )
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 8716 7A9F 989B ABD5 100F 4008 F266 55D6 2998 1641
Re: gzip
Posted by Daniel Morilha <dm...@gmail.com>.
so then might question would be is it possible to generate the gzip plugin
configuration into another transformation plug-in or this is only based on
the configuration file provided on plugins.conf ?
thanks,
On Mon, Nov 25, 2013 at 11:25 AM, Otto van der Schaaf <os...@gmail.com>wrote:
> The gzip plugin will run when configured to do so, plus when the
> accept-encoding request header meets the criteria (e.g. 'gzip' is specified)
>
> Regrettably, it doesn't support inflation at this moment.
> I think that should go into a separate plugin, so that can be configured
> into its own place into the plugin chain (probably before any
> transformations).
>
> Otto
>
>
> 2013/11/25 Daniel Morilha <dm...@gmail.com>
>
>> also AFAICT gzip plug-in has no inflate code.
>>
>>
>> On Mon, Nov 25, 2013 at 11:19 AM, Daniel Morilha <dm...@gmail.com>wrote:
>>
>>> the transformation runs for some of the requests and I didn't see how to
>>> tell the gzip to run or not... I don't want to unecessarily
>>> inflate/deflate. Does it make sense?
>>>
>>> I was inclined into call the gzip plug-in internal APIs myself based on
>>> those conditions but that would mean I would create the gzip headers myself
>>> and also make sure the plug-in gets loaded correctly, which sounds hackish.
>>>
>>> Thanks,
>>>
>>>
>>> On Mon, Nov 25, 2013 at 11:12 AM, Otto van der Schaaf <oschaaf@gmail.com
>>> > wrote:
>>>
>>>> Wouldn't it be a solution to configure the gzip plugin to run after
>>>> your own transformation?
>>>>
>>>> Otto van der Schaaf
>>>>
>>>>
>>>> 2013/11/25 Daniel Morilha <dm...@gmail.com>
>>>>
>>>>> Hi,
>>>>>
>>>>> I am writing a transformation plug-in and I bumped up into the problem
>>>>> of deflating the content after the transformation kicks in.
>>>>>
>>>>> I believe that's a common problem that most of the transformation
>>>>> plug-ins may suffer with.
>>>>>
>>>>> I am aware ESI implements gzip itself and after giving a quick look
>>>>> into the GZIP plug-in and realizing it will be tough to integrate into
>>>>> another transformation plug-in I am inclined to do the same.
>>>>>
>>>>> Couldn't we have the gzip capabilities baked into the C API. The
>>>>> Linked'in atscppapi provides a nice way where you can inflate before the
>>>>> transformers and defalte after all of them had passed. Only problem is the
>>>>> dependency it creates on their APIs.
>>>>>
>>>>> thanks,
>>>>> --
>>>>> Daniel Morilha (dmorilha@gmail.com)
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Daniel Morilha (dmorilha@gmail.com)
>>>
>>
>>
>>
>> --
>> Daniel Morilha (dmorilha@gmail.com)
>>
>
>
--
Daniel Morilha (dmorilha@gmail.com)
Re: gzip
Posted by Otto van der Schaaf <os...@gmail.com>.
The gzip plugin will run when configured to do so, plus when the
accept-encoding request header meets the criteria (e.g. 'gzip' is specified)
Regrettably, it doesn't support inflation at this moment.
I think that should go into a separate plugin, so that can be configured
into its own place into the plugin chain (probably before any
transformations).
Otto
2013/11/25 Daniel Morilha <dm...@gmail.com>
> also AFAICT gzip plug-in has no inflate code.
>
>
> On Mon, Nov 25, 2013 at 11:19 AM, Daniel Morilha <dm...@gmail.com>wrote:
>
>> the transformation runs for some of the requests and I didn't see how to
>> tell the gzip to run or not... I don't want to unecessarily
>> inflate/deflate. Does it make sense?
>>
>> I was inclined into call the gzip plug-in internal APIs myself based on
>> those conditions but that would mean I would create the gzip headers myself
>> and also make sure the plug-in gets loaded correctly, which sounds hackish.
>>
>> Thanks,
>>
>>
>> On Mon, Nov 25, 2013 at 11:12 AM, Otto van der Schaaf <os...@gmail.com>wrote:
>>
>>> Wouldn't it be a solution to configure the gzip plugin to run after your
>>> own transformation?
>>>
>>> Otto van der Schaaf
>>>
>>>
>>> 2013/11/25 Daniel Morilha <dm...@gmail.com>
>>>
>>>> Hi,
>>>>
>>>> I am writing a transformation plug-in and I bumped up into the problem
>>>> of deflating the content after the transformation kicks in.
>>>>
>>>> I believe that's a common problem that most of the transformation
>>>> plug-ins may suffer with.
>>>>
>>>> I am aware ESI implements gzip itself and after giving a quick look
>>>> into the GZIP plug-in and realizing it will be tough to integrate into
>>>> another transformation plug-in I am inclined to do the same.
>>>>
>>>> Couldn't we have the gzip capabilities baked into the C API. The
>>>> Linked'in atscppapi provides a nice way where you can inflate before the
>>>> transformers and defalte after all of them had passed. Only problem is the
>>>> dependency it creates on their APIs.
>>>>
>>>> thanks,
>>>> --
>>>> Daniel Morilha (dmorilha@gmail.com)
>>>>
>>>
>>>
>>
>>
>> --
>> Daniel Morilha (dmorilha@gmail.com)
>>
>
>
>
> --
> Daniel Morilha (dmorilha@gmail.com)
>
Re: gzip
Posted by Daniel Morilha <dm...@gmail.com>.
also AFAICT gzip plug-in has no inflate code.
On Mon, Nov 25, 2013 at 11:19 AM, Daniel Morilha <dm...@gmail.com> wrote:
> the transformation runs for some of the requests and I didn't see how to
> tell the gzip to run or not... I don't want to unecessarily
> inflate/deflate. Does it make sense?
>
> I was inclined into call the gzip plug-in internal APIs myself based on
> those conditions but that would mean I would create the gzip headers myself
> and also make sure the plug-in gets loaded correctly, which sounds hackish.
>
> Thanks,
>
>
> On Mon, Nov 25, 2013 at 11:12 AM, Otto van der Schaaf <os...@gmail.com>wrote:
>
>> Wouldn't it be a solution to configure the gzip plugin to run after your
>> own transformation?
>>
>> Otto van der Schaaf
>>
>>
>> 2013/11/25 Daniel Morilha <dm...@gmail.com>
>>
>>> Hi,
>>>
>>> I am writing a transformation plug-in and I bumped up into the problem
>>> of deflating the content after the transformation kicks in.
>>>
>>> I believe that's a common problem that most of the transformation
>>> plug-ins may suffer with.
>>>
>>> I am aware ESI implements gzip itself and after giving a quick look into
>>> the GZIP plug-in and realizing it will be tough to integrate into another
>>> transformation plug-in I am inclined to do the same.
>>>
>>> Couldn't we have the gzip capabilities baked into the C API. The
>>> Linked'in atscppapi provides a nice way where you can inflate before the
>>> transformers and defalte after all of them had passed. Only problem is the
>>> dependency it creates on their APIs.
>>>
>>> thanks,
>>> --
>>> Daniel Morilha (dmorilha@gmail.com)
>>>
>>
>>
>
>
> --
> Daniel Morilha (dmorilha@gmail.com)
>
--
Daniel Morilha (dmorilha@gmail.com)
Re: gzip
Posted by Daniel Morilha <dm...@gmail.com>.
the transformation runs for some of the requests and I didn't see how to
tell the gzip to run or not... I don't want to unecessarily
inflate/deflate. Does it make sense?
I was inclined into call the gzip plug-in internal APIs myself based on
those conditions but that would mean I would create the gzip headers myself
and also make sure the plug-in gets loaded correctly, which sounds hackish.
Thanks,
On Mon, Nov 25, 2013 at 11:12 AM, Otto van der Schaaf <os...@gmail.com>wrote:
> Wouldn't it be a solution to configure the gzip plugin to run after your
> own transformation?
>
> Otto van der Schaaf
>
>
> 2013/11/25 Daniel Morilha <dm...@gmail.com>
>
>> Hi,
>>
>> I am writing a transformation plug-in and I bumped up into the problem of
>> deflating the content after the transformation kicks in.
>>
>> I believe that's a common problem that most of the transformation
>> plug-ins may suffer with.
>>
>> I am aware ESI implements gzip itself and after giving a quick look into
>> the GZIP plug-in and realizing it will be tough to integrate into another
>> transformation plug-in I am inclined to do the same.
>>
>> Couldn't we have the gzip capabilities baked into the C API. The
>> Linked'in atscppapi provides a nice way where you can inflate before the
>> transformers and defalte after all of them had passed. Only problem is the
>> dependency it creates on their APIs.
>>
>> thanks,
>> --
>> Daniel Morilha (dmorilha@gmail.com)
>>
>
>
--
Daniel Morilha (dmorilha@gmail.com)
Re: gzip
Posted by Otto van der Schaaf <os...@gmail.com>.
Wouldn't it be a solution to configure the gzip plugin to run after your
own transformation?
Otto van der Schaaf
2013/11/25 Daniel Morilha <dm...@gmail.com>
> Hi,
>
> I am writing a transformation plug-in and I bumped up into the problem of
> deflating the content after the transformation kicks in.
>
> I believe that's a common problem that most of the transformation plug-ins
> may suffer with.
>
> I am aware ESI implements gzip itself and after giving a quick look into
> the GZIP plug-in and realizing it will be tough to integrate into another
> transformation plug-in I am inclined to do the same.
>
> Couldn't we have the gzip capabilities baked into the C API. The Linked'in
> atscppapi provides a nice way where you can inflate before the transformers
> and defalte after all of them had passed. Only problem is the dependency it
> creates on their APIs.
>
> thanks,
> --
> Daniel Morilha (dmorilha@gmail.com)
>