You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by Jean Baptiste Favre <we...@jbfavre.org> on 2013/12/07 00:09:31 UTC

Buffered transformation plugin question

Hello,

I need to cache some json API response.
The API sent only indented json.

To reduce network trafic and optimize cache, I need to be able to remove
space and new lines.

I'm considering to use gzip. Problem is, this would invalidate current
cache which is not an option. But, when ATS will revalidate URL, it'll
use gzip and both gziped & normal content will be cache.

In the meantime, I'm trying to implement a plugin to handle json
transformation.
>From what I understand, I need to write a buffered transformation plugin
to make sure I got the whole json. So I studied bnull_tranform plugin
and read the documentation.

>From what I read, I have to add my code into handle_output function.
I need to use data->output_reader to get content into a char, transform
it with json-c library function.
Char result will be added into a new TSIOBuffer.
Once done, I'll be able to write it into output_conn.

Am I right ?

My problem is that I don't know how to get data->output_reader content
as char.
As far as I've seen, output_reader is a TSIOBufferReader structure and I
can not find any definition of it.

As you may have noticed, I'm quite a newbie in C, but any advice will be
apreciated.

Best regards,
Jean-Baptiste

Re: Buffered transformation plugin question

Posted by Jean Baptiste Favre <we...@jbfavre.org>.
On 09/12/2013 19:01, James Peach wrote:
> On Dec 6, 2013, at 3:09 PM, Jean Baptiste Favre <we...@jbfavre.org> wrote:
> 
>> Hello,
>>
>> I need to cache some json API response.
>> The API sent only indented json.
>>
>> To reduce network trafic and optimize cache, I need to be able to remove
>> space and new lines.
> 
> The LinkedIn folks wrote a PageSpeed plugin. I think it can do this, <https://github.com/linkedin/ats_pagespeed>.
I already tested it. But, either I missed some configuration or
pagespeed does not support 'application/json' content-type :-/
And pagespeed homepage does not mention JSON support:
https://developers.google.com/speed/pagespeed/module?hl=en

>> I'm considering to use gzip. Problem is, this would invalidate current
>> cache which is not an option. But, when ATS will revalidate URL, it'll
>> use gzip and both gziped & normal content will be cache.
> 
> Would is really get invalidated? If the client would accept compressed content and there is already an uncompressed variant in the cache, would the gzip plugin be able to send a compressed version from cache?
If a gzip request has been cached (and config option "cache" is set to
true), then later request with or without gzip will be served from cache.

But, for now, I only have non-gzip content cached. After enabling gzip,
first gzip request is seen as a miss, so request is forwarded to origin
server.

>> In the meantime, I'm trying to implement a plugin to handle json
>> transformation.
>> From what I understand, I need to write a buffered transformation plugin
>> to make sure I got the whole json. So I studied bnull_tranform plugin
>> and read the documentation.
>>
>> From what I read, I have to add my code into handle_output function.
>> I need to use data->output_reader to get content into a char, transform
>> it with json-c library function.
>> Char result will be added into a new TSIOBuffer.
>> Once done, I'll be able to write it into output_conn.
>>
>> Am I right ?
>>
>> My problem is that I don't know how to get data->output_reader content
>> as char.
>> As far as I've seen, output_reader is a TSIOBufferReader structure and I
>> can not find any definition of it.
>>
>> As you may have noticed, I'm quite a newbie in C, but any advice will be
>> apreciated.
> 
> I haven't written a transformation, but it seems they can be tricky. As Daniel Morilha pointed out on a different thread, the LinkedIn C++ API is a good additional reference for transformation code, <https://github.com/linkedin/atscppapi/blob/master/src/TransformationPlugin.cc>

I now have an "almost working plugin".
It's mostly based on bnull-tranformation example. I'm able to compact
response from origin server.
Now have to compact as well cached request so that I can spare some LAN
bandwitch :)

I agree this type of plugin can be very tricky to write.

I'll have a deeper look on pagespeed API module. I already saw it but
missed the TransformationPlugin part.

Finally, I had some trouble to understand bnull-tranformation example.
In fact, it does not work at all as it is. There is a very small
modification to do.
I'll make a pull request as soon as I can. At the same time, I'll
release my module so that you guys can have a look on it a criticize it :)

Best regards,
Jean-Baptiste

Re: Buffered transformation plugin question

Posted by James Peach <jp...@apache.org>.
On Dec 6, 2013, at 3:09 PM, Jean Baptiste Favre <we...@jbfavre.org> wrote:

> Hello,
> 
> I need to cache some json API response.
> The API sent only indented json.
> 
> To reduce network trafic and optimize cache, I need to be able to remove
> space and new lines.

The LinkedIn folks wrote a PageSpeed plugin. I think it can do this, <https://github.com/linkedin/ats_pagespeed>.

> I'm considering to use gzip. Problem is, this would invalidate current
> cache which is not an option. But, when ATS will revalidate URL, it'll
> use gzip and both gziped & normal content will be cache.

Would is really get invalidated? If the client would accept compressed content and there is already an uncompressed variant in the cache, would the gzip plugin be able to send a compressed version from cache?

> In the meantime, I'm trying to implement a plugin to handle json
> transformation.
> From what I understand, I need to write a buffered transformation plugin
> to make sure I got the whole json. So I studied bnull_tranform plugin
> and read the documentation.
> 
> From what I read, I have to add my code into handle_output function.
> I need to use data->output_reader to get content into a char, transform
> it with json-c library function.
> Char result will be added into a new TSIOBuffer.
> Once done, I'll be able to write it into output_conn.
> 
> Am I right ?
> 
> My problem is that I don't know how to get data->output_reader content
> as char.
> As far as I've seen, output_reader is a TSIOBufferReader structure and I
> can not find any definition of it.
> 
> As you may have noticed, I'm quite a newbie in C, but any advice will be
> apreciated.

I haven't written a transformation, but it seems they can be tricky. As Daniel Morilha pointed out on a different thread, the LinkedIn C++ API is a good additional reference for transformation code, <https://github.com/linkedin/atscppapi/blob/master/src/TransformationPlugin.cc>

James