DO NOT REPLY [Bug 30356] New: - CIncludeTransformer + event aware caching

[bu...@apache.org]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=30356>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://issues.apache.org/bugzilla/show_bug.cgi?id=30356

CIncludeTransformer + event aware caching

           Summary: CIncludeTransformer + event aware caching
           Product: Cocoon 2
           Version: 2.1.5
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: Normal
          Priority: Other
         Component: general components
        AssignedTo: dev@cocoon.apache.org
        ReportedBy: oliver.powell@tvnz.co.nz
                CC: cmoss@tvnz.co.nz


>We're implementing Event Aware caching at the moment, and it's working
>well.  We're migrating from the Prism based cache which we've used 
>until now.  Our only hurdle at the moment is proving to be the 
>CIncludeTransformer.  Expires based caching is hard wired right into it 
>(and the DefaultIncludeCacheManager).

We've made some modifications to the DefaultIncludeCacheManager,
CIncludeTransformer and IncludeCacheManager classes to enable
CIncludeTransformer to "inherit" the validity of the source it included.
ExpiresValidity is still supported as well. I'll attach the source files after
committing the bug entry.

In a nutshell, the changes are:
1. DefaultIncludeCacheManager stream() - At the end of the method, when not
processing in parallel and have no valid cached response: Currently, after
resolving the URI to get the source object, the new response is only cached if
session.getExpires > 0. This has been changed to only require supportsCaching =
true. 

2. Then, it sets the response's validity = source.getValidity. If expires has a
value, then it also adds an ExpiredValidity and puts the lot in an
AggregatedValidity object.

3. Then, after the new CachedResponse has been stored in the include's
CacheStore, it also stores the same validity object against the
IncludeCacheManagerSession instance (using a new method setValidity). This is
necessary so that the CIncludeTransformer can return this same validity (it has
access to the session object) when the pipeline processor calls getValidity on it.

4. CIncludeTransformer: changed getValidity method to retrieve the validity
stored with the IncludeCacheManagerSession, instead of creating a new
ExpiresValidity.

For more info, I've placed comments next to the changes in the source code.

These classes have not been rigourously tested yet, in fact I haven't really
tested this with EventValidities yet. But I can verify that this code does work
when the ci:included source has a FileStampValidity:

Example sitemap (simplified, not an actual pipeline we are using):

<map:pipeline type="caching-point">
  <map:match pattern="cinclude_test/**">
    <map:generate type="serverpages" src="get_file_path.xsp">
      <map:parameter name="param" value="{1}" />
    </map:generate>
    <map:transform type="cinclude">
      <map:parameter name="support-caching" value="true"/>
    </map:transform>
    <map:serialize type="xml" />
  </map:match>

  <map:match pattern="file/**">
    <map:generate src="{1}.xml" />
    <map:serialize type="xml"/>
  </map:match>
</map:pipeline>

Where get_file_path.xsp looks up the file path from a db and then outputs the
following CInclude command:
<ci:include>
 <xsp:attribute name="src">cocoon:/file/<esql:get-string
column="file_path"/></xsp:attribute>
</ci:include>

Previously, without the expires parameter in the ci:include map:transform step,
the transform step would not be cacheable because getValidity would always
return null. This meant that the "cinclude_test/**" pipeline responses would not
get cached (well, not past the xsp generator point). This is a problem for us
because expires caching isn't suitable for our needs. We need Event Cache
functionality for these pipelines.

With the code changes, the CIncludeTransformer now takes on the validity of the
source it has included. This seems to me to be the desired approach, as anything
that invalidates that source will also automatically invalidate the include step
in the outer pipeline as well.

My next step is to test this with EventValidity - ie. the ci:included source has
an EventValidity object rather than FileStampValidity. This will of course mean
changing the pipeline cache to "event-aware" as well.

Note, I'm not suggesting that the changes should be done in exactly the way
shown in the source files. We just wanted to start off any feedback and
enhancements asap.

Thanks for your help!
Oliver