Re: Cache invalidation for varying media types

[Yongming Zhao <mi...@gmail.com>]

let us move to the dev list. 

- Yongming Zhao 赵永明

在 2014年4月24日，上午5:52，Norm Paxton <no...@familysearch.org> 写道：

> Regarding the previously raised issue about natural invalidation of PUT/POST requests:
> Further testing revealed that the solution provided in the previous email was incorrect.
>  
> I have been testing with the attached patch (patch against 4.1.2), which with fairly extensive testing appears to be working well.  However there are three concerns with it:
> 1>     The patch calls do_cache_delete_all_alts() directly, rather than using the state machine to specify the ‘next_action’.  Without having a detailed knowledge of the state machine and valid next-states, I felt it was safer to do this – at least for proof of concept pending a more complete patch.
> Looking for suggestion on correct TRANSACT_RETURN or next_action for this, or if direct call rather than state machine management is correct in this case.
> 2>     The invalidation happens at the entry of the PUT/POST rather than at the exit/return to client.  This would not normally be a problem.  However I have a use case where a POST calls a separate webapp to validate the data, which then calls back with a GET using the ID.  This GET in the middle of the POST caches the object again, before the POST took effect.
> Looking for suggestions on where to put the invalidation so that it happens at the exit.  (Answer to <1> may correct <2>)
> 3>     This patch only invalidates alternates if the requested object is in the cache.
> Scenario:  (Think large multi-faceted web application with multiple client plugins, where some plugins use XML and others use JSON)
> GET application/xml puts the object in the cache.  POST application/json tries to find itself in the cache, but non-existent so doesn’t invalidate the alternates.  The now-stale application/xml version of the object stays in the cache.
>                 Looking for suggestions to invalidate alternates even if the requested object is not in the cache.
>  
> Thanks in advance.
>  
> -Norm Paxton
>  
> From: Norm Paxton [mailto:norman.paxton@familysearch.org] 
> Sent: Wednesday, February 05, 2014 11:49 AM
> To: users@trafficserver.apache.org
> Subject: RE: Cache invalidation for varying media types
>  
> Whereas our desire is that a PUT or POST of one content-type causes an invalidation not only of that instance of the object, but also of all alternates of that object, does it make sense to simply add the PUT and POST as additional options to the two s->method checks in HttpTransact::delete_all_document_alternates_and_return(), as follows?
>  
>   if ((s->method != HTTP_WKSIDX_GET) && (s->method == HTTP_WKSIDX_DELETE || s->method == HTTP_WKSIDX_PURGE)) {
>  
> to:
>  
>   if ((s->method != HTTP_WKSIDX_GET) && (s->method == HTTP_WKSIDX_DELETE || s->method == HTTP_WKSIDX_PURGE ||
>         s->method == HTTP_WKSIDX_POST || s->method == HTTP_WKSIDX_PUT)) {
>  
> and
>  
>     if (s->method == HTTP_WKSIDX_PURGE || (valid_max_forwards && max_forwards <= 0)) {
>  
> to:
>  
>     if (s->method == HTTP_WKSIDX_PURGE || s->method == HTTP_WKSIDX_POST || s->method == HTTP_WKSIDX_PUT ||
>          (valid_max_forwards && max_forwards <= 0)) {
>  
>  
> I have built and tested as such, and it behaves at a high level as we desire.  Will we experience unanticipated side-effects from doing this?  Is there a suite of tests that I can run to ensure that all other behavior is unaffected?
>  
> Thanks.
>  
> -Norm Paxton
>  
> From: Norm Paxton [mailto:norman.paxton@familysearch.org] 
> Sent: Monday, February 03, 2014 4:01 PM
> To: users@trafficserver.apache.org
> Subject: Cache invalidation for varying media types
>  
> There was a thread in June or 2013 regarding cache invalidation of multiple media types.
> (thread:  http://mail-archives.apache.org/mod_mbox/trafficserver-users/201306.mbox/%3CCDD7790C.27424%25steve.owens@email.disney.com%3E)
>  
> Summary of the initial question (Steve Owens):
> Is there a configuration (Vary header, proxy.config.http.cache.vary_default_*, etc) that would cause that a PUT or POST on a specific URL would cause a cache invalidation on all variant media types of that URL?
>  
> Summary of ultimate response (Leif Hedstrom):
> This sounds like PURGE, but not currently in PUT or POST.  We could look at putting that behavior into PUT/POST.
>  
> My revisit:
>  
> Has any action been taken on this?  (I have searched the changes.log and not found anything pertinent, but it is possible that I missed something)
> I have a REST api serving data with both ‘application/json’ and ‘application/xml’ content-types for each URL.  Different clients will consume one or both of these content-types.  When a POST is invoked with ‘application/json’, my assumption is that the cache server should invalidate the object for ‘application/xml’ as well, as this is a variant representation of the same object.   In my testing, I consistently get unexpected (according to my understanding and expectations) results:
> Here’s a summarized log of what I see
>  
>   GET      application/xml                 TCP_HIT                               baseline
>   GET      application/json               TCP_HIT                               baseline
>  
>   POST   application/xml                 TCP_REFRESH_MISS       expected
>   GET      application/json               TCP_HIT                               unexpected/incorrect/stale data
>   GET      application/xml                 TCP_MISS                           expected
>   GET      application/xml                 TCP_HIT                               expected
>  
> I have tried setting the ‘proxy.config.http.cache.vary_default_text’ and ‘…vary_default_other’ to Accept, and also setting the Vary header to Accept in my response, all to no avail.  I have also tried Accept values of “text/html”, “text/json”, “text/xml”, “text/plain” – in case th e’proxy.config.http.cache.vary_default_text’ only looks at ‘text/’ media type prefixes, also with no luck.  (Based on ATS documentation, and other sources, I am lead to believe that the correct header to Vary on is Accept, as opposed to Content-Type.  Correct?)
>  
> I have confirmed that I can invoke a ‘PURGE’ on the URL, which will cause desired end-results, but this requires that upon any PUT/POST request, I will need to, as part of my processing, explicitly invoke a PURGE in code to invalidate the cache for that object, which is undesirable.  For some developers/situations, it may even be prohibitive.
>  
> Is there  plans to change the behavior in a future release, or am I stuck with self-invoking a PURGE request inside code handling my PUT/POST request?
> Or are there other options for configuring this that I have not found/tried?
>  
> Thanks
>  
> -Norm Paxton
>  
>  
> 
> 
> NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
>  
> 
> 
> NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
>  
> 
> 
> NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
> 
> <0001-Invalidate-PUT-and-POST.patch>