You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by "Roy T. Fielding" <fi...@gbiv.com> on 2006/12/07 01:42:22 UTC

Wrong etag sent with mod_deflate

Protocol issues really should be brought up on the dev list, with an
appropriate subject, and not left in bugzilla.

    http://issues.apache.org/bugzilla/show_bug.cgi?id=39727

Entities gzip:ed by mod_deflate still carries the same ETag as the  
plain entiy,
causing inconsistency in ETag aware proxy caches.

I'll have a look later and see if I can fix it, but let me know if there
is already a patch in the works (that doesn't rely on mod_filter).

....Roy

On Dec 6, 2006, at 1:31 PM, Henrik Nordstrom wrote:

> ons 2006-12-06 klockan 09:38 -0500 skrev Jeff Trawick:
>
>> Why other than ego do we want to make it hard to disable this output?
>
> Technical reason:
>
> Not advertising the brand and version makes it very hard for clients
> (user-agents and proxies) to apply workarounds when needed.
>
> As an example Squid currently has a workaround for how Apache handles
> ETag in responses which has been modified by mod_deflate. In future we
> hope to be able to disable that for versions known to be fixed.
>
> http://issues.apache.org/bugzilla/show_bug.cgi?id=39727
>
> Not sending the sever name and version will make this harder.
>
> Regards
> Henrik


Re: Wrong etag sent with mod_deflate

Posted by Henrik Nordstrom <hn...@squid-cache.org>.
lör 2006-12-09 klockan 19:02 +0100 skrev Justin Erenkrantz:

> AIUI, many caches do not allow the response to be cached at all if it
> doesn't have an ETag.

Most still caches it, but for example Mozilla has bugs vrt Vary handling
if there is no ETag and the conditions changes..

> In the ideal world, I think a weak ETag would be the 'right' thing

I don't have an opinion if you return a strong or weak ETag, but it must
still be different than the ETag of the identity encoded object, not
just the same ETag flagged as weak.

Your main decision if the ETag on the mod_deflate generated entity
should be weak or strong should be

a) If the original entity is weak, then the mod_deflate generated one
MUST be weak as well..

b) If mod_deflate can not be trusted to generate the exact same octet
representation on each request then the ETag of the generated entity
MUST be weak.

Else the ETag SHOULD be strong.

> however, the current spec doesn't allow conditional GETs to work with
> weak ETags.

Err.. Weak ETags is allowed in If-None-Match for GET/HEAD.

Regards
Henrik



Re: Wrong etag sent with mod_deflate

Posted by Ruediger Pluem <rp...@apache.org>.

On 12/09/2006 07:02 PM, Justin Erenkrantz wrote:
> On 12/9/06, Ruediger Pluem <rp...@apache.org> wrote:
> 
>> Thanks for giving the pointer to ap_meets_conditions. So content
>> compressed
>> by mod_deflate would not stand conditional requests based on ETags any
>> longer.
>> That would be bad. Would it help if we simply unset the ETag in
>> mod_deflate?
>> mod_filter does this in these situations or does this have any other
>> nasty
>> side effects?
> 
> 
> AIUI, many caches do not allow the response to be cached at all if it
> doesn't have an ETag.  This is why it was brought up that not doing

AFAICS this is not the case for mod_cache. As long as at least one of
the following headers is present mod_cache can cache the response if
all other conditions needed are true:

ETag
Last-Modified
Expires

Regards

Rüdiger

Re: Wrong etag sent with mod_deflate

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 12/9/06, Ruediger Pluem <rp...@apache.org> wrote:
> Thanks for giving the pointer to ap_meets_conditions. So content compressed
> by mod_deflate would not stand conditional requests based on ETags any longer.
> That would be bad. Would it help if we simply unset the ETag in mod_deflate?
> mod_filter does this in these situations or does this have any other nasty
> side effects?

AIUI, many caches do not allow the response to be cached at all if it
doesn't have an ETag.  This is why it was brought up that not doing
deflate at all might be better in some cases than removing the ETag.

> So what I understand from the current discussion is that
>
> 1. Using TE instead of CE would be RFC compliant and would relief us of
>    much problems except the one that none of the major browsers can handle
>    it and thus would effectively make mod_deflate useless.

Right.

> 2. There are two different points of view in the CE case:
>    Roy and Henrik say that a strong ETag arriving at mod_deflate must
>    be replaced with a different strong ETag within mod_deflate (e.g
>    by adding -gzip to it), because as mod_deflate is doing CE the entities
>    before and after mod_deflate are different and require different ETags.
>    Justin OTH says that it is sufficient to convert a strong ETag into a
>    weak one, right?

In the ideal world, I think a weak ETag would be the 'right' thing -
however, the current spec doesn't allow conditional GETs to work with
weak ETags.  Therefore, to allow conditional GETs, mod_deflate can
only produce strong ETags.  However, to make conditional GETs work and
to create a different ETag, the transformation has to be reversible -
which I believe may become a sticking point.

(BTW, I disagree with Roy and Henrik that the transformation that
mod_deflate is applying changes the actual meaning of the content; but
that's largely an irrelevant and academic point for this list.)

HTH.  -- justin

Re: Wrong etag sent with mod_deflate

Posted by Henrik Nordstrom <hn...@squid-cache.org>.
fre 2006-12-08 klockan 15:35 -0800 skrev Justin Erenkrantz:

> As Kevin mentioned, Squid is only using the ETag and is ignoring the
> Vary header.  That's the crux of the broken behavior on their part.
> If they want to point out minor RFC violations in Apache, then we can
> play that game as well.  (mod_cache deals with this Vary/ETag case
> just fine, FWIW.)

We are not at all ignoring Vary, but we are using If-None-Match to ask
the server which one of the N already cached entities belonging to the
resource URI is valid for this specific request, indirectly learning the
server side content negotiation logics used.

> The compromise I'd be willing to accept is to have mod_deflate support
> the 'TE: gzip' request header and add 'gzip' to the Transfer-Encoding
> bit - and to prefer that over any Accept-Encoding bits that are sent.

Would be a great move if you can not make it behave correct in the
content space.

But if you make mod_deflate behave according to the RFC then sending
Content-Encoding: gzip is fine to me. But TE is a much better fit from
the RFC point of view.

> The ETag can clearly remain the same in that case - even as a strong
> ETag.

Yes.

> So, Squid can change to send along TE: gzip (if it isn't
> already).

TE: gzip is likely to appear in 3.1.

> And, everyone else who sends Accept-Encoding gets the
> result in a way that doesn't pooch their cache if they try to do a
> later conditional request.

As long as mod_deflate continues ignoring the RFC wrt ETag there will
conflicts with various cache implementations.

> Is that acceptable?  -- justin

Intentionally not following a MUST level requirements in the RFC is not
an acceptable solution in my eyes. For one thing even if you ignore
everyone else it would make it impossible for Apache + mod_deflate to
claim RFC 2616 HTTP/1.1 compliance.

Regards
Henrik

Re: Wrong etag sent with mod_deflate

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 12/10/06, Roy T. Fielding <fi...@gbiv.com> wrote:
> No, you assume a broken implementation of ap_meets_conditions.
> If the function is implemented correctly, it still treats each etag
> as opaque.  Given an etag of "X", the deflate variant will be "X-gzip".
> If the content generator generates "10//foo-gzip", then the deflate
> filtered variant will use "10//foo-gzip-gzip".  The result is correct
> and no butt-ugly hacks are needed.  The same can be done for any
> content-changing filter, and the result will work as specified as
> long as the ordering of filters is consistent (and even if it isn't,
> the result remains correct with a slight inefficiency).
>
> The hard part is to fix ap_meet_conditions and/or the metadata
> generating aspect of 2.x filters, which may not even be possible
> without an API rewrite.  I am not asking anyone to accept partial
> solutions -- it is simply a bug that needs to be fixed, and I see no
> reason to assume that we can't fix it right.  Just give me some time
> to get trunk working on OS X again and remember how to code in C.

Thank you.

Right now, my intuition leads me to believe that to fix it properly
would indeed require rewriting our entire filter API.  At that point,
the floodgates open and let's just do 3.0 from scratch.  =)

My spidey sense says that if we were to follow the bucket design from
serf, there'd be a way out of this.  But, I'm not seeing how to do
that within the constraints of httpd's current design.  -- justin

Re: Wrong etag sent with mod_deflate

Posted by Henrik Nordstrom <hn...@squid-cache.org>.
ons 2006-12-13 klockan 08:51 -0500 skrev Brian Akins:

> However, on an initial request (ie, non-conditional) we do not have an etag from 
> the client, we only have info like Host, URI, Accept-*, etc.  So, how would the 
> cache identify which entity to serve in this case?

You have the URL and the "other" cached entities of that URL. It does
not matter if the client request was a conditional or not. The
conditions in the request is on the response to see if it should be a
200 or 304, not selectors on what entity to respond with. The selected
response entity is always the same for the same request, with or without
conditions.

Obviously on the very first request for a given URL you have nothing,
and that request is forwarded without any added condition. However,
after that every Vary cache miss on that URL is a If-None-Match
conditional to ask the server if any of the cached entity variants is
applicable for the current request.

> I have read it many times.. In our case - cnn.com, etc. - we have to decided to 
> be RFC "compliant" from the client to the cache server.  From the cache to the 
> origin, however, we are not as concerned.

And you are free to. A reverse proxy is by definition the origin server.
How it finds the content is of no concern to the RFC, just happens to be
HTTP and not plain files, NFS, database or whatever.

> In a reverse-proxy-cache, this is not 
> a big deal. However, in a "normal forward-proxy-cache," where one does not 
> control both cache and origin, one must be more careful.

Indeed.

But on the other hand it's actually reverse proxy configurations which
has pushed for 13.6 compliance in Squid as it's a lot easier for
processing intensive servers to evaluate If-None-Match than to render
the entity again, and when you depend on Accept-Language +
Accept-Encoding + User-Agent the number of request combinations becomes
quite significant, especially if there maybe only is two or three
variants under the URL.

Regards
Henrik

Re: Wrong etag sent with mod_deflate

Posted by Brian Akins <br...@turner.com>.
Henrik Nordstrom wrote:
> But the unique identity of the response entity is defined by request-URI
> + ETag and/or Content-Location. The cache is not supposed to evaluate
> Accept-* headers in determining the entity identity, only the origin
> server.

However, on an initial request (ie, non-conditional) we do not have an etag from 
the client, we only have info like Host, URI, Accept-*, etc.  So, how would the 
cache identify which entity to serve in this case?

> Please see RFC2616 13.6 Caching Negotiated Responses, it explains how
> the RFC intends that caches should operate wrt Vary, ETag and
> Content-Location in full detail.

I have read it many times.. In our case - cnn.com, etc. - we have to decided to 
be RFC "compliant" from the client to the cache server.  From the cache to the 
origin, however, we are not as concerned.  In a reverse-proxy-cache, this is not 
a big deal. However, in a "normal forward-proxy-cache," where one does not 
control both cache and origin, one must be more careful.


-- 
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies

Re: Wrong etag sent with mod_deflate

Posted by Henrik Nordstrom <hn...@squid-cache.org>.
tis 2006-12-12 klockan 09:20 -0500 skrev Brian Akins:

> Only conditional requests from clients, generally, have If-None-Match headers. 

Correct. It's a conditional. These days you also see them from Squid
btw.

> So the only way for a cache, on an initial request from a client, to determine 
> what object to serve is to use the Client supplied information - which doesn't 
> include an Etag, so you have to, usually, rely on URI first, and then the Vary 
> information.

Indeed. This is always the case. If-None-Match MUST NOT be used for
identification of which response to use. It's a conditional only.

But the unique identity of the response entity is defined by request-URI
+ ETag and/or Content-Location. The cache is not supposed to evaluate
Accept-* headers in determining the entity identity, only the origin
server.

The identity of the entity is important for

- Cache correctness, making sure updates invalidate cached copies where
needed.

- Avoiding duplicated storage

There may be any number of request header combinations in any Vary
dimensions all mapping to the same entity.

This logics is not at all unique for Accept-Encoding. The logics on how
a cache is supposed to operate applies equal to all Vary indicated
headers. The specs does not make any distinction between
Accept-Encoding, Accept-Language, User-Agent etc in how caches are
supposed to operate. It all boils down to the entity identified by URI +
ETag and/or Content-Location as returned in 200 and 304 responses
allowing the cache to map requests to entities.

Please see RFC2616 13.6 Caching Negotiated Responses, it explains how
the RFC intends that caches should operate wrt Vary, ETag and
Content-Location in full detail.

Regards
Henrik

Re: Wrong etag sent with mod_deflate

Posted by Brian Akins <br...@turner.com>.
Henrik Nordstrom wrote:
> mån 2006-12-11 klockan 14:25 -0500 skrev Brian Akins:
> 
>> So, multiple variants of the same object can have the same Etag, but still be 
>> different cached objects.
> 
> Your implementation ignores RFC 2616 13.6 Caching Negotiated Responses,
> but is otherwise fine. It's functionally compliant but not as effective
> as it could be.

That was a simplified explanation, we actually do not store a cache entry for 
every single variant.  In our case the only thing we actually ever care about is 
whether or not you support gzip.  So all the variants for "Vary: User-Agent, 
Accept-Encoding" actually boil down to 2 variants - gzip or no-gzip.

One of the major reasons we quit using squid was it "support" for Vary's. (This 
was pre-3.0, so things may have changed). Of course, at the time httpd wasn't 
any better - but it was alot easier to hack ;)


 >Variants is
 >identified by ETag or Content-Location. Only if there is neither ETag or
 >Content-Location in the response entity then is the response entity
 >identified by the Vary request headers.

Only conditional requests from clients, generally, have If-None-Match headers. 
So the only way for a cache, on an initial request from a client, to determine 
what object to serve is to use the Client supplied information - which doesn't 
include an Etag, so you have to, usually, rely on URI first, and then the Vary 
information.


-- 
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies

Re: Wrong etag sent with mod_deflate

Posted by Henrik Nordstrom <he...@henriknordstrom.net>.
mån 2006-12-11 klockan 14:25 -0500 skrev Brian Akins:

> So, multiple variants of the same object can have the same Etag, but still be 
> different cached objects.

Your implementation ignores RFC 2616 13.6 Caching Negotiated Responses,
but is otherwise fine. It's functionally compliant but not as effective
as it could be.

The drawback is that you end up with as many copies of the same entity
as there is slight variations in Vary related headers.

Example: A document available in en and de languages:

Accept-Language: en
Accept-Language: en-gb
Accept-Language: en, sv
Accept-Language; en, sv;q=0.7

etc all map to the en language entity, but if you do not follow 13.6
then the cache is unaware that these all results in the same entity and
N copies of the "Content-Language: en" response entity is cached.

In caches following 13.6 only a single copy of the "Content-Language:
en" response entity is stored, shared by all the Accept-Language header
combinations known to result in that entity identified by ETag. The
cache learns this request -> entity mapping by using If-None-Match.

Variants in HTTP is about the response entity. Request headers select
the variant, but they do not identify the variant. Variants is
identified by ETag or Content-Location. Only if there is neither ETag or
Content-Location in the response entity then is the response entity
identified by the Vary request headers.

Regards
Henrik

Re: Wrong etag sent with mod_deflate

Posted by Brian Akins <br...@turner.com>.
This is not a response to any post on this subject, but more of a comment.  Here 
is a real world example of how we use deflate and etags with our cache. (Note 
this is very similar to mod_cache, but I do not know the inner workings of it as 
well).

1. Generate key from URI and ap_get_servername
2. open cached object.  Is it "Vary?" no, goto step 5.
3. Is "Vary." Generate new key.
4. Open cached object.
5. Check expiry time, "exit" if expired.
6. Load headers.
7. Call ap_meets_conditions (etags, IMS, etc.)  If yes, return 304 (or whatever).
8. If not meets_conditions, serve from cache.

So, multiple variants of the same object can have the same Etag, but still be 
different cached objects.

This probably has no bearing on the current conversation, but perhaps I am not 
fully appreciating the core of the debate??

-- 
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies

Re: Wrong etag sent with mod_deflate

Posted by Henrik Nordstrom <hn...@squid-cache.org>.
sön 2006-12-10 klockan 17:01 +0100 skrev Ruediger Pluem:

> Are multiple transfer encodings applied to one message allowed?

Yes, but the last (outer) one should in any sane implementation be
chunked as it defines the message format. Use of other encodings is
allowed without chunked but the message is then terminated by closing
the connection and no clear indication if the transfer was complete
unless the transfer encoding used provides. The use of Content-Length is
forbidden in combination with Transfer-Encoding: gzip or any other
encoding modifying the length of the response.

RFC 2616 3.6 Transfer Codings and 14.41 Transfer-Encoding.

Regards
Henrik

Re: Wrong etag sent with mod_deflate

Posted by Ruediger Pluem <rp...@apache.org>.

On 12/10/2006 04:36 PM, Justin Erenkrantz wrote:
> On 12/10/06, Ruediger Pluem <rp...@apache.org> wrote:
> 
>> Question: How does a client signal that it can handle TE: gzip? Also
>> via Accept-Encoding: gzip?
>> Or is there another header for this?
> 
> 
> 'TE: gzip' is the header that the client sends in the request.  Apache
> would then respond with 'Transfer-Encoding: gzip'.  We would also have
> to tweak the chunking filter to also be able to chunk on top of that
> as well.  (Currently, it would bail if there is any Transfer-Encoding
> besides identity.  If a client actually supports TE: gzip, then it
> better also be able to handle chunking on top of that.)  -- justin

Are multiple transfer encodings applied to one message allowed? And if yes,
do we need to take care of their order in the Transfer-Encoding: header of the response?
How does this relate to the qvalue of section 14.39 of RFC2616?


Regards

Rüdiger


Re: Wrong etag sent with mod_deflate

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 12/10/06, Ruediger Pluem <rp...@apache.org> wrote:
> Question: How does a client signal that it can handle TE: gzip? Also via Accept-Encoding: gzip?
> Or is there another header for this?

'TE: gzip' is the header that the client sends in the request.  Apache
would then respond with 'Transfer-Encoding: gzip'.  We would also have
to tweak the chunking filter to also be able to chunk on top of that
as well.  (Currently, it would bail if there is any Transfer-Encoding
besides identity.  If a client actually supports TE: gzip, then it
better also be able to handle chunking on top of that.)  -- justin

Re: Wrong etag sent with mod_deflate

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 12/10/06, Ruediger Pluem <rp...@apache.org> wrote:
> Plus things could be even worse than with the default handler. Let's assume that
> we have a content handler outside of httpd (cgi script, something on a Tomcat) which
> sets its own ETag and also handles conditional requests. Lets assume further
> that checking the ETag supplied in a conditional request is a very cheap operation
> compared to generating the content.
> That would mean once we add mod_deflate into the output chain and have it modify the
> ETag, every conditional request for a compressed variant of the resource would
> cause the backend to regenerate the whole content as it would not be able to check on
> the mod_deflate modified ETag :-(.
> I guess the only way to get out of this is to use TE instead of CE in this case.

BTW, you just described Subversion and any other mod_dav backends.  =)

This gets to why ap_meets_conditions can't be done inside of the
current filter chain - it'd just be too darn expensive.  -- justin

Re: Wrong etag sent with mod_deflate

Posted by Henrik Nordstrom <hn...@squid-cache.org>.
lör 2006-12-09 klockan 22:51 -0800 skrev Roy T. Fielding:

> implies a different representation will be selected.  The standard
> was crafted to make caching as efficient as possible.  I just wish
> it was easier to read.

13.6 isn't very hard to read once you get to it, and brings a lot of
light on how ETag is supposed to work. I think the trouble is that
server authors generally do not read that section of the RFC as it's
looks related to caching only and it's not obvious it also explains the
server side logics and why ETag must work the way it's defined far
better than the ETag definition alone is capable of doing.

> When it comes to HTTP, Apache leads -- it does not follow blindly,
> nor does it wait for spineless browser developers to introduce a
> feature first.  We implement it and that act alone is sufficient
> to break the chicken-and-egg dilemma that everyone else uses to
> justify their lack of priorities.

And the Squid developers is very thankful for this. If you did not we
would still not be caching Vary responses at all, nor would we have
dared to venture down the ETag lane making Vary response caching
actually work.

It's unfortunate that we did not verify mod_deflate when this work was
done many years ago (2001), only mod_negotiation. The problems with
mod_deflate was only discovered this summer when we felt ready to get
ETag based caching into the mainline version.

Regards
Henrik

Re: Wrong etag sent with mod_deflate

Posted by Ruediger Pluem <rp...@apache.org>.

On 12/10/2006 12:51 PM, Ruediger Pluem wrote:

> 
> Anyway I guess one reason why we call ap_meets_conditions that early is that
> we want to avoid running through the whole filter chain for a response
> that has not been modified. OTH I fear that we are not in a position at this
> point of time to clearly decide what variant of this resource we really have
> been asked for (mod_deflate decides on this later depending on the request headers
> e.g. Accept-Encoding and some other things) and thus to decide if it has been
> modified or not.

Plus things could be even worse than with the default handler. Let's assume that
we have a content handler outside of httpd (cgi script, something on a Tomcat) which
sets its own ETag and also handles conditional requests. Lets assume further
that checking the ETag supplied in a conditional request is a very cheap operation
compared to generating the content.
That would mean once we add mod_deflate into the output chain and have it modify the
ETag, every conditional request for a compressed variant of the resource would
cause the backend to regenerate the whole content as it would not be able to check on
the mod_deflate modified ETag :-(.
I guess the only way to get out of this is to use TE instead of CE in this case.

Question: How does a client signal that it can handle TE: gzip? Also via Accept-Encoding: gzip?
Or is there another header for this?

Regards

Rüdiger

Re: Wrong etag sent with mod_deflate

Posted by Ruediger Pluem <rp...@apache.org>.

On 12/10/2006 07:51 AM, Roy T. Fielding wrote:
> On Dec 9, 2006, at 6:23 AM, Justin Erenkrantz wrote:
> 
>> On 12/9/06, Ruediger Pluem <rp...@apache.org> wrote:
>>
>>> Would the following patch address all your points for a CE 
>>> mod_deflate filter?
>>
>>
>> No - this patch breaks conditional GETs which is what I'm against.
> 
> 
> Right, the hard part is fixing the effect on ap_meet_conditions, but
> it is nice to have an example of the easy part.  Thanks Ruediger.
> 
>> See the problem here is that you have to teach ap_meets_conditions()
>> about this.  An ETag of "1234-gzip" needs to also satisfy a
>> conditional request when the ETag when ap_meets_conditions() is run is
>> "1234".  In other words, ap_meets_conditions() also needs to strip
>> "-gzip" if it is present before it does the ETag comparison.  But, the
>> issue is that there is no real way for us to implement this without a
>> butt-ugly hack.
> 
> 
> That's not true. The content generator and filters need to be  constructed
> for a 304 response in just the way that they would be for a 200  response,
> and then they must produce the metadata for that response before
> ap_meets_conditions() is even called.  That is the only valid way to
> produce a 304 or HEAD response.
> 
> ap_meets_conditions is supposed to evaluate the metadata produced and
> make its decision just prior to applying the method (sending the body).
> It doesn't need to know anything about the etag structure -- it just
> needs the same etag that would have been generated for a 200 response.
> 

I think this is the core of the problem with ap_meets_conditions here.
Lets take the example of a static file on the webserver that is handled
by the default handler.
Currently ap_meets_conditions is called by the default handler just before
it pushes down the brigade through the filters. If it stays there I think
Justin is right that adjusting it is a difficult and hacky thing.
It would work opaque if we would call it later in the filter chain
(I guess just before ap_http_header_filter would be the right place).
Not sure what other things this breaks.

Anyway I guess one reason why we call ap_meets_conditions that early is that
we want to avoid running through the whole filter chain for a response
that has not been modified. OTH I fear that we are not in a position at this
point of time to clearly decide what variant of this resource we really have
been asked for (mod_deflate decides on this later depending on the request headers
e.g. Accept-Encoding and some other things) and thus to decide if it has been
modified or not.

Regards

Rüdiger


Re: Wrong etag sent with mod_deflate

Posted by Henrik Nordstrom <hn...@squid-cache.org>.
sön 2006-12-10 klockan 11:06 +0200 skrev Issac Goldstand:

> IMHO, this fixes the symptom, but not the problem; what happens when 
> someone comes up with a model where the compression level can be changed 
> (e.g., gzip --fast .. --best)?  In such a case, the content-encoding 
> still works (since the decompresser will detect this on its own), but 
> the entity will still be different in different cases which turns us 
> back to the same "different ETag" problem. 

If you are worried about this then make it a weak (and modified) etag.
This kind of things is exactly what the weak property is meant to be
used for, where semantic equivalence is guaranteed but not octet
equivalence.

If you can it i's probably better if the gzip parameters can be encoded
into the etag, but it's not always obvious which approach is best unless
conditionals allowed to be weak can also be made to ignore those
parameters..

Regards
Henrik

Re: Wrong etag sent with mod_deflate

Posted by Issac Goldstand <ma...@beamartyr.net>.

Roy T. Fielding wrote:
> On Dec 9, 2006, at 6:23 AM, Justin Erenkrantz wrote:
>> However, I disagree with Roy in that we most certainly *do* treat the
>> ETag values as opaque - Subversion has its own ETag values - Roy's
>> position only works if you assume the core is assigning the ETag value
>> which has a set format - not a third-party module.  IMO, any valid
>> solution that we deploy must work *independently* of what any module
>> may set ETag to.  It is perfectly valid for a 3rd-party module to
>> include "-gzip" at the end of their ETag.  For example, if you had a
>> file called "foo-gzip" in revision 10, SVN would assign the ETag
>> "10//foo-gzip".  (And, I could construct a conflict where httpd would
>> hork the ETag incorrectly for any arbitrary value.)  -- justin
>
> No, you assume a broken implementation of ap_meets_conditions.
> If the function is implemented correctly, it still treats each etag
> as opaque.  Given an etag of "X", the deflate variant will be "X-gzip".
> If the content generator generates "10//foo-gzip", then the deflate
> filtered variant will use "10//foo-gzip-gzip".  The result is correct
> and no butt-ugly hacks are needed.  The same can be done for any
> content-changing filter, and the result will work as specified as
> long as the ordering of filters is consistent (and even if it isn't,
> the result remains correct with a slight inefficiency).
>
IMHO, this fixes the symptom, but not the problem; what happens when 
someone comes up with a model where the compression level can be changed 
(e.g., gzip --fast .. --best)?  In such a case, the content-encoding 
still works (since the decompresser will detect this on its own), but 
the entity will still be different in different cases which turns us 
back to the same "different ETag" problem. 

  Issac

PS.  Roy, just out of curiosity, since HTTP/1.1 was already a standard 
before I had any involvement with the community, why was there a 
differentiation between Content-Encoding and Transfer-Encoding in 
HTTP/1.1 in the first place?   It seems to me that we wouldn't need to 
deal with these problems if compression was made a property of the 
message, rather than the entity.  (I'm sure the answer is a simple and 
valid one; I'm just asking if you can point me in the correct direction)

Re: Wrong etag sent with mod_deflate

Posted by "Roy T. Fielding" <fi...@gbiv.com>.
On Dec 9, 2006, at 6:23 AM, Justin Erenkrantz wrote:
> On 12/9/06, Ruediger Pluem <rp...@apache.org> wrote:
>> Would the following patch address all your points for a CE  
>> mod_deflate filter?
>
> No - this patch breaks conditional GETs which is what I'm against.

Right, the hard part is fixing the effect on ap_meet_conditions, but
it is nice to have an example of the easy part.  Thanks Ruediger.

> See the problem here is that you have to teach ap_meets_conditions()
> about this.  An ETag of "1234-gzip" needs to also satisfy a
> conditional request when the ETag when ap_meets_conditions() is run is
> "1234".  In other words, ap_meets_conditions() also needs to strip
> "-gzip" if it is present before it does the ETag comparison.  But, the
> issue is that there is no real way for us to implement this without a
> butt-ugly hack.

That's not true. The content generator and filters need to be  
constructed
for a 304 response in just the way that they would be for a 200  
response,
and then they must produce the metadata for that response before
ap_meets_conditions() is even called.  That is the only valid way to
produce a 304 or HEAD response.

ap_meets_conditions is supposed to evaluate the metadata produced and
make its decision just prior to applying the method (sending the body).
It doesn't need to know anything about the etag structure -- it just
needs the same etag that would have been generated for a 200 response.

> However, I disagree with Roy in that we most certainly *do* treat the
> ETag values as opaque - Subversion has its own ETag values - Roy's
> position only works if you assume the core is assigning the ETag value
> which has a set format - not a third-party module.  IMO, any valid
> solution that we deploy must work *independently* of what any module
> may set ETag to.  It is perfectly valid for a 3rd-party module to
> include "-gzip" at the end of their ETag.  For example, if you had a
> file called "foo-gzip" in revision 10, SVN would assign the ETag
> "10//foo-gzip".  (And, I could construct a conflict where httpd would
> hork the ETag incorrectly for any arbitrary value.)  -- justin

No, you assume a broken implementation of ap_meets_conditions.
If the function is implemented correctly, it still treats each etag
as opaque.  Given an etag of "X", the deflate variant will be "X-gzip".
If the content generator generates "10//foo-gzip", then the deflate
filtered variant will use "10//foo-gzip-gzip".  The result is correct
and no butt-ugly hacks are needed.  The same can be done for any
content-changing filter, and the result will work as specified as
long as the ordering of filters is consistent (and even if it isn't,
the result remains correct with a slight inefficiency).

The hard part is to fix ap_meet_conditions and/or the metadata
generating aspect of 2.x filters, which may not even be possible
without an API rewrite.  I am not asking anyone to accept partial
solutions -- it is simply a bug that needs to be fixed, and I see no
reason to assume that we can't fix it right.  Just give me some time
to get trunk working on OS X again and remember how to code in C.

The one problem that might come up is if subversion is incorrectly
using mod_deflate as a makeshift content-encoding replacement for
transfer-encoding *and* assuming that the entity-tag will remain the
same for both variants when it makes webdav requests.  In that case,
subversion needs transfer encoding now, not later.

There is absolutely nothing new here.  Dynamic content encodings
were discussed back in 1993 when the Content-Encoding field was
(mistakenly) introduced, and transfer encoding was specifically
intended to avoid these problems.  IIRC, I had the same discussion
with Jeff Mogul over ten years ago and the algorithm that he put
in the spec does work if and only if it is implemented completely.
The reason it is specified in this way is because there are an
infinite number of ways that request header field values could vary
and yet (almost always) a very small number of active representations.
Therefore, an efficient cache should rely on the origin server's
instructions rather than assume that different selecting headers
implies a different representation will be selected.  The standard
was crafted to make caching as efficient as possible.  I just wish
it was easier to read.

When it comes to HTTP, Apache leads -- it does not follow blindly,
nor does it wait for spineless browser developers to introduce a
feature first.  We implement it and that act alone is sufficient
to break the chicken-and-egg dilemma that everyone else uses to
justify their lack of priorities.

....Roy

Re: Wrong etag sent with mod_deflate

Posted by Henrik Nordstrom <hn...@squid-cache.org>.
lör 2006-12-09 klockan 15:23 +0100 skrev Justin Erenkrantz:

> See the problem here is that you have to teach ap_meets_conditions()
> about this.  An ETag of "1234-gzip" needs to also satisfy a
> conditional request when the ETag when ap_meets_conditions() is run is
> "1234".  In other words, ap_meets_conditions() also needs to strip
> "-gzip" if it is present before it does the ETag comparison.  But, the
> issue is that there is no real way for us to implement this without a
> butt-ugly hack.

Be careful there.. Blindly stripping the decoration alone won't work
out. Consider for example If-None-Match. In specific If-None-Match with
the ETag of the gzip variant should only return 304 if the request would
cause Apache to send the gzip:ed variant of the entity.

If-None-Match: list of etags

returns 304 with the single correct ETag if any of the ETags in the
directive matches the current response to the current request.


Regards
Henrik

Re: Wrong etag sent with mod_deflate

Posted by Ruediger Pluem <rp...@apache.org>.

On 12/09/2006 03:23 PM, Justin Erenkrantz wrote:
> On 12/9/06, Ruediger Pluem <rp...@apache.org> wrote:

>>
>> Would the following patch address all your points for a CE mod_deflate
>> filter?
> 
> 
> No - this patch breaks conditional GETs which is what I'm against.

Ok, to be honest my question was more directed to Roy than to you, to
understand Roys ideas and plans from a patch level perspective. I
was pretty sure that you would not like it as you have expressed it clearly
before.

> 
> See the problem here is that you have to teach ap_meets_conditions()
> about this.  An ETag of "1234-gzip" needs to also satisfy a
> conditional request when the ETag when ap_meets_conditions() is run is
> "1234".  In other words, ap_meets_conditions() also needs to strip
> "-gzip" if it is present before it does the ETag comparison.  But, the
> issue is that there is no real way for us to implement this without a
> butt-ugly hack.

Thanks for giving the pointer to ap_meets_conditions. So content compressed
by mod_deflate would not stand conditional requests based on ETags any longer.
That would be bad. Would it help if we simply unset the ETag in mod_deflate?
mod_filter does this in these situations or does this have any other nasty
side effects?

So what I understand from the current discussion is that

1. Using TE instead of CE would be RFC compliant and would relief us of
   much problems except the one that none of the major browsers can handle
   it and thus would effectively make mod_deflate useless.

2. There are two different points of view in the CE case:
   Roy and Henrik say that a strong ETag arriving at mod_deflate must
   be replaced with a different strong ETag within mod_deflate (e.g
   by adding -gzip to it), because as mod_deflate is doing CE the entities
   before and after mod_deflate are different and require different ETags.
   Justin OTH says that it is sufficient to convert a strong ETag into a
   weak one, right?

Regards

Rüdiger



Re: Wrong etag sent with mod_deflate

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 12/9/06, Ruediger Pluem <rp...@apache.org> wrote:
> > The existing filter needs to modify the ETag field value (and
> > any other entity-dependent values that we can think of) or be
> > removed as a feature.  Weak etags are not a solution -- being able
> > to make range requests of large cached representations requires a
> > strong etag, and it really isn't hard to provide one.  It is better
> > to not deflate the response at all than to interfere with caching.
>
> Would the following patch address all your points for a CE mod_deflate filter?

No - this patch breaks conditional GETs which is what I'm against.

See the problem here is that you have to teach ap_meets_conditions()
about this.  An ETag of "1234-gzip" needs to also satisfy a
conditional request when the ETag when ap_meets_conditions() is run is
"1234".  In other words, ap_meets_conditions() also needs to strip
"-gzip" if it is present before it does the ETag comparison.  But, the
issue is that there is no real way for us to implement this without a
butt-ugly hack.

However, I disagree with Roy in that we most certainly *do* treat the
ETag values as opaque - Subversion has its own ETag values - Roy's
position only works if you assume the core is assigning the ETag value
which has a set format - not a third-party module.  IMO, any valid
solution that we deploy must work *independently* of what any module
may set ETag to.  It is perfectly valid for a 3rd-party module to
include "-gzip" at the end of their ETag.  For example, if you had a
file called "foo-gzip" in revision 10, SVN would assign the ETag
"10//foo-gzip".  (And, I could construct a conflict where httpd would
hork the ETag incorrectly for any arbitrary value.)  -- justin

Re: Wrong etag sent with mod_deflate

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 12/9/06, Roy T. Fielding <fi...@gbiv.com> wrote:
> The best solution is to not mess with content-encoding at all, which
> gets us out of both this consistency problem and related problems
> with the entity-header fields (content-md5, signatures, etc.).
> That is why transfer encoding was invented in the first place.

We don't live in a world that uses Transfer Encoding for gzip.
Firefox, MSIE, and Opera don't support it.  So, dropping Content
Encoding support in mod_deflate is a non-starter.

> We should have an implementation of deflate as a transfer encoding,
> but it should be configurable independent of the existing filter.
> Some people will want TE specifically to avoid the addition of Vary
> and all the other problems that content-changing filters cause.
> For example, an additional directive option for CE, TE, or "either".

As I said earlier, mod_deflate could respond if TE is sent - there's
no need for a directive here.  And it can sidestep the ETag violation
there.  It's a trivial addition to the current filter of just a few
lines.  And, it gives the one cache in the world that doesn't support
Vary a way out.  So, I feel that this resolves the RFC violation that
Squid sees as long as it sends TE: gzip instead.

> The existing filter needs to modify the ETag field value (and
> any other entity-dependent values that we can think of) or be
> removed as a feature.  Weak etags are not a solution -- being able
> to make range requests of large cached representations requires a
> strong etag, and it really isn't hard to provide one.  It is better
> to not deflate the response at all than to interfere with caching.

As Rudiger's patch shows, removing the ETag or appending junk in
mod_deflate isn't enough - you have to teach ap_meets_conditions() how
to know what it is that it's looking at.  I'm against adding ugly
hacks there that make it only know how to handle "-gzip".
(mod_deflate could in theory very well send "deflate" compression.)
So, any solution within ap_meets_conditions() needs to be generic and
not a one-off just for mod_deflate.

> In any case, I won't accept anyone's votes on this issue until there
> is a patch that can be voted on, and the technical considerations of
> security and correctness take priority over other trade-offs.  RTC.

The patch you have been outlining is straightforward - but ultimately
broken because you haven't sketched a way to handle the
ap_meets_conditions() problem.  I'm merely informing you that I will
veto any approach that breaks conditional GETs with real browsers.  I
couldn't care less what a broken proxy cache does (especially if we
can give it way not to be broken) if it means that mod_deflate no
longer supports browser caches.  -- justin

Re: Wrong etag sent with mod_deflate

Posted by Ruediger Pluem <rp...@apache.org>.

On 12/09/2006 06:52 AM, Roy T. Fielding wrote:

> 
> The best solution is to not mess with content-encoding at all, which
> gets us out of both this consistency problem and related problems
> with the entity-header fields (content-md5, signatures, etc.).
> That is why transfer encoding was invented in the first place.
> 
> We should have an implementation of deflate as a transfer encoding,
> but it should be configurable independent of the existing filter.
> Some people will want TE specifically to avoid the addition of Vary
> and all the other problems that content-changing filters cause.
> For example, an additional directive option for CE, TE, or "either".

I think fixing the current CE filter is easier right now then to
add the option above. I think this can be done in a second step
and sounds like a good idea to me.

> 
> The existing filter needs to modify the ETag field value (and
> any other entity-dependent values that we can think of) or be
> removed as a feature.  Weak etags are not a solution -- being able
> to make range requests of large cached representations requires a
> strong etag, and it really isn't hard to provide one.  It is better
> to not deflate the response at all than to interfere with caching.

Would the following patch address all your points for a CE mod_deflate filter?

Index: modules/filters/mod_deflate.c
===================================================================
--- modules/filters/mod_deflate.c       (Revision 484803)
+++ modules/filters/mod_deflate.c       (Arbeitskopie)
@@ -320,6 +320,7 @@
     if (!ctx) {
         char *token;
         const char *encoding;
+        const char *etag;

         /* only work on main request/no subrequests */
         if (r->main != NULL) {
@@ -483,7 +484,26 @@
         else {
             apr_table_mergen(r->headers_out, "Content-Encoding", "gzip");
         }
+        /*
+         * Unset headers which are no longer valid after we have compressed
+         * the content.
+         */
         apr_table_unset(r->headers_out, "Content-Length");
+        apr_table_unset(r->headers_out, "Content-MD5");
+        /* Adjust ETag if present */
+        etag = apr_table_get(r->headers_out, "ETag");
+        if (etag) {
+            if (*etag) {
+                /* Remove the '"' at the end of the ETag */
+                etag[strlen(etag) - 1] = '\0';
+                apr_table_set(r->headers_out, "ETag",
+                              apr_pstrcat(r->pool, etag, "-gzip\"", NULL));
+            }
+            else {
+                /* Does not seem to be a valid ETag. So remove it. */
+                apr_table_unset(r->headers_out, "ETag");
+            }
+        }

         /* initialize deflate output buffer */
         ctx->stream.next_out = ctx->buffer;

Regards

Rüdiger


Re: Wrong etag sent with mod_deflate

Posted by "Roy T. Fielding" <fi...@gbiv.com>.
On Dec 8, 2006, at 3:35 PM, Justin Erenkrantz wrote:

> On 12/8/06, Roy T. Fielding <fi...@gbiv.com> wrote:
>> What we should be doing is sending transfer-encoding, not content-
>> encoding,
>> and get past the chicken and egg dilemma of that feature in HTTP.
>> If we are changing content-encoding, then we must behave as if there
>> are two different "files" on the server representing the resource.
>> That means tweaking the etag and being prepared to handle that tweak
>> on future conditional requests.
>
> There's just no way to know how to handle any ETag modification on
> future requests.  So, that's a non-starter.  Therefore, any fix for
> this edge case which breaks cacheability in the common case of real
> browsers I would find unacceptable.

It isn't necessary to handle "any" ETag modification -- our ETag
generation is fairly limited and is not opaque to the server.
We only need to avoid conflicts between the content-encoded variant
and the non-encoded variant, which is guaranteed if the encoded
variant has "-gzip" appended to the existing entity-tag.  That will
work fine with the common case of real browsers -- far better than
the current case which will deliver invalid content if a browser
tries to complete a partial download from a cache.

>> In other words, Henrik has it right.  It is our responsibility to
>> assign different etags to different variants because doing otherwise
>> may result in errors on shared caches that use the etag as a variant
>> identifier.
>
> As Kevin mentioned, Squid is only using the ETag and is ignoring the
> Vary header.  That's the crux of the broken behavior on their part.

Then they will still be broken regardless of what we do here.  It simply
isn't a relevant issue.

> If they want to point out minor RFC violations in Apache, then we can
> play that game as well.  (mod_cache deals with this Vary/ETag case
> just fine, FWIW.)

Unlike Squid, RFC compliance is part of our mission, at least when
it isn't due to a bug in the spec.  This is not a bug in the spec.

A high-efficiency response cache is expected to have multiple
representations of a given resource cached.  The cache key is the
URI.  If the set of varying header field values that generated the
cached response is different from the request set, then a
conditional GET request is made containing ALL of the cached
entity tags in an If-None-Match field (in accordance with the Vary
requirements).  If the server says that any one of the representations,
as indicated by the ETag in a 304 response, is okay, then the cached
representation with that entity tag is sent to the user-agent
regardless of the Vary calculation.  In short, if we have two active
representations that have the same etag, then we have violated the
spec and created an unnecessary interoperability problem:

    If the selecting request header fields for the cached entry do not
    match the selecting request header fields of the new request, then
    the cache MUST NOT use a cached entry to satisfy the request unless
    it first relays the new request to the origin server in a  
conditional
    request and the server responds with 304 (Not Modified),  
including an
    entity tag or Content-Location that indicates the entity to be used.

    If an entity tag was assigned to a cached representation, the
    forwarded request SHOULD be conditional and include the entity tags
    in an If-None-Match header field from all its cache entries for the
    resource. This conveys to the server the set of entities currently
    held by the cache, so that if any one of these entities matches the
    requested entity, the server can use the ETag header field in its  
304
    (Not Modified) response to tell the cache which entry is  
appropriate.
    If the entity-tag of the new response matches that of an existing
    entry, the new response SHOULD be used to update the header  
fields of
    the existing entry, and the result MUST be returned to the client.

In other words, the conditional request containing all of the entity
tags satisfies the semantics of Vary when the server responds with
304 and one of those entity tags.

And, no, mod_cache doesn't "deal with it" -- it just isn't a
very efficient cache.

> The compromise I'd be willing to accept is to have mod_deflate support
> the 'TE: gzip' request header and add 'gzip' to the Transfer-Encoding
> bit - and to prefer that over any Accept-Encoding bits that are sent.
> The ETag can clearly remain the same in that case - even as a strong
> ETag.  So, Squid can change to send along TE: gzip (if it isn't
> already).  And, everyone else who sends Accept-Encoding gets the
> result in a way that doesn't pooch their cache if they try to do a
> later conditional request.
>
> Is that acceptable?  -- justin

The best solution is to not mess with content-encoding at all, which
gets us out of both this consistency problem and related problems
with the entity-header fields (content-md5, signatures, etc.).
That is why transfer encoding was invented in the first place.

We should have an implementation of deflate as a transfer encoding,
but it should be configurable independent of the existing filter.
Some people will want TE specifically to avoid the addition of Vary
and all the other problems that content-changing filters cause.
For example, an additional directive option for CE, TE, or "either".

The existing filter needs to modify the ETag field value (and
any other entity-dependent values that we can think of) or be
removed as a feature.  Weak etags are not a solution -- being able
to make range requests of large cached representations requires a
strong etag, and it really isn't hard to provide one.  It is better
to not deflate the response at all than to interfere with caching.

In any case, I won't accept anyone's votes on this issue until there
is a patch that can be voted on, and the technical considerations of
security and correctness take priority over other trade-offs.  RTC.

....Roy

Re: Wrong etag sent with mod_deflate

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 12/8/06, Roy T. Fielding <fi...@gbiv.com> wrote:
> What we should be doing is sending transfer-encoding, not content-
> encoding,
> and get past the chicken and egg dilemma of that feature in HTTP.
> If we are changing content-encoding, then we must behave as if there
> are two different "files" on the server representing the resource.
> That means tweaking the etag and being prepared to handle that tweak
> on future conditional requests.

There's just no way to know how to handle any ETag modification on
future requests.  So, that's a non-starter.  Therefore, any fix for
this edge case which breaks cacheability in the common case of real
browsers I would find unacceptable.

> In other words, Henrik has it right.  It is our responsibility to
> assign different etags to different variants because doing otherwise
> may result in errors on shared caches that use the etag as a variant
> identifier.

As Kevin mentioned, Squid is only using the ETag and is ignoring the
Vary header.  That's the crux of the broken behavior on their part.
If they want to point out minor RFC violations in Apache, then we can
play that game as well.  (mod_cache deals with this Vary/ETag case
just fine, FWIW.)

The compromise I'd be willing to accept is to have mod_deflate support
the 'TE: gzip' request header and add 'gzip' to the Transfer-Encoding
bit - and to prefer that over any Accept-Encoding bits that are sent.
The ETag can clearly remain the same in that case - even as a strong
ETag.  So, Squid can change to send along TE: gzip (if it isn't
already).  And, everyone else who sends Accept-Encoding gets the
result in a way that doesn't pooch their cache if they try to do a
later conditional request.

Is that acceptable?  -- justin

Re: Wrong etag sent with mod_deflate

Posted by Henrik Nordstrom <hn...@squid-cache.org>.
fre 2006-12-08 klockan 11:44 -0800 skrev Roy T. Fielding:

> In other words, Henrik has it right.  It is our responsibility to
> assign different etags to different variants because doing otherwise
> may result in errors on shared caches that use the etag as a variant
> identifier.

Thanks ;-)

Regards
Henrik

Re: Wrong etag sent with mod_deflate

Posted by "Roy T. Fielding" <fi...@gbiv.com>.
Argh, my stupid ISP is losing apache email again because they use  
spamcop.

On Dec 7, 2006, at 2:45 PM, Henrik Nordstrom wrote:
> tor 2006-12-07 klockan 02:42 +0100 skrev Justin Erenkrantz:
>
>> -1 on adding semantic junk to the existing ETag (and keeping it
>> strong); that's blatantly uncool.  Any generated ETag from  
>> mod_deflate
>> should either be the original strong version or a weak version of any
>> previous etag.  mod_deflate by *definition* is just creating a weak
>> version of the prior entity.

No, it is changing the content-encoding value, which is changing the
entity.  The purpose of etag for caching is two-fold: 1) for freshness
checks, and 2) handling conditional range/authoring requests.  That is
why the spec is full of gobbledygook on etag handling -- it was
stretched at the last minute to reuse a very simple freshness check as
a form of variant identifier.

What we should be doing is sending transfer-encoding, not content- 
encoding,
and get past the chicken and egg dilemma of that feature in HTTP.
If we are changing content-encoding, then we must behave as if there
are two different "files" on the server representing the resource.
That means tweaking the etag and being prepared to handle that tweak
on future conditional requests.

In other words, Henrik has it right.  It is our responsibility to
assign different etags to different variants because doing otherwise
may result in errors on shared caches that use the etag as a variant
identifier.

....Roy

Re: Wrong etag sent with mod_deflate

Posted by Henrik Nordstrom <he...@henriknordstrom.net>.
tor 2006-12-07 klockan 02:42 +0100 skrev Justin Erenkrantz:

> -1 on adding semantic junk to the existing ETag (and keeping it
> strong); that's blatantly uncool.  Any generated ETag from mod_deflate
> should either be the original strong version or a weak version of any
> previous etag.  mod_deflate by *definition* is just creating a weak
> version of the prior entity.

You basically only have two choices:

a) Make mod_deflate not send an ETag on modified responses.

b) Modify the value (within the quotes) of the ETag somehow. And if
mod_deflate can not be trusted to always return the same octet
representation make sure to use an weak ETag unless the ETag generation
is also tightly coupled to the octet representation guaranteing a
different ETag should mod_deflate encode slightly different.

And to be fully compliant you also need to pay attention to the
Content-Location header. Here I don't see much choice but to not send
Content-Location in mod_deflate mangled responses (but can be kept on
the original response, no problem there).

RFC 2616 13.6 Caching Negotiated Responses, last paragraph.

> mod_deflate does properly stick in the Vary header, so caches already
> have enough knowledge to know what's going on anyway even without a
> fix.  (This is probably why mod_cache doesn't flag it as an error.)

> My opinion is to fix the protocol and move on...  -- justin

The protocol is quite fine as it is, and not easy to change. As it is
now it's mainly a matter of understanding that mod_deflate does create a
completely new entity from the original one. To the protocol it's
exactly the same as when using mod_negotiate and having both the
identity and gzip encoded entities on disk. The fact that you do this
encoding on the fly is of no concern to HTTP.

Another option is to explore the use gzip transfer encoding instead of
content encodin. In transfer encoding none of these problems apply as
it's done on the transport level and not entity level, but it's not that
well supported in clients unfortunately..

Regards
Henrik

Re: Wrong etag sent with mod_deflate

Posted by Henrik Nordstrom <hn...@squid-cache.org>.
fre 2006-12-08 klockan 14:47 +0100 skrev Justin Erenkrantz:

> mod_deflate is certainly not creating a new resource

It is creating a new HTTP entity. Not a new object on your server, but
still a new unique HTTP entity with different characteristics from the
identity encoding.

If we were talking about transfer-encoding then you would be correct as
it only alter the encoding for transfer purposes and not the HTTP entity
as such, but this is content-encoding. Content encoding is a property of
the response entity.

The main reason why things get blurred is because the creation of this
entity is done on the fly instead of creating a new resource on the
server like HTTP expects. As result you need to be very careful with the
ETag and Content-Location headers.

Not modifying ETag (including just making it weak) says that the
identity and gzip encodings is semantically equivalent, and can be
exchanged freely. In other words says it's fine to send gzip encoding to
all clients (which we all know it's not).

Not modifying/removing Content-Location is less harmful but will cause
cache bouncing, as each time the cache sees a new response entity for a
given URI any older ones with the same Content-Location will get removed
from the cache.

Regards
Henrik

Re: Wrong etag sent with mod_deflate

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 12/8/06, Henrik Nordstrom <hn...@squid-cache.org> wrote:
> The protocol is quite fine as it is, and not easy to change. As it is
> now it's mainly a matter of understanding that mod_deflate does create a
> completely new entity from the original one. To the protocol it's
> exactly the same as when using mod_negotiate and having both the
> identity and gzip encoded entities on disk. The fact that you do this
> encoding on the fly is of no concern to HTTP.

mod_deflate is certainly not creating a new resource - it's modifying
the representation.  There is no legitimate reason for it to modify
the ETag other than to mark it as weak.  That the caching bits in the
RFC didn't understand this speak to the fact that it's quite subtle.
-- justin

Re: Wrong etag sent with mod_deflate

Posted by Henrik Nordstrom <hn...@squid-cache.org>.
tor 2006-12-07 klockan 02:42 +0100 skrev Justin Erenkrantz:

> -1 on adding semantic junk to the existing ETag (and keeping it
> strong); that's blatantly uncool.  Any generated ETag from mod_deflate
> should either be the original strong version or a weak version of any
> previous etag.  mod_deflate by *definition* is just creating a weak
> version of the prior entity.

You basically only have two choices:

a) Make mod_deflate not send an ETag on modified responses.

b) Modify the value (within the quotes) of the ETag somehow. And if
mod_deflate can not be trusted to always return the same octet
representation make sure to use an weak ETag unless the ETag generation
is also tightly coupled to the octet representation guaranteing a
different ETag should mod_deflate encode slightly different.

And to be fully compliant you also need to pay attention to the
Content-Location header. Here I don't see much choice but to not send
Content-Location in mod_deflate mangled responses (but can be kept on
the original response, no problem there).

RFC 2616 13.6 Caching Negotiated Responses, last paragraph.

> mod_deflate does properly stick in the Vary header, so caches already
> have enough knowledge to know what's going on anyway even without a
> fix.  (This is probably why mod_cache doesn't flag it as an error.)

> My opinion is to fix the protocol and move on...  -- justin

The protocol is quite fine as it is, and not easy to change. As it is
now it's mainly a matter of understanding that mod_deflate does create a
completely new entity from the original one. To the protocol it's
exactly the same as when using mod_negotiate and having both the
identity and gzip encoded entities on disk. The fact that you do this
encoding on the fly is of no concern to HTTP.

Another option is to explore the use gzip transfer encoding instead of
content encodin. In transfer encoding none of these problems apply as
it's done on the transport level and not entity level, but it's not that
well supported in clients unfortunately..

Regards
Henrik

Re: Wrong etag sent with mod_deflate

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 12/7/06, Justin Erenkrantz <ju...@erenkrantz.com> wrote:
> mod_deflate should just add the W/ prefix if it's not already there.  -- justin

But, that'll break caches as we're not allowed to serve If-Match with
weak entity tags.  Feh.

-1 on adding semantic junk to the existing ETag (and keeping it
strong); that's blatantly uncool.  Any generated ETag from mod_deflate
should either be the original strong version or a weak version of any
previous etag.  mod_deflate by *definition* is just creating a weak
version of the prior entity.

mod_deflate does properly stick in the Vary header, so caches already
have enough knowledge to know what's going on anyway even without a
fix.  (This is probably why mod_cache doesn't flag it as an error.)

My opinion is to fix the protocol and move on...  -- justin

Re: Wrong etag sent with mod_deflate

Posted by Henrik Nordstrom <he...@henriknordstrom.net>.
fre 2006-12-08 klockan 14:40 +0100 skrev Justin Erenkrantz:

> Uh, no, they *are* semantically equivalent - but, yes, not
> syntactically (bit-for-bit) equivalent.  You inflate the response and
> you get exactly what the ETag originally represented.

To entities is only semantically equivalent if they can be interchanged
freely at the HTTP level with no semantic difference in the end-user
result.

identiy and gzip encoding can not be said to bidirectionally have the
same semantic meaning as a gzip encoded entity is pure rubbish to a
recipient not understanding gzip. No more than a Swedish translation of
a document could be said to be semantically equivalent to a Greek
translation of the same document.

Content-Encoding is a case of unidirectional semantic equivalence where
the identity encoding can be substituted for the gzip encoding with kept
semantics, but for ETag bidirectional semantic equivalence is required
which is not fulfilled as gzip encoding can not be substituted for
identity encoding without risking a significant semantic difference to
the recipient.

The only real difference of a weak etag compared to a strong one is that
the weak one does not guarantee octet equality. All other restrictions
apply. Plus a bunch of protocol restrictions where weak etags is not
allowed to be used.

Regards
Henrik

Re: Wrong etag sent with mod_deflate

Posted by Henrik Nordstrom <hn...@squid-cache.org>.
fre 2006-12-08 klockan 14:40 +0100 skrev Justin Erenkrantz:

> Uh, no, they *are* semantically equivalent - but, yes, not
> syntactically (bit-for-bit) equivalent.  You inflate the response and
> you get exactly what the ETag originally represented.

To entities is only semantically equivalent if they can be interchanged
freely at the HTTP level with no semantic difference in the end-user
result.

identiy and gzip encoding can not be said to bidirectionally have the
same semantic meaning as a gzip encoded entity is pure rubbish to a
recipient not understanding gzip. No more than a Swedish translation of
a document could be said to be semantically equivalent to a Greek
translation of the same document.

Content-Encoding is a case of unidirectional semantic equivalence where
the identity encoding can be substituted for the gzip encoding with kept
semantics, but for ETag bidirectional semantic equivalence is required
which is not fulfilled as gzip encoding can not be substituted for
identity encoding without risking a significant semantic difference to
the recipient.

The only real difference of a weak etag compared to a strong one is that
the weak one does not guarantee octet equality. All other restrictions
apply. Plus a bunch of protocol restrictions where weak etags is not
allowed to be used.

Regards
Henrik

Re: Wrong etag sent with mod_deflate

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 12/8/06, Henrik Nordstrom <hn...@squid-cache.org> wrote:
> No, that won't work. You still be just as non-conforming by doing that.
> But if mod_deflate may to produce different octet-level results on
> different requests for the same original entity then it must do this in
> addition to other transforms of the ETag.
>
> The identity and gzip encodings is not bidirectionally semantically
> equivalent, and additionally normal conditional comparing W/"X" to "X"
> is true.

Uh, no, they *are* semantically equivalent - but, yes, not
syntactically (bit-for-bit) equivalent.  You inflate the response and
you get exactly what the ETag originally represented.

> See RFC 2616 3.3.3 Weak and Strong Validators
>
> You must make the value of the ETag differ between the two entities.

mod_deflate is clearly only doing a semantic (weak) transformation.  -- justin

Re: Wrong etag sent with mod_deflate

Posted by Henrik Nordstrom <hn...@squid-cache.org>.
tor 2006-12-07 klockan 02:31 +0100 skrev Justin Erenkrantz:

> mod_deflate should just add the W/ prefix if it's not already there.  -- justin

No, that won't work. You still be just as non-conforming by doing that.
But if mod_deflate may to produce different octet-level results on
different requests for the same original entity then it must do this in
addition to other transforms of the ETag.

The identity and gzip encodings is not bidirectionally semantically
equivalent, and additionally normal conditional comparing W/"X" to "X"
is true.

See RFC 2616 3.3.3 Weak and Strong Validators

You must make the value of the ETag differ between the two entities.

Regards
Henrik

Re: Wrong etag sent with mod_deflate

Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 12/7/06, Roy T. Fielding <fi...@gbiv.com> wrote:
> Entities gzip:ed by mod_deflate still carries the same ETag as the
> plain entiy,
> causing inconsistency in ETag aware proxy caches.
>
> I'll have a look later and see if I can fix it, but let me know if there
> is already a patch in the works (that doesn't rely on mod_filter).

mod_deflate should just add the W/ prefix if it's not already there.  -- justin

Re: Wrong etag sent with mod_deflate

Posted by Chris Elving <Ch...@Sun.COM>.
Roy T. Fielding wrote:

> Protocol issues really should be brought up on the dev list, with an
> appropriate subject, and not left in bugzilla.

FWIW, there was a dev list thread on this 3 years ago with the subject 
"mod_deflate and transfer / content encoding problem".

http://www.mail-archive.com/dev@httpd.apache.org/msg18366.html