You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Stefan Eissing <st...@greenbytes.de> on 2019/04/09 10:30:56 UTC
keep-alive and vary in 304 responses
I just did some tests with https://redbot.org/ (the site tester by Mark Nottingham) against our server and it notifies of 2 things:
1. The "Keep-Alive" header is deprecated. I tried to "Header unset Keep-Alive" but that has no effect. Seems to be added very late.
Do we have a way to suppress it?
2. Validation responses lose the "Vary" header from the unconditional response. This happens on resources where mod_deflate is active.
The 200 response without any "if-modified-since" etc. carries "Vary: Accept-Encoding" as it should.
The 304 response with "if-modified-since" has no "Vary" header.
The code in mod_deflate looks as if it tries to add it in any case, where is it lost?
Help in any of the two issues above is appreciated!
-Stefan
Re: keep-alive and vary in 304 responses
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 09.04.2019 um 16:03 schrieb Stefan Eissing <st...@greenbytes.de>:
>
>
>
>> Am 09.04.2019 um 15:41 schrieb Stefan Eissing <st...@greenbytes.de>:
>>
>>>
>>> Am 09.04.2019 um 13:36 schrieb Stefan Eissing <st...@greenbytes.de>:
>>>
>>>
>>>
>>>> Am 09.04.2019 um 13:27 schrieb Eric Covener <co...@gmail.com>:
>>>>
>>>> On Tue, Apr 9, 2019 at 6:31 AM Stefan Eissing
>>>> <st...@greenbytes.de> wrote:
>>>>>
>>>>> I just did some tests with https://redbot.org/ (the site tester by Mark Nottingham) against our server and it notifies of 2 things:
>>>>>
>>>>> 1. The "Keep-Alive" header is deprecated. I tried to "Header unset Keep-Alive" but that has no effect. Seems to be added very late.
>>>>> Do we have a way to suppress it?
>>>>
>>>> Doesn't look like it, maybe a good "help wanted" kind of task if
>>>> anyone is lurking.
>>>
>>> We rarely ever eat volunteers!
>>>
>>>>> 2. Validation responses lose the "Vary" header from the unconditional response. This happens on resources where mod_deflate is active.
>>>>> The 200 response without any "if-modified-since" etc. carries "Vary: Accept-Encoding" as it should.
>>>>> The 304 response with "if-modified-since" has no "Vary" header.
>>>>> The code in mod_deflate looks as if it tries to add it in any case, where is it lost?
>>>>
>>>> bailing here, seems harmless if we add Vary to a 304 even when we
>>>> don't know the original was long enough?
>>>>
>>>> [Tue Apr 09 07:26:12.183430 2019] [deflate:trace1] [pid 37010:tid
>>>> 123145306648576] mod_deflate.c(590): [client 127.0.0.1:64941] Not
>>>> compressing very small response of 0 bytes
>>>
>>> Ah, nice!
>>>
>>> Maybe if AP_STATUS_IS_HEADER_ONLY(r), the length check should not run?
>>
>> Meh!
>>
>>> curl -D xxx -H 'Accept-Encoding: gzip' https://eissing.org
>> HTTP/2 200
>> date: Tue, 09 Apr 2019 13:36:10 GMT
>> server: Apache/2.4.39 (Ubuntu)
>> strict-transport-security: max-age=15768000
>> last-modified: Thu, 10 Aug 2017 11:16:47 GMT
>> etag: "9a6-55664550a25c0-gzip"
>> accept-ranges: bytes
>> cache-control: max-age=3600
>> vary: Accept-Encoding
>> content-encoding: gzip
>> content-length: 1015
>> content-type: text/html
>>
>>> curl -D xxx -H 'Accept-Encoding: gzip' -H 'If-Modified-Since: Thu, 10 Aug 2017 11:16:47 GMT' https://eissing.org
>> HTTP/2 304
>> date: Tue, 09 Apr 2019 13:36:35 GMT
>> server: Apache/2.4.39 (Ubuntu)
>> etag: "9a6-55664550a25c0"
>> cache-control: max-age=3600
>>
>> Same for HTTP/1.1 requests. And same for https://httpd.apache.org
>>
>> The output filters are not in play at all. Was it always like this?
>
> Update: its the short-hand header list that gets written in case of 304 in http_filters.c:1429 pp
>
> Continuing to investigate what is happening here...
Finally, not being the sharpest today, there is - of course - http_protocol.c:1248 which removes all resource filters for "error" responses.
Hmmpf. Makes sense as it keeps complexity from filters. The price being that we allow only a very limited set as response headers. Because all else could be altered by the filters we did not run. Which loses the "Vary" set by resource filters.
Hmm, something to chew on...
-Stefan
Re: keep-alive and vary in 304 responses
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 09.04.2019 um 15:41 schrieb Stefan Eissing <st...@greenbytes.de>:
>
>>
>> Am 09.04.2019 um 13:36 schrieb Stefan Eissing <st...@greenbytes.de>:
>>
>>
>>
>>> Am 09.04.2019 um 13:27 schrieb Eric Covener <co...@gmail.com>:
>>>
>>> On Tue, Apr 9, 2019 at 6:31 AM Stefan Eissing
>>> <st...@greenbytes.de> wrote:
>>>>
>>>> I just did some tests with https://redbot.org/ (the site tester by Mark Nottingham) against our server and it notifies of 2 things:
>>>>
>>>> 1. The "Keep-Alive" header is deprecated. I tried to "Header unset Keep-Alive" but that has no effect. Seems to be added very late.
>>>> Do we have a way to suppress it?
>>>
>>> Doesn't look like it, maybe a good "help wanted" kind of task if
>>> anyone is lurking.
>>
>> We rarely ever eat volunteers!
>>
>>>> 2. Validation responses lose the "Vary" header from the unconditional response. This happens on resources where mod_deflate is active.
>>>> The 200 response without any "if-modified-since" etc. carries "Vary: Accept-Encoding" as it should.
>>>> The 304 response with "if-modified-since" has no "Vary" header.
>>>> The code in mod_deflate looks as if it tries to add it in any case, where is it lost?
>>>
>>> bailing here, seems harmless if we add Vary to a 304 even when we
>>> don't know the original was long enough?
>>>
>>> [Tue Apr 09 07:26:12.183430 2019] [deflate:trace1] [pid 37010:tid
>>> 123145306648576] mod_deflate.c(590): [client 127.0.0.1:64941] Not
>>> compressing very small response of 0 bytes
>>
>> Ah, nice!
>>
>> Maybe if AP_STATUS_IS_HEADER_ONLY(r), the length check should not run?
>
> Meh!
>
>> curl -D xxx -H 'Accept-Encoding: gzip' https://eissing.org
> HTTP/2 200
> date: Tue, 09 Apr 2019 13:36:10 GMT
> server: Apache/2.4.39 (Ubuntu)
> strict-transport-security: max-age=15768000
> last-modified: Thu, 10 Aug 2017 11:16:47 GMT
> etag: "9a6-55664550a25c0-gzip"
> accept-ranges: bytes
> cache-control: max-age=3600
> vary: Accept-Encoding
> content-encoding: gzip
> content-length: 1015
> content-type: text/html
>
>> curl -D xxx -H 'Accept-Encoding: gzip' -H 'If-Modified-Since: Thu, 10 Aug 2017 11:16:47 GMT' https://eissing.org
> HTTP/2 304
> date: Tue, 09 Apr 2019 13:36:35 GMT
> server: Apache/2.4.39 (Ubuntu)
> etag: "9a6-55664550a25c0"
> cache-control: max-age=3600
>
> Same for HTTP/1.1 requests. And same for https://httpd.apache.org
>
> The output filters are not in play at all. Was it always like this?
Update: its the short-hand header list that gets written in case of 304 in http_filters.c:1429 pp
Continuing to investigate what is happening here...
Re: keep-alive and vary in 304 responses
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 09.04.2019 um 13:36 schrieb Stefan Eissing <st...@greenbytes.de>:
>
>
>
>> Am 09.04.2019 um 13:27 schrieb Eric Covener <co...@gmail.com>:
>>
>> On Tue, Apr 9, 2019 at 6:31 AM Stefan Eissing
>> <st...@greenbytes.de> wrote:
>>>
>>> I just did some tests with https://redbot.org/ (the site tester by Mark Nottingham) against our server and it notifies of 2 things:
>>>
>>> 1. The "Keep-Alive" header is deprecated. I tried to "Header unset Keep-Alive" but that has no effect. Seems to be added very late.
>>> Do we have a way to suppress it?
>>
>> Doesn't look like it, maybe a good "help wanted" kind of task if
>> anyone is lurking.
>
> We rarely ever eat volunteers!
>
>>> 2. Validation responses lose the "Vary" header from the unconditional response. This happens on resources where mod_deflate is active.
>>> The 200 response without any "if-modified-since" etc. carries "Vary: Accept-Encoding" as it should.
>>> The 304 response with "if-modified-since" has no "Vary" header.
>>> The code in mod_deflate looks as if it tries to add it in any case, where is it lost?
>>
>> bailing here, seems harmless if we add Vary to a 304 even when we
>> don't know the original was long enough?
>>
>> [Tue Apr 09 07:26:12.183430 2019] [deflate:trace1] [pid 37010:tid
>> 123145306648576] mod_deflate.c(590): [client 127.0.0.1:64941] Not
>> compressing very small response of 0 bytes
>
> Ah, nice!
>
> Maybe if AP_STATUS_IS_HEADER_ONLY(r), the length check should not run?
Meh!
> curl -D xxx -H 'Accept-Encoding: gzip' https://eissing.org
HTTP/2 200
date: Tue, 09 Apr 2019 13:36:10 GMT
server: Apache/2.4.39 (Ubuntu)
strict-transport-security: max-age=15768000
last-modified: Thu, 10 Aug 2017 11:16:47 GMT
etag: "9a6-55664550a25c0-gzip"
accept-ranges: bytes
cache-control: max-age=3600
vary: Accept-Encoding
content-encoding: gzip
content-length: 1015
content-type: text/html
> curl -D xxx -H 'Accept-Encoding: gzip' -H 'If-Modified-Since: Thu, 10 Aug 2017 11:16:47 GMT' https://eissing.org
HTTP/2 304
date: Tue, 09 Apr 2019 13:36:35 GMT
server: Apache/2.4.39 (Ubuntu)
etag: "9a6-55664550a25c0"
cache-control: max-age=3600
Same for HTTP/1.1 requests. And same for https://httpd.apache.org
The output filters are not in play at all. Was it always like this?
-Stefan *scratches his head*
Re: keep-alive and vary in 304 responses
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 09.04.2019 um 13:27 schrieb Eric Covener <co...@gmail.com>:
>
> On Tue, Apr 9, 2019 at 6:31 AM Stefan Eissing
> <st...@greenbytes.de> wrote:
>>
>> I just did some tests with https://redbot.org/ (the site tester by Mark Nottingham) against our server and it notifies of 2 things:
>>
>> 1. The "Keep-Alive" header is deprecated. I tried to "Header unset Keep-Alive" but that has no effect. Seems to be added very late.
>> Do we have a way to suppress it?
>
> Doesn't look like it, maybe a good "help wanted" kind of task if
> anyone is lurking.
We rarely ever eat volunteers!
>> 2. Validation responses lose the "Vary" header from the unconditional response. This happens on resources where mod_deflate is active.
>> The 200 response without any "if-modified-since" etc. carries "Vary: Accept-Encoding" as it should.
>> The 304 response with "if-modified-since" has no "Vary" header.
>> The code in mod_deflate looks as if it tries to add it in any case, where is it lost?
>
> bailing here, seems harmless if we add Vary to a 304 even when we
> don't know the original was long enough?
>
> [Tue Apr 09 07:26:12.183430 2019] [deflate:trace1] [pid 37010:tid
> 123145306648576] mod_deflate.c(590): [client 127.0.0.1:64941] Not
> compressing very small response of 0 bytes
Ah, nice!
Maybe if AP_STATUS_IS_HEADER_ONLY(r), the length check should not run?
> --
> Eric Covener
> covener@gmail.com
Re: keep-alive and vary in 304 responses
Posted by Eric Covener <co...@gmail.com>.
On Tue, Apr 9, 2019 at 6:31 AM Stefan Eissing
<st...@greenbytes.de> wrote:
>
> I just did some tests with https://redbot.org/ (the site tester by Mark Nottingham) against our server and it notifies of 2 things:
>
> 1. The "Keep-Alive" header is deprecated. I tried to "Header unset Keep-Alive" but that has no effect. Seems to be added very late.
> Do we have a way to suppress it?
Doesn't look like it, maybe a good "help wanted" kind of task if
anyone is lurking.
>
> 2. Validation responses lose the "Vary" header from the unconditional response. This happens on resources where mod_deflate is active.
> The 200 response without any "if-modified-since" etc. carries "Vary: Accept-Encoding" as it should.
> The 304 response with "if-modified-since" has no "Vary" header.
> The code in mod_deflate looks as if it tries to add it in any case, where is it lost?
bailing here, seems harmless if we add Vary to a 304 even when we
don't know the original was long enough?
[Tue Apr 09 07:26:12.183430 2019] [deflate:trace1] [pid 37010:tid
123145306648576] mod_deflate.c(590): [client 127.0.0.1:64941] Not
compressing very small response of 0 bytes
--
Eric Covener
covener@gmail.com
Re: keep-alive and vary in 304 responses
Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, Apr 10, 2019 at 11:49 AM Julian Reschke <ju...@gmx.de> wrote:
>
> On 10.04.2019 09:24, Mario Brandt wrote:
> > On Tue, 9 Apr 2019 at 12:31, Stefan Eissing
> > <st...@greenbytes.de> wrote:
> >>
> >> I just did some tests with https://redbot.org/ (the site tester by Mark Nottingham) against our server and it notifies of 2 things:
> >>
> >> 1. The "Keep-Alive" header is deprecated. I tried to "Header unset Keep-Alive" but that has no effect. Seems to be added very late.
> >> Do we have a way to suppress it?
> >
> > That is true for HTTP/2. What about the clients that don't support
> > HTTP/2 yet? I wonder if it is possible to suppress that header only if
> > HTTP2 or better is used.
>
> Keep-Alive is defined in RFC 2068 and was removed in RFC 2616 (almost 20
> years ago).
Still it can be usefull in HTTP/1 as Connection/hop-by-hop header, for
instance the timeout= value can help synchronise a gateway and an
origin to avoid reusing a closed connection.
Can hop-by-hop headers be deprecated since they are not specifically
defined/listed by the RFC (7230)? The Keep-Alive header can possibly
live as a negotiated "application" header..
Regards,
Yann.
Re: keep-alive and vary in 304 responses
Posted by Julian Reschke <ju...@gmx.de>.
On 10.04.2019 09:24, Mario Brandt wrote:
> On Tue, 9 Apr 2019 at 12:31, Stefan Eissing
> <st...@greenbytes.de> wrote:
>>
>> I just did some tests with https://redbot.org/ (the site tester by Mark Nottingham) against our server and it notifies of 2 things:
>>
>> 1. The "Keep-Alive" header is deprecated. I tried to "Header unset Keep-Alive" but that has no effect. Seems to be added very late.
>> Do we have a way to suppress it?
>
> That is true for HTTP/2. What about the clients that don't support
> HTTP/2 yet? I wonder if it is possible to suppress that header only if
> HTTP2 or better is used.
Keep-Alive is defined in RFC 2068 and was removed in RFC 2616 (almost 20
years ago).
Best regards, Julian
Re: keep-alive and vary in 304 responses
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 10.04.2019 um 09:24 schrieb Mario Brandt <jb...@gmail.com>:
>
> On Tue, 9 Apr 2019 at 12:31, Stefan Eissing
> <st...@greenbytes.de> wrote:
>>
>> I just did some tests with https://redbot.org/ (the site tester by Mark Nottingham) against our server and it notifies of 2 things:
>>
>> 1. The "Keep-Alive" header is deprecated. I tried to "Header unset Keep-Alive" but that has no effect. Seems to be added very late.
>> Do we have a way to suppress it?
>
> That is true for HTTP/2. What about the clients that don't support
> HTTP/2 yet? I wonder if it is possible to suppress that header only if
> HTTP2 or better is used.
Our HTTP/2 implementation does indeed suppress it as it does not honor the *KeepAlive* settings that apply to HTTP/1.1
As Apache will only generate "Keep-Alive" response headers if the client included a "Connection: keep-alive" in its request, I now regard this as a non-issue.
Cheers, Stefan
Re: keep-alive and vary in 304 responses
Posted by Mario Brandt <jb...@gmail.com>.
On Tue, 9 Apr 2019 at 12:31, Stefan Eissing
<st...@greenbytes.de> wrote:
>
> I just did some tests with https://redbot.org/ (the site tester by Mark Nottingham) against our server and it notifies of 2 things:
>
> 1. The "Keep-Alive" header is deprecated. I tried to "Header unset Keep-Alive" but that has no effect. Seems to be added very late.
> Do we have a way to suppress it?
That is true for HTTP/2. What about the clients that don't support
HTTP/2 yet? I wonder if it is possible to suppress that header only if
HTTP2 or better is used.
Mario
Re: keep-alive and vary in 304 responses
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 10.04.2019 um 12:48 schrieb Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com>:
>
>
>
>
> C2 General
>
>> -----Ursprüngliche Nachricht-----
>> Von: Stefan Eissing <st...@greenbytes.de>
>> Gesendet: Mittwoch, 10. April 2019 12:10
>> An: dev@httpd.apache.org
>> Betreff: Re: keep-alive and vary in 304 responses
>>
>>
>>> Am 09.04.2019 um 18:48 schrieb Roy T. Fielding <fi...@gbiv.com>:
>>>
>>>>
>>>> 2. Validation responses lose the "Vary" header from the unconditional
>> response. This happens on resources where mod_deflate is active.
>>>> The 200 response without any "if-modified-since" etc. carries "Vary:
>> Accept-Encoding" as it should.
>>>> The 304 response with "if-modified-since" has no "Vary" header.
>>>> The code in mod_deflate looks as if it tries to add it in any case,
>> where is it lost?
>>>
>>> That's one of many bugs related to design problems with mod_deflate.
>> Basically,
>>> it chooses to insert itself into the response handling after the other
>> checks are done,
>>> which means a 304 doesn't even know yet whether the response would use
>> gzip.
>>> The solution (which I never found time to implement) is to replace
>> dynamic gzip with
>>> a transfer encoding generator, and/or to rewrite the mod_deflate CE
>> encoder to act more
>>> like mod_cache.
>>
>> Thinking out loud here:
>>
>> Acting more like mod_cache would then split the deflate output filter
>> into a resource and a protocol type one, I assume?
>
> IMHO the "secret" of mod_cache is that its filter is also inserted via the
> insert_error_filter hook and not just via the insert_filter hook (like mod_expires and mod_header do).
Thanks for the pointer. The first lines of this actually show:
/* ignore everything except for 5xx errors */
if (r->status < HTTP_INTERNAL_SERVER_ERROR) {
return;
}
So, this is not applied on 304s. And 304 is the only non-200 status that
content-encoding filters are really interested in, I think.
> With mod_deflate this will be more complex as mod_deflate does not add itself.
> It is either added by the core via SetOuputFilter or via mod_filter and they
> all only consider insert_filter and not insert_error_filter. We probably need
> a possibility to tell filters to be added in insert_error_filter.
Could not a deflate *protocol* filter always insert itself and check if
its resource counterpart is present in r->output_filters? This would leave
the application of resource filters unchanged.
Only in error responses, e.g. 304, would the protocol deflate filter not
find its buddy. If the r->output_filter got preserved at the request, it could.
>
> Regards
>
> Rüdiger
AW: keep-alive and vary in 304 responses
Posted by Plüm,
Rüdiger,
Vodafone Group <ru...@vodafone.com>.
C2 General
> -----Ursprüngliche Nachricht-----
> Von: Stefan Eissing <st...@greenbytes.de>
> Gesendet: Mittwoch, 10. April 2019 12:10
> An: dev@httpd.apache.org
> Betreff: Re: keep-alive and vary in 304 responses
>
>
> > Am 09.04.2019 um 18:48 schrieb Roy T. Fielding <fi...@gbiv.com>:
> >
> >>
> >> 2. Validation responses lose the "Vary" header from the unconditional
> response. This happens on resources where mod_deflate is active.
> >> The 200 response without any "if-modified-since" etc. carries "Vary:
> Accept-Encoding" as it should.
> >> The 304 response with "if-modified-since" has no "Vary" header.
> >> The code in mod_deflate looks as if it tries to add it in any case,
> where is it lost?
> >
> > That's one of many bugs related to design problems with mod_deflate.
> Basically,
> > it chooses to insert itself into the response handling after the other
> checks are done,
> > which means a 304 doesn't even know yet whether the response would use
> gzip.
> > The solution (which I never found time to implement) is to replace
> dynamic gzip with
> > a transfer encoding generator, and/or to rewrite the mod_deflate CE
> encoder to act more
> > like mod_cache.
>
> Thinking out loud here:
>
> Acting more like mod_cache would then split the deflate output filter
> into a resource and a protocol type one, I assume?
IMHO the "secret" of mod_cache is that its filter is also inserted via the
insert_error_filter hook and not just via the insert_filter hook (like mod_expires and mod_header do).
With mod_deflate this will be more complex as mod_deflate does not add itself.
It is either added by the core via SetOuputFilter or via mod_filter and they
all only consider insert_filter and not insert_error_filter. We probably need
a possibility to tell filters to be added in insert_error_filter.
Regards
Rüdiger
Re: keep-alive and vary in 304 responses
Posted by "Roy T. Fielding" <fi...@gbiv.com>.
> On Apr 10, 2019, at 3:10 AM, Stefan Eissing <st...@greenbytes.de> wrote:
>
>
>> Am 09.04.2019 um 18:48 schrieb Roy T. Fielding <fi...@gbiv.com>:
>>
>>>
>>> 2. Validation responses lose the "Vary" header from the unconditional response. This happens on resources where mod_deflate is active.
>>> The 200 response without any "if-modified-since" etc. carries "Vary: Accept-Encoding" as it should.
>>> The 304 response with "if-modified-since" has no "Vary" header.
>>> The code in mod_deflate looks as if it tries to add it in any case, where is it lost?
>>
>> That's one of many bugs related to design problems with mod_deflate. Basically,
>> it chooses to insert itself into the response handling after the other checks are done,
>> which means a 304 doesn't even know yet whether the response would use gzip.
>> The solution (which I never found time to implement) is to replace dynamic gzip with
>> a transfer encoding generator, and/or to rewrite the mod_deflate CE encoder to act more
>> like mod_cache.
>
> Thinking out loud here:
>
> Acting more like mod_cache would then split the deflate output filter into a resource and a protocol type one, I assume?
Yeah, probably. What I meant was that deflate would act like a proxy to the resource
backend, provide variants transparently regardless of the client's request, and maintain
a cache of (at least) the metadata for available variants. That would allow the metadata
to be consistent over time, and possibly allow the dynamically compressed variants
to also be cached for later use.
It's also reasonable (though perhaps less cache efficient) to add "Vary: Accept-Encoding"
on any response that has a chance of being dynamically compressed even if we decide
that *this* response will not be.
....Roy
Re: keep-alive and vary in 304 responses
Posted by Julian Reschke <ju...@gmx.de>.
On 10.04.2019 12:49, Yann Ylavic wrote:
> On Wed, Apr 10, 2019 at 12:10 PM Stefan Eissing
> <st...@greenbytes.de> wrote:
>>
>>> Am 09.04.2019 um 18:48 schrieb Roy T. Fielding <fi...@gbiv.com>:
>>>>
>>>> 2. Validation responses lose the "Vary" header from the unconditional response. This happens on resources where mod_deflate is active.
>>>> The 200 response without any "if-modified-since" etc. carries "Vary: Accept-Encoding" as it should.
>>>> The 304 response with "if-modified-since" has no "Vary" header.
>>>> The code in mod_deflate looks as if it tries to add it in any case, where is it lost?
>>>
>>> That's one of many bugs related to design problems with mod_deflate. Basically,
>>> it chooses to insert itself into the response handling after the other checks are done,
>>> which means a 304 doesn't even know yet whether the response would use gzip.
>>> The solution (which I never found time to implement) is to replace dynamic gzip with
>>> a transfer encoding generator,
>
> Do user-agents support "Transfer-Encoding: gzip, chunked" currently?
> That'd be the best/easier solution I think.
> ...
AFAIR, the only transfer encoding supported by common UAs is "chunked".
Best regards, Julian
Re: keep-alive and vary in 304 responses
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 10.04.2019 um 12:53 schrieb Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com>:
>
>
>
>
> C2 General
>
>> -----Ursprüngliche Nachricht-----
>> Von: Yann Ylavic <yl...@gmail.com>
>> Gesendet: Mittwoch, 10. April 2019 12:49
>> An: httpd-dev <de...@httpd.apache.org>
>> Betreff: Re: keep-alive and vary in 304 responses
>>
>> On Wed, Apr 10, 2019 at 12:10 PM Stefan Eissing
>> <st...@greenbytes.de> wrote:
>>>
>>>> Am 09.04.2019 um 18:48 schrieb Roy T. Fielding <fi...@gbiv.com>:
>>>>>
>>>>> 2. Validation responses lose the "Vary" header from the
>> unconditional response. This happens on resources where mod_deflate is
>> active.
>>>>> The 200 response without any "if-modified-since" etc. carries
>> "Vary: Accept-Encoding" as it should.
>>>>> The 304 response with "if-modified-since" has no "Vary" header.
>>>>> The code in mod_deflate looks as if it tries to add it in any
>> case, where is it lost?
>>>>
>>>> That's one of many bugs related to design problems with mod_deflate.
>> Basically,
>>>> it chooses to insert itself into the response handling after the
>> other checks are done,
>>>> which means a 304 doesn't even know yet whether the response would
>> use gzip.
>>>> The solution (which I never found time to implement) is to replace
>> dynamic gzip with
>>>> a transfer encoding generator,
>>
>> Do user-agents support "Transfer-Encoding: gzip, chunked" currently?
>> That'd be the best/easier solution I think.
>
> But TE is only hop-by-hop isn't it?. This would mean no e2e compression?
> Also we need to keep in mind similar issues for similar filters e.g. mod_brotli
+1
> Regards
>
> Rüdiger
Re: keep-alive and vary in 304 responses
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 10.04.2019 um 16:34 schrieb Julian Reschke <ju...@gmx.de>:
>
> On 10.04.2019 16:10, Stefan Eissing wrote:
>>
>>
>>> Am 10.04.2019 um 15:57 schrieb Julian Reschke <ju...@gmx.de>:
>>>
>>> On 10.04.2019 14:53, Plüm, Rüdiger, Vodafone Group wrote:
>>>> ...
>>>> Not sure about this. I guess with TE each hop could be different in what it accepts
>>>> and generates. This is different from CE. As far as I understand the accept-encoding header
>>>> is only for CE not for TE.
>>>> ...
>>>
>>> Right (that would be "TE").
>>>
>>> That aside, maybe it's time to remind that HTTP/2 doesn't have transfer
>>> codings?
>>
>> You mean, it does not have "chunked".
>
> Not my reading...:
>
> "The only exception to this is the TE header field, which MAY be present
> in an HTTP/2 request; when it is, it MUST NOT contain any value other
> than "trailers"."
>
> <https://greenbytes.de/tech/webdav/rfc7540.html#rfc.section.8.1.2.2.p.2>
Yeah, your reading is correct. There seems to be no wriggle room about that one.
This discourages efforts to extend transfer-encodings for HTTP/1.1. Using HTTP/2 to the client in a reverse proxy is quite common. Those proxies could pass transfer-encoded, gzipped resources only by uncompressing them. Bleh!
I think I will make an attempt at improving mod_deflate with a second filter and see how complicated that gets. Alternate ideas always welcome. But content-encoding it seems to be.
-Stefan
PS. The "keep-alive" warning indicator at redbot.org for testing an Apache server is gone. Mark agreed to remove the "Connection: keep-alive" request header, so Apache will not generate one in response. All's well.
Re: keep-alive and vary in 304 responses
Posted by Julian Reschke <ju...@gmx.de>.
On 10.04.2019 16:10, Stefan Eissing wrote:
>
>
>> Am 10.04.2019 um 15:57 schrieb Julian Reschke <ju...@gmx.de>:
>>
>> On 10.04.2019 14:53, Plüm, Rüdiger, Vodafone Group wrote:
>>> ...
>>> Not sure about this. I guess with TE each hop could be different in what it accepts
>>> and generates. This is different from CE. As far as I understand the accept-encoding header
>>> is only for CE not for TE.
>>> ...
>>
>> Right (that would be "TE").
>>
>> That aside, maybe it's time to remind that HTTP/2 doesn't have transfer
>> codings?
>
> You mean, it does not have "chunked".
Not my reading...:
"The only exception to this is the TE header field, which MAY be present
in an HTTP/2 request; when it is, it MUST NOT contain any value other
than "trailers"."
<https://greenbytes.de/tech/webdav/rfc7540.html#rfc.section.8.1.2.2.p.2>
Re: keep-alive and vary in 304 responses
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 10.04.2019 um 15:57 schrieb Julian Reschke <ju...@gmx.de>:
>
> On 10.04.2019 14:53, Plüm, Rüdiger, Vodafone Group wrote:
>> ...
>> Not sure about this. I guess with TE each hop could be different in what it accepts
>> and generates. This is different from CE. As far as I understand the accept-encoding header
>> is only for CE not for TE.
>> ...
>
> Right (that would be "TE").
>
> That aside, maybe it's time to remind that HTTP/2 doesn't have transfer
> codings?
You mean, it does not have "chunked".
> Best regards, Julian
Re: AW: keep-alive and vary in 304 responses
Posted by Julian Reschke <ju...@gmx.de>.
On 10.04.2019 14:53, Plüm, Rüdiger, Vodafone Group wrote:
> ...
> Not sure about this. I guess with TE each hop could be different in what it accepts
> and generates. This is different from CE. As far as I understand the accept-encoding header
> is only for CE not for TE.
> ...
Right (that would be "TE").
That aside, maybe it's time to remind that HTTP/2 doesn't have transfer
codings?
Best regards, Julian
AW: keep-alive and vary in 304 responses
Posted by Plüm,
Rüdiger,
Vodafone Group <ru...@vodafone.com>.
C2 General
> -----Ursprüngliche Nachricht-----
> Von: Stefan Eissing <st...@greenbytes.de>
> Gesendet: Mittwoch, 10. April 2019 14:40
> An: dev@httpd.apache.org
> Betreff: Re: keep-alive and vary in 304 responses
>
>
>
> > Am 10.04.2019 um 14:07 schrieb Plüm, Rüdiger, Vodafone Group
> <ru...@vodafone.com>:
> >
> >
> >
> >
> > C2 General
> >
> >> -----Ursprüngliche Nachricht-----
> >> Von: Yann Ylavic <yl...@gmail.com>
> >> Gesendet: Mittwoch, 10. April 2019 14:04
> >> An: httpd-dev <de...@httpd.apache.org>
> >> Betreff: Re: keep-alive and vary in 304 responses
> >>
> >> On Wed, Apr 10, 2019 at 1:03 PM Plüm, Rüdiger, Vodafone Group
> >> <ru...@vodafone.com> wrote:
> >>>
> >>>> -----Ursprüngliche Nachricht-----
> >>>> Von: Yann Ylavic <yl...@gmail.com>
> >>>> Gesendet: Mittwoch, 10. April 2019 12:49
> >>>>
> >>>> Do user-agents support "Transfer-Encoding: gzip, chunked"
> currently?
> >>>> That'd be the best/easier solution I think.
> >>>
> >>> But TE is only hop-by-hop isn't it?. This would mean no e2e
> >> compression?
> >>
> >> A proxy can still forward hop-by-hop things, e.g. mod_proxy forwards
> >> "chunked" encoding if no previous handler/input-filter reads the
> body.
> >
> > My understanding was that mod_proxy "dechunks" the body and "chunks"
> it again.
> > It does not pass it transparently.
> > This would be a resource intensive job with compression.
>
> I believe that a proxy could de-chunk without removing the gzip.
Not sure about this. I guess with TE each hop could be different in what it accepts
and generates. This is different from CE. As far as I understand the accept-encoding header
is only for CE not for TE.
> But how would our output filter infrastructure treat that? mod_deflate
> stays inactive when faced with an existing Content-Encoding, but does
> not
> watch out for Transfer-Encodings already applied. No resource type
> output
> filter does, right?
Right.
Regards
Rüdiger
Re: keep-alive and vary in 304 responses
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 10.04.2019 um 14:07 schrieb Plüm, Rüdiger, Vodafone Group <ru...@vodafone.com>:
>
>
>
>
> C2 General
>
>> -----Ursprüngliche Nachricht-----
>> Von: Yann Ylavic <yl...@gmail.com>
>> Gesendet: Mittwoch, 10. April 2019 14:04
>> An: httpd-dev <de...@httpd.apache.org>
>> Betreff: Re: keep-alive and vary in 304 responses
>>
>> On Wed, Apr 10, 2019 at 1:03 PM Plüm, Rüdiger, Vodafone Group
>> <ru...@vodafone.com> wrote:
>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Yann Ylavic <yl...@gmail.com>
>>>> Gesendet: Mittwoch, 10. April 2019 12:49
>>>>
>>>> Do user-agents support "Transfer-Encoding: gzip, chunked" currently?
>>>> That'd be the best/easier solution I think.
>>>
>>> But TE is only hop-by-hop isn't it?. This would mean no e2e
>> compression?
>>
>> A proxy can still forward hop-by-hop things, e.g. mod_proxy forwards
>> "chunked" encoding if no previous handler/input-filter reads the body.
>
> My understanding was that mod_proxy "dechunks" the body and "chunks" it again.
> It does not pass it transparently.
> This would be a resource intensive job with compression.
I believe that a proxy could de-chunk without removing the gzip.
But how would our output filter infrastructure treat that? mod_deflate
stays inactive when faced with an existing Content-Encoding, but does not
watch out for Transfer-Encodings already applied. No resource type output
filter does, right?
Interesting topic.
- Stefan
AW: keep-alive and vary in 304 responses
Posted by Plüm,
Rüdiger,
Vodafone Group <ru...@vodafone.com>.
C2 General
> -----Ursprüngliche Nachricht-----
> Von: Yann Ylavic <yl...@gmail.com>
> Gesendet: Mittwoch, 10. April 2019 14:04
> An: httpd-dev <de...@httpd.apache.org>
> Betreff: Re: keep-alive and vary in 304 responses
>
> On Wed, Apr 10, 2019 at 1:03 PM Plüm, Rüdiger, Vodafone Group
> <ru...@vodafone.com> wrote:
> >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Yann Ylavic <yl...@gmail.com>
> > > Gesendet: Mittwoch, 10. April 2019 12:49
> > >
> > > Do user-agents support "Transfer-Encoding: gzip, chunked" currently?
> > > That'd be the best/easier solution I think.
> >
> > But TE is only hop-by-hop isn't it?. This would mean no e2e
> compression?
>
> A proxy can still forward hop-by-hop things, e.g. mod_proxy forwards
> "chunked" encoding if no previous handler/input-filter reads the body.
My understanding was that mod_proxy "dechunks" the body and "chunks" it again.
It does not pass it transparently.
This would be a resource intensive job with compression.
Regards
Rüdiger
Re: keep-alive and vary in 304 responses
Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, Apr 10, 2019 at 1:03 PM Plüm, Rüdiger, Vodafone Group
<ru...@vodafone.com> wrote:
>
> > -----Ursprüngliche Nachricht-----
> > Von: Yann Ylavic <yl...@gmail.com>
> > Gesendet: Mittwoch, 10. April 2019 12:49
> >
> > Do user-agents support "Transfer-Encoding: gzip, chunked" currently?
> > That'd be the best/easier solution I think.
>
> But TE is only hop-by-hop isn't it?. This would mean no e2e compression?
A proxy can still forward hop-by-hop things, e.g. mod_proxy forwards
"chunked" encoding if no previous handler/input-filter reads the body.
With CE or TE, if the plain response body is needed by an intermediary
it will be gunziped, otherwise it probably can pass through.
Regards,
Yann.
AW: keep-alive and vary in 304 responses
Posted by Plüm,
Rüdiger,
Vodafone Group <ru...@vodafone.com>.
C2 General
> -----Ursprüngliche Nachricht-----
> Von: Yann Ylavic <yl...@gmail.com>
> Gesendet: Mittwoch, 10. April 2019 12:49
> An: httpd-dev <de...@httpd.apache.org>
> Betreff: Re: keep-alive and vary in 304 responses
>
> On Wed, Apr 10, 2019 at 12:10 PM Stefan Eissing
> <st...@greenbytes.de> wrote:
> >
> > > Am 09.04.2019 um 18:48 schrieb Roy T. Fielding <fi...@gbiv.com>:
> > >>
> > >> 2. Validation responses lose the "Vary" header from the
> unconditional response. This happens on resources where mod_deflate is
> active.
> > >> The 200 response without any "if-modified-since" etc. carries
> "Vary: Accept-Encoding" as it should.
> > >> The 304 response with "if-modified-since" has no "Vary" header.
> > >> The code in mod_deflate looks as if it tries to add it in any
> case, where is it lost?
> > >
> > > That's one of many bugs related to design problems with mod_deflate.
> Basically,
> > > it chooses to insert itself into the response handling after the
> other checks are done,
> > > which means a 304 doesn't even know yet whether the response would
> use gzip.
> > > The solution (which I never found time to implement) is to replace
> dynamic gzip with
> > > a transfer encoding generator,
>
> Do user-agents support "Transfer-Encoding: gzip, chunked" currently?
> That'd be the best/easier solution I think.
But TE is only hop-by-hop isn't it?. This would mean no e2e compression?
Also we need to keep in mind similar issues for similar filters e.g. mod_brotli
Regards
Rüdiger
Re: keep-alive and vary in 304 responses
Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, Apr 10, 2019 at 12:10 PM Stefan Eissing
<st...@greenbytes.de> wrote:
>
> > Am 09.04.2019 um 18:48 schrieb Roy T. Fielding <fi...@gbiv.com>:
> >>
> >> 2. Validation responses lose the "Vary" header from the unconditional response. This happens on resources where mod_deflate is active.
> >> The 200 response without any "if-modified-since" etc. carries "Vary: Accept-Encoding" as it should.
> >> The 304 response with "if-modified-since" has no "Vary" header.
> >> The code in mod_deflate looks as if it tries to add it in any case, where is it lost?
> >
> > That's one of many bugs related to design problems with mod_deflate. Basically,
> > it chooses to insert itself into the response handling after the other checks are done,
> > which means a 304 doesn't even know yet whether the response would use gzip.
> > The solution (which I never found time to implement) is to replace dynamic gzip with
> > a transfer encoding generator,
Do user-agents support "Transfer-Encoding: gzip, chunked" currently?
That'd be the best/easier solution I think.
> > and/or to rewrite the mod_deflate CE encoder to act more
> > like mod_cache.
>
> Thinking out loud here:
>
> Acting more like mod_cache would then split the deflate output filter into a resource and a protocol type one, I assume?
>
> The protocol one would
> a. check if its resource counterpart was active
> b. or, if the requests status is 304, if it would have been active
> add "Vary: Accept-Encoding" if one of those is true
>
> For b. a bit of heuristic seems necessary:
> - detect that the resource filter has been added before *)
> - assume it applies when content-length is not set
> - check content-length otherwise
> - perform the other (r->subprocess_env etc.) checks as in resource filter
>
>
> *) It could detect the resource filter in r->output_filters, however ap_send_error_respons() overwrites this with r->proto_output_filters. Seems like a good idea to save them for later inspection?
I didn't think of it yet but it looks more complicated.
Still the etag would have to differ for the same gziped or not
resource, but not with the TE solution right?
Regards,
Yann.
Re: keep-alive and vary in 304 responses
Posted by Stefan Eissing <st...@greenbytes.de>.
> Am 09.04.2019 um 18:48 schrieb Roy T. Fielding <fi...@gbiv.com>:
>
>>
>> 2. Validation responses lose the "Vary" header from the unconditional response. This happens on resources where mod_deflate is active.
>> The 200 response without any "if-modified-since" etc. carries "Vary: Accept-Encoding" as it should.
>> The 304 response with "if-modified-since" has no "Vary" header.
>> The code in mod_deflate looks as if it tries to add it in any case, where is it lost?
>
> That's one of many bugs related to design problems with mod_deflate. Basically,
> it chooses to insert itself into the response handling after the other checks are done,
> which means a 304 doesn't even know yet whether the response would use gzip.
> The solution (which I never found time to implement) is to replace dynamic gzip with
> a transfer encoding generator, and/or to rewrite the mod_deflate CE encoder to act more
> like mod_cache.
Thinking out loud here:
Acting more like mod_cache would then split the deflate output filter into a resource and a protocol type one, I assume?
The protocol one would
a. check if its resource counterpart was active
b. or, if the requests status is 304, if it would have been active
add "Vary: Accept-Encoding" if one of those is true
For b. a bit of heuristic seems necessary:
- detect that the resource filter has been added before *)
- assume it applies when content-length is not set
- check content-length otherwise
- perform the other (r->subprocess_env etc.) checks as in resource filter
*) It could detect the resource filter in r->output_filters, however ap_send_error_respons() overwrites this with r->proto_output_filters. Seems like a good idea to save them for later inspection?
-Stefan
HTTP Protocol workshop notes
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Tue, Apr 9, 2019 at 11:48 AM Roy T. Fielding <fi...@gbiv.com> wrote:
> > On Apr 9, 2019, at 3:30 AM, Stefan Eissing <st...@greenbytes.de>
> wrote:
> >
> > I just did some tests with https://redbot.org/ (the site tester by Mark
> Nottingham) against our server and it notifies of 2 things:
> >
> > 1. The "Keep-Alive" header is deprecated. I tried to "Header unset
> Keep-Alive" but that has no effect. Seems to be added very late.
> > Do we have a way to suppress it?
>
> Please hold off on that one. We received several requests that it not be
> deprecated
> because the information that Apache sends is useful for avoiding request
> loss when
> the max is reached.
>
Speaking of external requests and HTTP interests, as usual Daniel has done
a very good job summarizing this year's workshop that Stefan had brought to
our attention a number times;
https://daniel.haxx.se/blog/2019/04/02/the-http-workshop-2019-begins/
https://daniel.haxx.se/blog/2019/04/04/more-amsterdamned-workshop/
https://daniel.haxx.se/blog/2019/04/04/workshop-season-4-finale/
Worth a skim.
Re: keep-alive and vary in 304 responses
Posted by "Roy T. Fielding" <fi...@gbiv.com>.
> On Apr 9, 2019, at 3:30 AM, Stefan Eissing <st...@greenbytes.de> wrote:
>
> I just did some tests with https://redbot.org/ (the site tester by Mark Nottingham) against our server and it notifies of 2 things:
>
> 1. The "Keep-Alive" header is deprecated. I tried to "Header unset Keep-Alive" but that has no effect. Seems to be added very late.
> Do we have a way to suppress it?
Please hold off on that one. We received several requests that it not be deprecated
because the information that Apache sends is useful for avoiding request loss when
the max is reached.
> 2. Validation responses lose the "Vary" header from the unconditional response. This happens on resources where mod_deflate is active.
> The 200 response without any "if-modified-since" etc. carries "Vary: Accept-Encoding" as it should.
> The 304 response with "if-modified-since" has no "Vary" header.
> The code in mod_deflate looks as if it tries to add it in any case, where is it lost?
That's one of many bugs related to design problems with mod_deflate. Basically,
it chooses to insert itself into the response handling after the other checks are done,
which means a 304 doesn't even know yet whether the response would use gzip.
The solution (which I never found time to implement) is to replace dynamic gzip with
a transfer encoding generator, and/or to rewrite the mod_deflate CE encoder to act more
like mod_cache.
....Roy